Jul 21 01:34:10.772264 ip-10-0-139-183 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 21 01:34:10.772448 ip-10-0-139-183 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 21 01:34:10.772570 ip-10-0-139-183 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 21 01:34:10.772951 ip-10-0-139-183 systemd[1]: Failed to start Kubernetes Kubelet. Jul 21 01:34:20.893039 ip-10-0-139-183 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 21 01:34:20.893059 ip-10-0-139-183 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 723b71cfc9a24bd6bc04e2bf61616db4 -- Jul 21 01:36:16.301773 ip-10-0-139-183 systemd[1]: Starting Kubernetes Kubelet... Jul 21 01:36:16.662853 ip-10-0-139-183 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 21 01:36:16.662853 ip-10-0-139-183 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 21 01:36:16.662853 ip-10-0-139-183 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 21 01:36:16.662853 ip-10-0-139-183 kubenswrapper[2580]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 21 01:36:16.662853 ip-10-0-139-183 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 21 01:36:16.664747 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.664656 2580 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 21 01:36:16.669641 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.669616 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 01:36:16.669641 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.669638 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 01:36:16.669641 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.669643 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 01:36:16.669641 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.669648 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 01:36:16.669897 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.669652 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 01:36:16.669897 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.669657 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 01:36:16.669897 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.669660 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 01:36:16.669897 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.669664 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 01:36:16.669897 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.669668 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 01:36:16.669897 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.669672 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 01:36:16.669897 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.669675 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 01:36:16.669897 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.669679 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 01:36:16.669897 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.669683 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 01:36:16.669897 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.669689 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 01:36:16.669897 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.669693 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 01:36:16.669897 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.669697 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 01:36:16.669897 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.669701 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 01:36:16.669897 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.669705 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 01:36:16.669897 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.669709 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 01:36:16.669897 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.669713 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 01:36:16.669897 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.669719 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 01:36:16.669897 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.669723 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 01:36:16.669897 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.669727 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 21 01:36:16.669897 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.669731 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 01:36:16.670727 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.669735 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 01:36:16.670727 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.669739 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 01:36:16.670727 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.669743 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 01:36:16.670727 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.669751 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 01:36:16.670727 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.669755 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 01:36:16.670727 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.669759 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 01:36:16.670727 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.669763 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 01:36:16.670727 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.669767 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 01:36:16.670727 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.669771 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 01:36:16.670727 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.669775 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 01:36:16.670727 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.669780 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 01:36:16.670727 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.669785 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 01:36:16.670727 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.669790 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 01:36:16.670727 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.669794 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 01:36:16.670727 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.669799 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 01:36:16.670727 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.669804 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 01:36:16.670727 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.669808 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 01:36:16.670727 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.669815 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 01:36:16.670727 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.669820 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 01:36:16.670727 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.669824 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 01:36:16.671581 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.669831 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 01:36:16.671581 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.669839 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 01:36:16.671581 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.669845 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 01:36:16.671581 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.669849 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 01:36:16.671581 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.669853 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 01:36:16.671581 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.669858 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 01:36:16.671581 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.669864 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 01:36:16.671581 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.669869 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 01:36:16.671581 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.669873 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 01:36:16.671581 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.669877 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 01:36:16.671581 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.669881 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 01:36:16.671581 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.669886 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 01:36:16.671581 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.669890 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 01:36:16.671581 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.669894 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 01:36:16.671581 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.669898 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 01:36:16.671581 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.669903 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 01:36:16.671581 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.669907 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 01:36:16.671581 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.669912 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 01:36:16.671581 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.669916 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 01:36:16.672343 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.669920 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 01:36:16.672343 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.669925 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 01:36:16.672343 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.669929 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 01:36:16.672343 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.669935 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 01:36:16.672343 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.669939 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 01:36:16.672343 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.669943 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 01:36:16.672343 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.669948 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 01:36:16.672343 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.669952 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 01:36:16.672343 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.669958 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 01:36:16.672343 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.669962 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 01:36:16.672343 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.669967 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 01:36:16.672343 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.669972 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 01:36:16.672343 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.669976 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 01:36:16.672343 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.669980 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 01:36:16.672343 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.669985 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 01:36:16.672343 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.669990 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 01:36:16.672343 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.669994 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 01:36:16.672343 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.669998 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 01:36:16.672343 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.670003 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 01:36:16.672343 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.670007 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 01:36:16.673200 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.670014 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 01:36:16.673200 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.670019 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 01:36:16.673200 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.670023 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 01:36:16.673200 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.670027 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 01:36:16.673200 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.670034 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 01:36:16.673200 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.670041 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 01:36:16.673200 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.670048 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 01:36:16.673200 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.670054 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 01:36:16.673200 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.670060 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 01:36:16.673200 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.670064 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 01:36:16.673200 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.670069 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 01:36:16.673200 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.670074 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 01:36:16.673200 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.670078 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 01:36:16.673200 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.670083 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 01:36:16.673200 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.670089 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 01:36:16.673200 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.670095 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 01:36:16.673200 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671307 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 01:36:16.673200 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671317 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 01:36:16.673200 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671323 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 01:36:16.674129 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671328 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 01:36:16.674129 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671333 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 01:36:16.674129 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671337 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 01:36:16.674129 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671343 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 01:36:16.674129 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671347 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 01:36:16.674129 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671351 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 01:36:16.674129 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671356 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 01:36:16.674129 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671360 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 01:36:16.674129 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671365 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 01:36:16.674129 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671370 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 01:36:16.674129 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671374 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 21 01:36:16.674129 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671378 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 01:36:16.674129 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671382 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 01:36:16.674129 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671386 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 01:36:16.674129 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671392 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 01:36:16.674129 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671397 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 01:36:16.674129 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671402 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 01:36:16.674129 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671406 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 01:36:16.674129 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671411 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 01:36:16.674129 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671415 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 01:36:16.674912 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671419 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 01:36:16.674912 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671424 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 01:36:16.674912 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671428 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 01:36:16.674912 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671432 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 01:36:16.674912 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671437 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 01:36:16.674912 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671441 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 01:36:16.674912 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671463 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 01:36:16.674912 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671469 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 01:36:16.674912 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671473 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 01:36:16.674912 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671477 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 01:36:16.674912 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671481 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 01:36:16.674912 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671486 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 01:36:16.674912 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671490 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 01:36:16.674912 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671494 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 01:36:16.674912 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671500 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 01:36:16.674912 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671507 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 01:36:16.674912 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671512 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 01:36:16.674912 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671516 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 01:36:16.674912 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671521 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 01:36:16.674912 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671525 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 01:36:16.675557 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671529 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 01:36:16.675557 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671534 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 01:36:16.675557 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671538 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 01:36:16.675557 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671542 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 01:36:16.675557 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671546 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 01:36:16.675557 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671551 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 01:36:16.675557 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671555 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 01:36:16.675557 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671560 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 01:36:16.675557 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671566 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 01:36:16.675557 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671570 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 01:36:16.675557 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671575 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 01:36:16.675557 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671580 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 01:36:16.675557 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671585 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 01:36:16.675557 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671589 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 01:36:16.675557 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671593 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 01:36:16.675557 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671598 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 01:36:16.675557 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671602 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 01:36:16.675557 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671606 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 01:36:16.675557 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671610 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 01:36:16.676113 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671614 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 01:36:16.676113 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671619 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 01:36:16.676113 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671623 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 01:36:16.676113 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671627 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 01:36:16.676113 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671632 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 01:36:16.676113 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671636 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 01:36:16.676113 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671641 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 01:36:16.676113 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671645 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 01:36:16.676113 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671650 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 01:36:16.676113 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671654 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 01:36:16.676113 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671659 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 01:36:16.676113 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671663 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 01:36:16.676113 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671667 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 01:36:16.676113 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671672 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 01:36:16.676113 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671676 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 01:36:16.676113 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671680 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 01:36:16.676113 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671686 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 01:36:16.676113 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671692 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 01:36:16.676113 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671697 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 01:36:16.676113 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671701 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 01:36:16.676627 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671706 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 01:36:16.676627 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671712 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 01:36:16.676627 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671719 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 01:36:16.676627 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671723 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 01:36:16.676627 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671727 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 01:36:16.676627 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671731 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 01:36:16.676627 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671736 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 01:36:16.676627 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671740 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 01:36:16.676627 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671744 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 01:36:16.676627 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671749 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 01:36:16.676627 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671753 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 01:36:16.676627 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671757 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 01:36:16.676627 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671761 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 01:36:16.676627 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671766 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 01:36:16.676627 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671770 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 01:36:16.676627 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671774 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 01:36:16.676627 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.671779 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 01:36:16.676627 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672405 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 01:36:16.676627 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672413 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 01:36:16.676627 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672418 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 01:36:16.677195 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672423 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 01:36:16.677195 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672427 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 01:36:16.677195 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672432 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 01:36:16.677195 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672436 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 01:36:16.677195 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672440 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 01:36:16.677195 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672463 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 01:36:16.677195 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672471 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 01:36:16.677195 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672477 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 01:36:16.677195 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672482 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 01:36:16.677195 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672486 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 01:36:16.677195 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672491 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 01:36:16.677195 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672495 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 01:36:16.677195 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672500 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 01:36:16.677195 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672505 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 01:36:16.677195 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672510 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 01:36:16.677195 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672514 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 01:36:16.677195 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672518 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 01:36:16.677195 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672522 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 01:36:16.677195 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672527 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 01:36:16.677697 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672531 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 01:36:16.677697 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672535 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 01:36:16.677697 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672539 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 01:36:16.677697 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672544 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 01:36:16.677697 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672548 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 01:36:16.677697 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672556 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 01:36:16.677697 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672561 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 01:36:16.677697 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672567 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 01:36:16.677697 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672572 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 01:36:16.677697 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672576 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 01:36:16.677697 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672580 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 01:36:16.677697 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672585 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 01:36:16.677697 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672589 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 01:36:16.677697 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672594 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 01:36:16.677697 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672598 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 01:36:16.677697 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672602 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 01:36:16.677697 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672606 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 01:36:16.677697 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672610 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 01:36:16.677697 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672615 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 01:36:16.677697 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672619 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 01:36:16.678167 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672623 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 01:36:16.678167 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672628 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 01:36:16.678167 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672632 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 01:36:16.678167 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672636 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 01:36:16.678167 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672640 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 01:36:16.678167 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672645 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 01:36:16.678167 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672649 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 01:36:16.678167 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672655 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 01:36:16.678167 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672659 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 01:36:16.678167 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672663 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 01:36:16.678167 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672667 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 01:36:16.678167 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672672 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 01:36:16.678167 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672676 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 01:36:16.678167 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672680 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 01:36:16.678167 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672684 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 01:36:16.678167 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672688 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 01:36:16.678167 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672692 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 01:36:16.678167 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672698 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 01:36:16.678167 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672702 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 01:36:16.678167 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672707 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 01:36:16.678653 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672711 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 01:36:16.678653 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672715 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 01:36:16.678653 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672719 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 01:36:16.678653 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672723 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 01:36:16.678653 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672727 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 01:36:16.678653 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672735 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 01:36:16.678653 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672741 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 01:36:16.678653 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672746 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 01:36:16.678653 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672750 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 01:36:16.678653 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672754 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 01:36:16.678653 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672758 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 01:36:16.678653 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672762 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 01:36:16.678653 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672767 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 01:36:16.678653 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672771 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 01:36:16.678653 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672776 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 01:36:16.678653 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672780 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 01:36:16.678653 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672784 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 01:36:16.678653 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672788 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 01:36:16.678653 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672792 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 01:36:16.679118 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672796 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 01:36:16.679118 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672813 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 01:36:16.679118 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672818 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 01:36:16.679118 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672822 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 01:36:16.679118 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672826 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 01:36:16.679118 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672830 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 01:36:16.679118 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672834 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 01:36:16.679118 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672838 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 01:36:16.679118 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672842 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 01:36:16.679118 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672846 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 01:36:16.679118 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672852 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 01:36:16.679118 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672857 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 01:36:16.679118 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672861 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 01:36:16.679118 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672867 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 01:36:16.679118 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672873 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 01:36:16.679118 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672877 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 01:36:16.679118 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672881 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 21 01:36:16.679118 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.672885 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 01:36:16.679118 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.672992 2580 flags.go:64] FLAG: --address="0.0.0.0" Jul 21 01:36:16.679118 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673003 2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 21 01:36:16.679118 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673012 2580 flags.go:64] FLAG: --anonymous-auth="true" Jul 21 01:36:16.679633 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673024 2580 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 21 01:36:16.679633 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673030 2580 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 21 01:36:16.679633 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673035 2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 21 01:36:16.679633 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673042 2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 21 01:36:16.679633 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673049 2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 21 01:36:16.679633 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673054 2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 21 01:36:16.679633 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673059 2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 21 01:36:16.679633 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673065 2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 21 01:36:16.679633 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673070 2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 21 01:36:16.679633 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673075 2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 21 01:36:16.679633 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673080 2580 flags.go:64] FLAG: --cgroup-root="" Jul 21 01:36:16.679633 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673084 2580 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 21 01:36:16.679633 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673089 2580 flags.go:64] FLAG: --client-ca-file="" Jul 21 01:36:16.679633 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673095 2580 flags.go:64] FLAG: --cloud-provider="external" Jul 21 01:36:16.679633 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673101 2580 flags.go:64] FLAG: --cluster-dns="[]" Jul 21 01:36:16.679633 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673108 2580 flags.go:64] FLAG: --cluster-domain="" Jul 21 01:36:16.679633 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673112 2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 21 01:36:16.679633 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673117 2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 21 01:36:16.679633 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673123 2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 21 01:36:16.679633 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673128 2580 flags.go:64] FLAG: --container-log-max-files="5" Jul 21 01:36:16.679633 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673134 2580 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 21 01:36:16.679633 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673143 2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 21 01:36:16.679633 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673148 2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 21 01:36:16.680250 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673153 2580 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 21 01:36:16.680250 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673158 2580 flags.go:64] FLAG: --contention-profiling="false" Jul 21 01:36:16.680250 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673163 2580 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 21 01:36:16.680250 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673167 2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 21 01:36:16.680250 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673173 2580 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 21 01:36:16.680250 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673177 2580 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 21 01:36:16.680250 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673184 2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 21 01:36:16.680250 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673189 2580 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 21 01:36:16.680250 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673193 2580 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 21 01:36:16.680250 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673198 2580 flags.go:64] FLAG: --enable-load-reader="false" Jul 21 01:36:16.680250 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673202 2580 flags.go:64] FLAG: --enable-server="true" Jul 21 01:36:16.680250 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673207 2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 21 01:36:16.680250 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673213 2580 flags.go:64] FLAG: --event-burst="100" Jul 21 01:36:16.680250 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673218 2580 flags.go:64] FLAG: --event-qps="50" Jul 21 01:36:16.680250 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673223 2580 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 21 01:36:16.680250 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673228 2580 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 21 01:36:16.680250 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673233 2580 flags.go:64] FLAG: --eviction-hard="" Jul 21 01:36:16.680250 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673239 2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 21 01:36:16.680250 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673244 2580 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 21 01:36:16.680250 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673249 2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 21 01:36:16.680250 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673254 2580 flags.go:64] FLAG: --eviction-soft="" Jul 21 01:36:16.680250 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673258 2580 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 21 01:36:16.680250 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673263 2580 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 21 01:36:16.680250 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673269 2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 21 01:36:16.680250 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673274 2580 flags.go:64] FLAG: --experimental-mounter-path="" Jul 21 01:36:16.680922 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673279 2580 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 21 01:36:16.680922 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673283 2580 flags.go:64] FLAG: --fail-swap-on="true" Jul 21 01:36:16.680922 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673288 2580 flags.go:64] FLAG: --feature-gates="" Jul 21 01:36:16.680922 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673294 2580 flags.go:64] FLAG: --file-check-frequency="20s" Jul 21 01:36:16.680922 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673298 2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 21 01:36:16.680922 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673306 2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 21 01:36:16.680922 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673312 2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 21 01:36:16.680922 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673317 2580 flags.go:64] FLAG: --healthz-port="10248" Jul 21 01:36:16.680922 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673321 2580 flags.go:64] FLAG: --help="false" Jul 21 01:36:16.680922 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673326 2580 flags.go:64] FLAG: --hostname-override="ip-10-0-139-183.ec2.internal" Jul 21 01:36:16.680922 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673331 2580 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 21 01:36:16.680922 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673336 2580 flags.go:64] FLAG: --http-check-frequency="20s" Jul 21 01:36:16.680922 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673341 2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 21 01:36:16.680922 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673346 2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 21 01:36:16.680922 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673352 2580 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 21 01:36:16.680922 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673356 2580 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 21 01:36:16.680922 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673361 2580 flags.go:64] FLAG: --image-service-endpoint="" Jul 21 01:36:16.680922 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673365 2580 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 21 01:36:16.680922 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673370 2580 flags.go:64] FLAG: --kube-api-burst="100" Jul 21 01:36:16.680922 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673374 2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 21 01:36:16.680922 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673379 2580 flags.go:64] FLAG: --kube-api-qps="50" Jul 21 01:36:16.680922 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673384 2580 flags.go:64] FLAG: --kube-reserved="" Jul 21 01:36:16.680922 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673388 2580 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 21 01:36:16.680922 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673392 2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 21 01:36:16.681573 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673398 2580 flags.go:64] FLAG: --kubelet-cgroups="" Jul 21 01:36:16.681573 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673402 2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 21 01:36:16.681573 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673407 2580 flags.go:64] FLAG: --lock-file="" Jul 21 01:36:16.681573 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673411 2580 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 21 01:36:16.681573 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673416 2580 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 21 01:36:16.681573 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673421 2580 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 21 01:36:16.681573 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673430 2580 flags.go:64] FLAG: --log-json-split-stream="false" Jul 21 01:36:16.681573 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673443 2580 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 21 01:36:16.681573 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673467 2580 flags.go:64] FLAG: --log-text-split-stream="false" Jul 21 01:36:16.681573 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673472 2580 flags.go:64] FLAG: --logging-format="text" Jul 21 01:36:16.681573 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673477 2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 21 01:36:16.681573 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673482 2580 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 21 01:36:16.681573 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673487 2580 flags.go:64] FLAG: --manifest-url="" Jul 21 01:36:16.681573 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673494 2580 flags.go:64] FLAG: --manifest-url-header="" Jul 21 01:36:16.681573 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673501 2580 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 21 01:36:16.681573 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673507 2580 flags.go:64] FLAG: --max-open-files="1000000" Jul 21 01:36:16.681573 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673512 2580 flags.go:64] FLAG: --max-pods="110" Jul 21 01:36:16.681573 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673517 2580 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 21 01:36:16.681573 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673522 2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 21 01:36:16.681573 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673527 2580 flags.go:64] FLAG: --memory-manager-policy="None" Jul 21 01:36:16.681573 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673532 2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 21 01:36:16.681573 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673536 2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 21 01:36:16.681573 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673541 2580 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 21 01:36:16.681573 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673546 2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 21 01:36:16.681573 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673558 2580 flags.go:64] FLAG: --node-status-max-images="50" Jul 21 01:36:16.682176 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673563 2580 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 21 01:36:16.682176 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673567 2580 flags.go:64] FLAG: --oom-score-adj="-999" Jul 21 01:36:16.682176 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673572 2580 flags.go:64] FLAG: --pod-cidr="" Jul 21 01:36:16.682176 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673577 2580 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 21 01:36:16.682176 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673585 2580 flags.go:64] FLAG: --pod-manifest-path="" Jul 21 01:36:16.682176 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673590 2580 flags.go:64] FLAG: --pod-max-pids="-1" Jul 21 01:36:16.682176 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673595 2580 flags.go:64] FLAG: --pods-per-core="0" Jul 21 01:36:16.682176 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673600 2580 flags.go:64] FLAG: --port="10250" Jul 21 01:36:16.682176 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673605 2580 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 21 01:36:16.682176 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673609 2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0447c3b064265a4e5" Jul 21 01:36:16.682176 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673614 2580 flags.go:64] FLAG: --qos-reserved="" Jul 21 01:36:16.682176 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673619 2580 flags.go:64] FLAG: --read-only-port="10255" Jul 21 01:36:16.682176 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673623 2580 flags.go:64] FLAG: --register-node="true" Jul 21 01:36:16.682176 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673628 2580 flags.go:64] FLAG: --register-with-taints="" Jul 21 01:36:16.682176 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673634 2580 flags.go:64] FLAG: --registry-burst="10" Jul 21 01:36:16.682176 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673640 2580 flags.go:64] FLAG: --registry-qps="5" Jul 21 01:36:16.682176 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673645 2580 flags.go:64] FLAG: --reserved-cpus="" Jul 21 01:36:16.682176 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673649 2580 flags.go:64] FLAG: --reserved-memory="" Jul 21 01:36:16.682176 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673655 2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 21 01:36:16.682176 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673660 2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 21 01:36:16.682176 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673667 2580 flags.go:64] FLAG: --rotate-certificates="false" Jul 21 01:36:16.682176 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673671 2580 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 21 01:36:16.682176 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673676 2580 flags.go:64] FLAG: --runonce="false" Jul 21 01:36:16.682176 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673681 2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 21 01:36:16.682176 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673686 2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 21 01:36:16.682784 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673691 2580 flags.go:64] FLAG: --seccomp-default="false" Jul 21 01:36:16.682784 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673695 2580 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 21 01:36:16.682784 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673699 2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 21 01:36:16.682784 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673704 2580 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 21 01:36:16.682784 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673709 2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 21 01:36:16.682784 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673714 2580 flags.go:64] FLAG: --storage-driver-password="root" Jul 21 01:36:16.682784 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673718 2580 flags.go:64] FLAG: --storage-driver-secure="false" Jul 21 01:36:16.682784 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673723 2580 flags.go:64] FLAG: --storage-driver-table="stats" Jul 21 01:36:16.682784 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673727 2580 flags.go:64] FLAG: --storage-driver-user="root" Jul 21 01:36:16.682784 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673732 2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 21 01:36:16.682784 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673737 2580 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 21 01:36:16.682784 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673741 2580 flags.go:64] FLAG: --system-cgroups="" Jul 21 01:36:16.682784 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673747 2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 21 01:36:16.682784 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673755 2580 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 21 01:36:16.682784 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673760 2580 flags.go:64] FLAG: --tls-cert-file="" Jul 21 01:36:16.682784 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673765 2580 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 21 01:36:16.682784 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673773 2580 flags.go:64] FLAG: --tls-min-version="" Jul 21 01:36:16.682784 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673777 2580 flags.go:64] FLAG: --tls-private-key-file="" Jul 21 01:36:16.682784 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673782 2580 flags.go:64] FLAG: --topology-manager-policy="none" Jul 21 01:36:16.682784 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673787 2580 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 21 01:36:16.682784 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673792 2580 flags.go:64] FLAG: --topology-manager-scope="container" Jul 21 01:36:16.682784 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673796 2580 flags.go:64] FLAG: --v="2" Jul 21 01:36:16.682784 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673803 2580 flags.go:64] FLAG: --version="false" Jul 21 01:36:16.682784 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673810 2580 flags.go:64] FLAG: --vmodule="" Jul 21 01:36:16.682784 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673816 2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 21 01:36:16.683386 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.673821 2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 21 01:36:16.683386 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674033 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 01:36:16.683386 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674043 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 01:36:16.683386 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674048 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 01:36:16.683386 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674053 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 01:36:16.683386 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674057 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 01:36:16.683386 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674061 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 01:36:16.683386 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674065 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 21 01:36:16.683386 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674069 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 01:36:16.683386 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674074 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 01:36:16.683386 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674078 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 01:36:16.683386 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674082 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 01:36:16.683386 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674086 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 01:36:16.683386 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674091 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 01:36:16.683386 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674095 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 01:36:16.683386 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674100 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 01:36:16.683386 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674104 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 01:36:16.683386 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674108 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 01:36:16.683386 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674112 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 01:36:16.683386 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674117 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 01:36:16.683881 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674121 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 01:36:16.683881 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674125 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 01:36:16.683881 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674130 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 01:36:16.683881 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674133 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 01:36:16.683881 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674139 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 01:36:16.683881 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674145 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 01:36:16.683881 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674149 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 01:36:16.683881 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674153 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 01:36:16.683881 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674157 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 01:36:16.683881 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674162 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 01:36:16.683881 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674166 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 01:36:16.683881 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674172 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 01:36:16.683881 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674177 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 01:36:16.683881 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674184 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 01:36:16.683881 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674190 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 01:36:16.683881 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674194 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 01:36:16.683881 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674198 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 01:36:16.683881 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674205 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 01:36:16.683881 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674211 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 01:36:16.684384 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674216 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 01:36:16.684384 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674221 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 01:36:16.684384 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674225 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 01:36:16.684384 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674230 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 01:36:16.684384 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674234 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 01:36:16.684384 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674239 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 01:36:16.684384 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674243 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 01:36:16.684384 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674247 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 01:36:16.684384 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674252 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 01:36:16.684384 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674256 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 01:36:16.684384 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674261 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 01:36:16.684384 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674265 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 01:36:16.684384 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674269 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 01:36:16.684384 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674273 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 01:36:16.684384 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674278 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 01:36:16.684384 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674282 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 01:36:16.684384 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674286 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 01:36:16.684384 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674290 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 01:36:16.684384 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674294 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 01:36:16.684384 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674298 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 01:36:16.684892 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674303 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 01:36:16.684892 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674307 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 01:36:16.684892 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674311 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 01:36:16.684892 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674316 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 01:36:16.684892 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674320 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 01:36:16.684892 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674325 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 01:36:16.684892 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674331 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 01:36:16.684892 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674338 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 01:36:16.684892 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674342 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 01:36:16.684892 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674347 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 01:36:16.684892 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674351 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 01:36:16.684892 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674355 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 01:36:16.684892 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674359 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 01:36:16.684892 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674363 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 01:36:16.684892 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674367 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 01:36:16.684892 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674372 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 01:36:16.684892 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674376 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 01:36:16.684892 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674381 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 01:36:16.684892 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674385 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 01:36:16.684892 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674389 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 01:36:16.685363 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674393 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 01:36:16.685363 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674397 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 01:36:16.685363 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674403 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 01:36:16.685363 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674407 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 01:36:16.685363 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674411 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 01:36:16.685363 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674415 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 01:36:16.685363 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674420 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 01:36:16.685363 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674424 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 01:36:16.685363 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674428 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 01:36:16.685363 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674432 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 01:36:16.685363 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674439 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 01:36:16.685363 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674444 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 01:36:16.685363 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674466 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 01:36:16.685363 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674471 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 01:36:16.685363 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674476 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 01:36:16.685363 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674479 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 01:36:16.685363 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674483 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 01:36:16.685363 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674488 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 01:36:16.685363 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674495 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 01:36:16.685824 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674500 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 01:36:16.685824 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.674505 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 01:36:16.685824 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.674518 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 21 01:36:16.685824 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.681311 2580 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 21 01:36:16.685824 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.681326 2580 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 21 01:36:16.685824 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.682051 2580 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 21 01:36:16.685824 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.682060 2580 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 21 01:36:16.685824 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682118 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 01:36:16.685824 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682123 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 01:36:16.685824 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682127 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 01:36:16.685824 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682129 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 01:36:16.685824 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682133 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 01:36:16.685824 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682136 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 01:36:16.685824 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682139 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 21 01:36:16.685824 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682142 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 01:36:16.686191 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682145 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 01:36:16.686191 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682148 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 01:36:16.686191 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682152 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 01:36:16.686191 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682154 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 01:36:16.686191 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682157 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 01:36:16.686191 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682160 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 01:36:16.686191 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682163 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 01:36:16.686191 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682165 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 01:36:16.686191 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682168 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 01:36:16.686191 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682171 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 01:36:16.686191 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682174 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 01:36:16.686191 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682176 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 01:36:16.686191 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682179 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 01:36:16.686191 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682182 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 01:36:16.686191 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682185 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 01:36:16.686191 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682188 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 01:36:16.686191 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682191 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 01:36:16.686191 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682194 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 01:36:16.686191 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682197 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 01:36:16.686191 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682200 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 01:36:16.686728 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682203 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 01:36:16.686728 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682206 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 01:36:16.686728 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682209 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 01:36:16.686728 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682212 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 01:36:16.686728 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682214 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 01:36:16.686728 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682217 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 01:36:16.686728 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682219 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 01:36:16.686728 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682222 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 01:36:16.686728 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682224 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 01:36:16.686728 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682227 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 01:36:16.686728 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682229 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 01:36:16.686728 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682232 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 01:36:16.686728 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682235 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 01:36:16.686728 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682237 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 01:36:16.686728 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682240 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 01:36:16.686728 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682242 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 01:36:16.686728 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682245 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 01:36:16.686728 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682249 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 01:36:16.686728 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682252 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 01:36:16.687178 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682256 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 01:36:16.687178 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682260 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 01:36:16.687178 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682262 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 01:36:16.687178 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682265 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 01:36:16.687178 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682269 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 01:36:16.687178 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682273 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 01:36:16.687178 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682277 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 01:36:16.687178 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682280 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 01:36:16.687178 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682283 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 01:36:16.687178 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682286 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 01:36:16.687178 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682289 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 01:36:16.687178 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682292 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 01:36:16.687178 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682295 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 01:36:16.687178 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682297 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 01:36:16.687178 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682300 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 01:36:16.687178 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682309 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 01:36:16.687178 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682312 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 01:36:16.687178 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682315 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 01:36:16.687178 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682318 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 01:36:16.687644 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682320 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 01:36:16.687644 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682323 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 01:36:16.687644 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682326 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 01:36:16.687644 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682328 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 01:36:16.687644 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682331 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 01:36:16.687644 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682334 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 01:36:16.687644 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682336 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 01:36:16.687644 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682339 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 01:36:16.687644 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682342 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 01:36:16.687644 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682344 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 01:36:16.687644 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682347 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 01:36:16.687644 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682349 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 01:36:16.687644 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682352 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 01:36:16.687644 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682355 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 01:36:16.687644 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682357 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 01:36:16.687644 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682360 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 01:36:16.687644 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682362 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 01:36:16.687644 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682365 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 01:36:16.687644 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682368 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 01:36:16.687644 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682370 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 01:36:16.688132 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682373 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 01:36:16.688132 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682375 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 01:36:16.688132 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682378 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 01:36:16.688132 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682381 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 01:36:16.688132 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682384 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 01:36:16.688132 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682387 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 01:36:16.688132 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682390 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 01:36:16.688132 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682392 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 01:36:16.688132 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682395 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 01:36:16.688132 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682398 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 01:36:16.688132 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682401 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 01:36:16.688132 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682403 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 01:36:16.688132 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682405 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 01:36:16.688132 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.682410 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 21 01:36:16.688486 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682545 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 01:36:16.688486 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682551 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 01:36:16.688486 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682554 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 01:36:16.688486 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682557 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 01:36:16.688486 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682560 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 01:36:16.688486 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682563 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 01:36:16.688486 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682566 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 01:36:16.688486 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682568 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 01:36:16.688486 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682571 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 01:36:16.688486 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682573 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 01:36:16.688486 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682576 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 01:36:16.688486 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682578 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 01:36:16.688486 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682581 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 01:36:16.688486 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682584 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 01:36:16.688486 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682587 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 01:36:16.688486 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682589 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 01:36:16.688486 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682593 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 01:36:16.688486 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682597 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 01:36:16.688486 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682600 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 01:36:16.688962 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682603 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 01:36:16.688962 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682606 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 01:36:16.688962 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682609 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 01:36:16.688962 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682612 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 01:36:16.688962 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682615 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 01:36:16.688962 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682618 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 01:36:16.688962 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682621 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 01:36:16.688962 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682624 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 01:36:16.688962 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682626 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 01:36:16.688962 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682630 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 21 01:36:16.688962 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682632 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 01:36:16.688962 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682635 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 01:36:16.688962 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682637 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 01:36:16.688962 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682640 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 01:36:16.688962 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682642 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 01:36:16.688962 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682645 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 01:36:16.688962 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682648 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 01:36:16.688962 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682651 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 01:36:16.688962 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682653 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 01:36:16.688962 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682656 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 01:36:16.689444 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682659 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 01:36:16.689444 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682661 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 01:36:16.689444 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682664 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 01:36:16.689444 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682666 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 01:36:16.689444 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682669 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 01:36:16.689444 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682671 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 01:36:16.689444 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682674 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 01:36:16.689444 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682676 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 01:36:16.689444 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682679 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 01:36:16.689444 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682682 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 01:36:16.689444 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682685 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 01:36:16.689444 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682688 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 01:36:16.689444 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682690 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 01:36:16.689444 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682692 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 01:36:16.689444 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682695 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 01:36:16.689444 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682698 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 01:36:16.689444 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682701 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 01:36:16.689444 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682704 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 01:36:16.689444 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682708 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 01:36:16.689928 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682711 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 01:36:16.689928 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682714 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 01:36:16.689928 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682716 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 01:36:16.689928 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682719 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 01:36:16.689928 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682722 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 01:36:16.689928 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682724 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 01:36:16.689928 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682727 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 01:36:16.689928 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682729 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 01:36:16.689928 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682732 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 01:36:16.689928 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682734 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 01:36:16.689928 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682737 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 01:36:16.689928 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682740 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 01:36:16.689928 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682742 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 01:36:16.689928 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682744 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 01:36:16.689928 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682747 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 01:36:16.689928 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682750 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 01:36:16.689928 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682752 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 01:36:16.689928 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682755 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 01:36:16.689928 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682757 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 01:36:16.689928 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682759 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 01:36:16.690418 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682762 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 01:36:16.690418 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682764 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 01:36:16.690418 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682767 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 01:36:16.690418 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682769 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 01:36:16.690418 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682772 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 01:36:16.690418 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682774 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 01:36:16.690418 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682777 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 01:36:16.690418 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682779 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 01:36:16.690418 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682782 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 01:36:16.690418 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682785 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 01:36:16.690418 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682788 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 01:36:16.690418 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682790 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 01:36:16.690418 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682793 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 01:36:16.690418 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682796 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 01:36:16.690418 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682798 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 01:36:16.690418 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682801 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 01:36:16.690418 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682804 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 01:36:16.690418 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682807 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 01:36:16.690418 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682810 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 01:36:16.690962 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682814 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 01:36:16.690962 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:16.682816 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 01:36:16.690962 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.682821 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 21 01:36:16.690962 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.682930 2580 server.go:962] "Client rotation is on, will bootstrap in background" Jul 21 01:36:16.690962 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.685343 2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 21 01:36:16.690962 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.686799 2580 server.go:1019] "Starting client certificate rotation" Jul 21 01:36:16.690962 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.686900 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 21 01:36:16.690962 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.686929 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 21 01:36:16.690962 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.689324 2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 21 01:36:16.692098 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.692084 2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 21 01:36:16.701939 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.701916 2580 log.go:25] "Validated CRI v1 runtime API" Jul 21 01:36:16.703914 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.703895 2580 log.go:25] "Validated CRI v1 image API" Jul 21 01:36:16.705499 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.705479 2580 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 21 01:36:16.707527 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.707507 2580 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 a98ad5cc-91d2-46f9-8402-b75d68da5503:/dev/nvme0n1p3 e6ff8a08-3b60-434d-b534-ab6ad1ea84d2:/dev/nvme0n1p4] Jul 21 01:36:16.707581 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.707527 2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 21 01:36:16.713094 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.712984 2580 manager.go:216] Machine: {Timestamp:2026-07-21 01:36:16.71187584 +0000 UTC m=+0.299041829 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3105168 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec289306cbf1dbfb0921e4944f12e90d SystemUUID:ec289306-cbf1-dbfb-0921-e4944f12e90d BootID:723b71cf-c9a2-4bd6-bc04-e2bf61616db4 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:77:10:4b:67:47 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:77:10:4b:67:47 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ea:7f:15:74:94:26 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 21 01:36:16.713094 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.713092 2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 21 01:36:16.713286 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.713175 2580 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 21 01:36:16.714917 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.714898 2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 21 01:36:16.716068 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.716043 2580 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 21 01:36:16.716206 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.716071 2580 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-183.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 21 01:36:16.717159 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.717148 2580 topology_manager.go:138] "Creating topology manager with none policy" Jul 21 01:36:16.717208 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.717161 2580 container_manager_linux.go:309] "Creating device plugin manager" Jul 21 01:36:16.717208 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.717168 2580 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 21 01:36:16.717735 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.717725 2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 21 01:36:16.718968 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.718958 2580 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 21 01:36:16.722649 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.722637 2580 state_mem.go:36] "Initialized new in-memory state store" Jul 21 01:36:16.722762 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.722753 2580 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 21 01:36:16.722791 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.722782 2580 kubelet.go:486] "Attempting to sync node with API server" Jul 21 01:36:16.722827 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.722792 2580 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 21 01:36:16.722827 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.722806 2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 21 01:36:16.722827 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.722815 2580 kubelet.go:398] "Adding apiserver pod source" Jul 21 01:36:16.722827 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.722824 2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 21 01:36:16.723229 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.723217 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 21 01:36:16.723284 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.723240 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 21 01:36:16.725648 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.725630 2580 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 21 01:36:16.727184 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.727170 2580 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 21 01:36:16.727243 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.727201 2580 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 21 01:36:16.727791 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.727778 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 21 01:36:16.727838 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.727805 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 21 01:36:16.727838 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.727812 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 21 01:36:16.727838 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.727818 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 21 01:36:16.727838 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.727827 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 21 01:36:16.727838 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.727835 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 21 01:36:16.727971 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.727957 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 21 01:36:16.727971 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.727963 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 21 01:36:16.727971 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.727969 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 21 01:36:16.728047 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.727975 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 21 01:36:16.728047 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.727987 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 21 01:36:16.728047 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.727995 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 21 01:36:16.729335 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.729320 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 21 01:36:16.729335 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.729334 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 21 01:36:16.732192 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:16.732167 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-183.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 21 01:36:16.732293 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:16.732274 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 21 01:36:16.732337 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.732319 2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-183.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 21 01:36:16.732749 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.732736 2580 server.go:1268] "Started kubelet" Jul 21 01:36:16.732850 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.732820 2580 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 21 01:36:16.735185 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.732821 2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 21 01:36:16.735315 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.735299 2580 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 21 01:36:16.736001 ip-10-0-139-183 systemd[1]: Started Kubernetes Kubelet. Jul 21 01:36:16.736572 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.736557 2580 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 21 01:36:16.738103 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.737640 2580 server.go:310] "Adding debug handlers to kubelet server" Jul 21 01:36:16.740494 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:16.740477 2580 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 21 01:36:16.742585 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:16.741887 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-183.ec2.internal.18c42a39051ae510 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-183.ec2.internal,UID:ip-10-0-139-183.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-183.ec2.internal,},FirstTimestamp:2026-07-21 01:36:16.73271016 +0000 UTC m=+0.319876129,LastTimestamp:2026-07-21 01:36:16.73271016 +0000 UTC m=+0.319876129,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-183.ec2.internal,}" Jul 21 01:36:16.745107 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.745089 2580 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 21 01:36:16.745198 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:16.745180 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-183.ec2.internal\" not found" Jul 21 01:36:16.745266 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.745235 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 21 01:36:16.745650 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.745635 2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 21 01:36:16.745650 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.745635 2580 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 21 01:36:16.745777 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.745660 2580 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 21 01:36:16.745835 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.745779 2580 reconstruct.go:97] "Volume reconstruction finished" Jul 21 01:36:16.745835 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.745789 2580 reconciler.go:29] "Reconciler: start to sync state" Jul 21 01:36:16.745949 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.745934 2580 factory.go:55] Registering systemd factory Jul 21 01:36:16.745992 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.745966 2580 factory.go:223] Registration of the systemd container factory successfully Jul 21 01:36:16.746203 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.746192 2580 factory.go:153] Registering CRI-O factory Jul 21 01:36:16.746203 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.746205 2580 factory.go:223] Registration of the crio container factory successfully Jul 21 01:36:16.746299 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.746254 2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 21 01:36:16.746299 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.746277 2580 factory.go:103] Registering Raw factory Jul 21 01:36:16.746299 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.746292 2580 manager.go:1196] Started watching for new ooms in manager Jul 21 01:36:16.746668 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.746656 2580 manager.go:349] Starting recovery of all containers Jul 21 01:36:16.751078 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:16.751047 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 21 01:36:16.751173 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:16.751129 2580 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-183.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 21 01:36:16.756383 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.756366 2580 manager.go:354] Recovery completed Jul 21 01:36:16.760711 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.760699 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 01:36:16.762888 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.762864 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-183.ec2.internal" event="NodeHasSufficientMemory" Jul 21 01:36:16.762954 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.762895 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-183.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 01:36:16.762954 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.762909 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-183.ec2.internal" event="NodeHasSufficientPID" Jul 21 01:36:16.762954 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.762920 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-79w7j" Jul 21 01:36:16.763549 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.763538 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 01:36:16.763612 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.763537 2580 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 21 01:36:16.763612 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.763558 2580 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 21 01:36:16.763612 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.763573 2580 state_mem.go:36] "Initialized new in-memory state store" Jul 21 01:36:16.764516 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.764497 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-183.ec2.internal" event="NodeHasSufficientMemory" Jul 21 01:36:16.764731 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.764536 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-183.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 01:36:16.764731 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.764550 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-183.ec2.internal" event="NodeHasSufficientPID" Jul 21 01:36:16.764731 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:16.764625 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-183.ec2.internal.18c42a3906e74ccf default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-183.ec2.internal,UID:ip-10-0-139-183.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-183.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-183.ec2.internal,},FirstTimestamp:2026-07-21 01:36:16.762883279 +0000 UTC m=+0.350049255,LastTimestamp:2026-07-21 01:36:16.762883279 +0000 UTC m=+0.350049255,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-183.ec2.internal,}" Jul 21 01:36:16.765847 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.765835 2580 policy_none.go:49] "None policy: Start" Jul 21 01:36:16.765910 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.765851 2580 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 21 01:36:16.765910 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.765861 2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 21 01:36:16.767689 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.767678 2580 policy_none.go:47] "Start" Jul 21 01:36:16.768698 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.768681 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-79w7j" Jul 21 01:36:16.775820 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:16.775749 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-183.ec2.internal.18c42a3906e79308 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-183.ec2.internal,UID:ip-10-0-139-183.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-139-183.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-139-183.ec2.internal,},FirstTimestamp:2026-07-21 01:36:16.762901256 +0000 UTC m=+0.350067230,LastTimestamp:2026-07-21 01:36:16.762901256 +0000 UTC m=+0.350067230,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-183.ec2.internal,}" Jul 21 01:36:16.813898 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.813882 2580 manager.go:337] "Starting Device Plugin manager" Jul 21 01:36:16.813981 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:16.813917 2580 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 21 01:36:16.813981 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.813928 2580 server.go:87] "Starting device plugin registration server" Jul 21 01:36:16.814252 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.814240 2580 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 21 01:36:16.814292 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.814255 2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 21 01:36:16.814377 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.814358 2580 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 21 01:36:16.814494 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.814465 2580 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 21 01:36:16.814494 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.814479 2580 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 21 01:36:16.815045 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:16.815020 2580 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 21 01:36:16.815131 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:16.815082 2580 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-183.ec2.internal\" not found" Jul 21 01:36:16.868888 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.868855 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 21 01:36:16.870914 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.870042 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 21 01:36:16.870914 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.870081 2580 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 21 01:36:16.870914 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.870102 2580 kubelet.go:2448] "Starting kubelet main sync loop" Jul 21 01:36:16.870914 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:16.870134 2580 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 21 01:36:16.872275 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.872259 2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 01:36:16.915960 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.915708 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 01:36:16.917344 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.917320 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-183.ec2.internal" event="NodeHasSufficientMemory" Jul 21 01:36:16.917447 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.917359 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-183.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 01:36:16.917447 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.917370 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-183.ec2.internal" event="NodeHasSufficientPID" Jul 21 01:36:16.917447 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.917397 2580 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-183.ec2.internal" Jul 21 01:36:16.926975 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.926950 2580 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-183.ec2.internal" Jul 21 01:36:16.927094 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:16.926981 2580 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-183.ec2.internal\": node \"ip-10-0-139-183.ec2.internal\" not found" Jul 21 01:36:16.954582 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:16.954551 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-183.ec2.internal\" not found" Jul 21 01:36:16.971230 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.971205 2580 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-139-183.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-183.ec2.internal"] Jul 21 01:36:16.971328 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.971274 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 01:36:16.972290 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.972274 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-183.ec2.internal" event="NodeHasSufficientMemory" Jul 21 01:36:16.972363 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.972303 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-183.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 01:36:16.972363 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.972314 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-183.ec2.internal" event="NodeHasSufficientPID" Jul 21 01:36:16.974645 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.974632 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 01:36:16.974753 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.974740 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-183.ec2.internal" Jul 21 01:36:16.974805 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.974777 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 01:36:16.975703 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.975674 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-183.ec2.internal" event="NodeHasSufficientMemory" Jul 21 01:36:16.975894 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.975709 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-183.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 01:36:16.975894 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.975725 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-183.ec2.internal" event="NodeHasSufficientPID" Jul 21 01:36:16.975894 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.975674 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-183.ec2.internal" event="NodeHasSufficientMemory" Jul 21 01:36:16.975894 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.975811 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-183.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 01:36:16.975894 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.975832 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-183.ec2.internal" event="NodeHasSufficientPID" Jul 21 01:36:16.978068 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.978050 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-183.ec2.internal" Jul 21 01:36:16.978162 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.978080 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 01:36:16.978802 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.978785 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-183.ec2.internal" event="NodeHasSufficientMemory" Jul 21 01:36:16.978894 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.978816 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-183.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 01:36:16.978894 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:16.978830 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-183.ec2.internal" event="NodeHasSufficientPID" Jul 21 01:36:17.001585 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:17.001550 2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-183.ec2.internal\" not found" node="ip-10-0-139-183.ec2.internal" Jul 21 01:36:17.006226 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:17.006207 2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-183.ec2.internal\" not found" node="ip-10-0-139-183.ec2.internal" Jul 21 01:36:17.047085 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:17.047055 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fb35f72ee7f4b217fd778d25ec504876-config\") pod \"kube-apiserver-proxy-ip-10-0-139-183.ec2.internal\" (UID: \"fb35f72ee7f4b217fd778d25ec504876\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-183.ec2.internal" Jul 21 01:36:17.047085 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:17.047086 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b7aa145ad43083f158fe9d117b8cccf9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-183.ec2.internal\" (UID: \"b7aa145ad43083f158fe9d117b8cccf9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-183.ec2.internal" Jul 21 01:36:17.047246 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:17.047106 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7aa145ad43083f158fe9d117b8cccf9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-183.ec2.internal\" (UID: \"b7aa145ad43083f158fe9d117b8cccf9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-183.ec2.internal" Jul 21 01:36:17.055442 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:17.055419 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-183.ec2.internal\" not found" Jul 21 01:36:17.149064 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:17.149031 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b7aa145ad43083f158fe9d117b8cccf9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-183.ec2.internal\" (UID: \"b7aa145ad43083f158fe9d117b8cccf9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-183.ec2.internal" Jul 21 01:36:17.149064 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:17.149051 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b7aa145ad43083f158fe9d117b8cccf9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-183.ec2.internal\" (UID: \"b7aa145ad43083f158fe9d117b8cccf9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-183.ec2.internal" Jul 21 01:36:17.149165 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:17.149086 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7aa145ad43083f158fe9d117b8cccf9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-183.ec2.internal\" (UID: \"b7aa145ad43083f158fe9d117b8cccf9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-183.ec2.internal" Jul 21 01:36:17.149165 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:17.149106 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fb35f72ee7f4b217fd778d25ec504876-config\") pod \"kube-apiserver-proxy-ip-10-0-139-183.ec2.internal\" (UID: \"fb35f72ee7f4b217fd778d25ec504876\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-183.ec2.internal" Jul 21 01:36:17.149165 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:17.149122 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7aa145ad43083f158fe9d117b8cccf9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-183.ec2.internal\" (UID: \"b7aa145ad43083f158fe9d117b8cccf9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-183.ec2.internal" Jul 21 01:36:17.149165 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:17.149145 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fb35f72ee7f4b217fd778d25ec504876-config\") pod \"kube-apiserver-proxy-ip-10-0-139-183.ec2.internal\" (UID: \"fb35f72ee7f4b217fd778d25ec504876\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-183.ec2.internal" Jul 21 01:36:17.156326 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:17.156305 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-183.ec2.internal\" not found" Jul 21 01:36:17.257213 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:17.257150 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-183.ec2.internal\" not found" Jul 21 01:36:17.306051 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:17.306029 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-183.ec2.internal" Jul 21 01:36:17.309565 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:17.309542 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-183.ec2.internal" Jul 21 01:36:17.358320 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:17.358288 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-183.ec2.internal\" not found" Jul 21 01:36:17.459038 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:17.458997 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-183.ec2.internal\" not found" Jul 21 01:36:17.559377 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:17.559285 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-183.ec2.internal\" not found" Jul 21 01:36:17.659831 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:17.659776 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-183.ec2.internal\" not found" Jul 21 01:36:17.687006 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:17.686965 2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 21 01:36:17.687538 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:17.687136 2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 21 01:36:17.747010 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:17.746980 2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 01:36:17.760942 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:17.760911 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-183.ec2.internal\" not found" Jul 21 01:36:17.775281 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:17.775231 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-20 01:31:17 +0000 UTC" deadline="2028-03-04 18:45:14.53929506 +0000 UTC" Jul 21 01:36:17.775281 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:17.775274 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14225h8m56.764024188s" Jul 21 01:36:17.830851 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:17.830772 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 01:36:17.832345 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:17.832324 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-183.ec2.internal" event="NodeHasSufficientMemory" Jul 21 01:36:17.832466 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:17.832359 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-183.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 01:36:17.832466 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:17.832370 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-183.ec2.internal" event="NodeHasSufficientPID" Jul 21 01:36:17.861672 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:17.861635 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-183.ec2.internal\" not found" Jul 21 01:36:17.909508 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:17.909459 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7aa145ad43083f158fe9d117b8cccf9.slice/crio-720120c260d26e3163487207db7cfd53e425359d347cb03aca3eafdae3064a29 WatchSource:0}: Error finding container 720120c260d26e3163487207db7cfd53e425359d347cb03aca3eafdae3064a29: Status 404 returned error can't find the container with id 720120c260d26e3163487207db7cfd53e425359d347cb03aca3eafdae3064a29 Jul 21 01:36:17.915017 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:17.914996 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 01:36:17.921994 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:17.921964 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb35f72ee7f4b217fd778d25ec504876.slice/crio-67f63760309118c1b822e73805d908735ad169ebcb32701b0ea13ea99b7d8c61 WatchSource:0}: Error finding container 67f63760309118c1b822e73805d908735ad169ebcb32701b0ea13ea99b7d8c61: Status 404 returned error can't find the container with id 67f63760309118c1b822e73805d908735ad169ebcb32701b0ea13ea99b7d8c61 Jul 21 01:36:17.950839 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:17.950802 2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 01:36:18.045661 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.045619 2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-183.ec2.internal" Jul 21 01:36:18.056326 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.056294 2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 21 01:36:18.057294 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.057274 2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-183.ec2.internal" Jul 21 01:36:18.069304 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.069278 2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 21 01:36:18.283229 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.283146 2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 01:36:18.724522 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.724421 2580 apiserver.go:52] "Watching apiserver" Jul 21 01:36:18.731894 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.731861 2580 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 21 01:36:18.733034 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.733001 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-smxdt","openshift-cluster-node-tuning-operator/tuned-fm8n4","openshift-image-registry/node-ca-zxcjm","openshift-multus/multus-sz95c","openshift-network-diagnostics/network-check-target-pczgw","openshift-network-operator/iptables-alerter-8688j","openshift-ovn-kubernetes/ovnkube-node-5m9tn","kube-system/global-pull-secret-syncer-24nj2","kube-system/kube-apiserver-proxy-ip-10-0-139-183.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bs276","openshift-dns/node-resolver-5n4nl","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-183.ec2.internal","openshift-multus/multus-additional-cni-plugins-sm5jf","openshift-multus/network-metrics-daemon-vdpxq"] Jul 21 01:36:18.735689 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.735663 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pczgw" Jul 21 01:36:18.735814 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:18.735765 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pczgw" podUID="1527759a-5529-4c49-a9e5-cdc780b0b21a" Jul 21 01:36:18.739026 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.738814 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5m9tn" Jul 21 01:36:18.741752 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.741235 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 21 01:36:18.741752 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.741486 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 21 01:36:18.743059 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.742162 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8688j" Jul 21 01:36:18.743059 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.742339 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-52l9p\"" Jul 21 01:36:18.743059 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.742619 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 21 01:36:18.743059 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.742730 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 21 01:36:18.743059 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.742968 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 21 01:36:18.743059 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.743005 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 21 01:36:18.744416 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.744395 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-smxdt" Jul 21 01:36:18.745275 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.745253 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 21 01:36:18.745372 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.745360 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-lttgh\"" Jul 21 01:36:18.745480 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.745462 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 21 01:36:18.745743 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.745722 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 21 01:36:18.746410 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.746395 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 21 01:36:18.747047 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.746960 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 21 01:36:18.747282 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.747243 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 21 01:36:18.747385 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.747324 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-zdfzr\"" Jul 21 01:36:18.748027 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.747892 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-sz95c" Jul 21 01:36:18.750029 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.750010 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 21 01:36:18.750199 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.750122 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 21 01:36:18.750383 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.750360 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 21 01:36:18.750569 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.750548 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zxcjm" Jul 21 01:36:18.750799 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.750781 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 21 01:36:18.751104 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.751085 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-xbkjd\"" Jul 21 01:36:18.752607 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.752592 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 21 01:36:18.753292 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.753273 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 21 01:36:18.753369 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.753335 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-mlcvp\"" Jul 21 01:36:18.753433 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.753377 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 21 01:36:18.753679 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.753656 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fm8n4" Jul 21 01:36:18.756275 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.756255 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-24nj2" Jul 21 01:36:18.756373 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:18.756349 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-24nj2" podUID="17bdceb8-8016-4d0b-8b3c-f36507279428" Jul 21 01:36:18.756519 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.756496 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-c5k5h\"" Jul 21 01:36:18.756662 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.756641 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 21 01:36:18.757537 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.756974 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 21 01:36:18.759108 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.759081 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/77f8ae4c-96ea-49a3-ac6d-93df2beaa785-os-release\") pod \"multus-sz95c\" (UID: \"77f8ae4c-96ea-49a3-ac6d-93df2beaa785\") " pod="openshift-multus/multus-sz95c" Jul 21 01:36:18.759214 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.759140 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/77f8ae4c-96ea-49a3-ac6d-93df2beaa785-cni-binary-copy\") pod \"multus-sz95c\" (UID: \"77f8ae4c-96ea-49a3-ac6d-93df2beaa785\") " pod="openshift-multus/multus-sz95c" Jul 21 01:36:18.759214 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.759175 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8a8276e8-8761-4775-950c-742e4384c570-host-slash\") pod \"ovnkube-node-5m9tn\" (UID: \"8a8276e8-8761-4775-950c-742e4384c570\") " pod="openshift-ovn-kubernetes/ovnkube-node-5m9tn" Jul 21 01:36:18.759214 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.759208 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8a8276e8-8761-4775-950c-742e4384c570-run-ovn\") pod \"ovnkube-node-5m9tn\" (UID: \"8a8276e8-8761-4775-950c-742e4384c570\") " pod="openshift-ovn-kubernetes/ovnkube-node-5m9tn" Jul 21 01:36:18.759378 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.759278 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8a8276e8-8761-4775-950c-742e4384c570-ovn-node-metrics-cert\") pod \"ovnkube-node-5m9tn\" (UID: \"8a8276e8-8761-4775-950c-742e4384c570\") " pod="openshift-ovn-kubernetes/ovnkube-node-5m9tn" Jul 21 01:36:18.759650 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.759630 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bs276" Jul 21 01:36:18.761619 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.761590 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2b1299d5-1664-4e47-b8af-381bc1a7d7fe-host-slash\") pod \"iptables-alerter-8688j\" (UID: \"2b1299d5-1664-4e47-b8af-381bc1a7d7fe\") " pod="openshift-network-operator/iptables-alerter-8688j" Jul 21 01:36:18.761719 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.761642 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/77f8ae4c-96ea-49a3-ac6d-93df2beaa785-host-var-lib-cni-multus\") pod \"multus-sz95c\" (UID: \"77f8ae4c-96ea-49a3-ac6d-93df2beaa785\") " pod="openshift-multus/multus-sz95c" Jul 21 01:36:18.761719 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.761685 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/77f8ae4c-96ea-49a3-ac6d-93df2beaa785-etc-kubernetes\") pod \"multus-sz95c\" (UID: \"77f8ae4c-96ea-49a3-ac6d-93df2beaa785\") " pod="openshift-multus/multus-sz95c" Jul 21 01:36:18.761833 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.761719 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8a8276e8-8761-4775-950c-742e4384c570-host-run-netns\") pod \"ovnkube-node-5m9tn\" (UID: \"8a8276e8-8761-4775-950c-742e4384c570\") " pod="openshift-ovn-kubernetes/ovnkube-node-5m9tn" Jul 21 01:36:18.761833 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.761751 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e548f5b6-6507-4431-b027-88917b43191f-konnectivity-ca\") pod \"konnectivity-agent-smxdt\" (UID: \"e548f5b6-6507-4431-b027-88917b43191f\") " pod="kube-system/konnectivity-agent-smxdt" Jul 21 01:36:18.761833 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.761780 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/77f8ae4c-96ea-49a3-ac6d-93df2beaa785-system-cni-dir\") pod \"multus-sz95c\" (UID: \"77f8ae4c-96ea-49a3-ac6d-93df2beaa785\") " pod="openshift-multus/multus-sz95c" Jul 21 01:36:18.761833 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.761806 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/77f8ae4c-96ea-49a3-ac6d-93df2beaa785-host-run-multus-certs\") pod \"multus-sz95c\" (UID: \"77f8ae4c-96ea-49a3-ac6d-93df2beaa785\") " pod="openshift-multus/multus-sz95c" Jul 21 01:36:18.762053 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.761840 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlprc\" (UniqueName: \"kubernetes.io/projected/1527759a-5529-4c49-a9e5-cdc780b0b21a-kube-api-access-mlprc\") pod \"network-check-target-pczgw\" (UID: \"1527759a-5529-4c49-a9e5-cdc780b0b21a\") " pod="openshift-network-diagnostics/network-check-target-pczgw" Jul 21 01:36:18.762053 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.761873 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8a8276e8-8761-4775-950c-742e4384c570-systemd-units\") pod \"ovnkube-node-5m9tn\" (UID: \"8a8276e8-8761-4775-950c-742e4384c570\") " pod="openshift-ovn-kubernetes/ovnkube-node-5m9tn" Jul 21 01:36:18.762053 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.761901 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8a8276e8-8761-4775-950c-742e4384c570-log-socket\") pod \"ovnkube-node-5m9tn\" (UID: \"8a8276e8-8761-4775-950c-742e4384c570\") " pod="openshift-ovn-kubernetes/ovnkube-node-5m9tn" Jul 21 01:36:18.762053 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.761981 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8a8276e8-8761-4775-950c-742e4384c570-env-overrides\") pod \"ovnkube-node-5m9tn\" (UID: \"8a8276e8-8761-4775-950c-742e4384c570\") " pod="openshift-ovn-kubernetes/ovnkube-node-5m9tn" Jul 21 01:36:18.762267 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.762019 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvkr6\" (UniqueName: \"kubernetes.io/projected/8a8276e8-8761-4775-950c-742e4384c570-kube-api-access-gvkr6\") pod \"ovnkube-node-5m9tn\" (UID: \"8a8276e8-8761-4775-950c-742e4384c570\") " pod="openshift-ovn-kubernetes/ovnkube-node-5m9tn" Jul 21 01:36:18.762267 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.762116 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/77f8ae4c-96ea-49a3-ac6d-93df2beaa785-multus-conf-dir\") pod \"multus-sz95c\" (UID: \"77f8ae4c-96ea-49a3-ac6d-93df2beaa785\") " pod="openshift-multus/multus-sz95c" Jul 21 01:36:18.762267 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.762179 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8a8276e8-8761-4775-950c-742e4384c570-ovnkube-config\") pod \"ovnkube-node-5m9tn\" (UID: \"8a8276e8-8761-4775-950c-742e4384c570\") " pod="openshift-ovn-kubernetes/ovnkube-node-5m9tn" Jul 21 01:36:18.762267 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.762229 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8a8276e8-8761-4775-950c-742e4384c570-ovnkube-script-lib\") pod \"ovnkube-node-5m9tn\" (UID: \"8a8276e8-8761-4775-950c-742e4384c570\") " pod="openshift-ovn-kubernetes/ovnkube-node-5m9tn" Jul 21 01:36:18.762267 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.762261 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/77f8ae4c-96ea-49a3-ac6d-93df2beaa785-multus-cni-dir\") pod \"multus-sz95c\" (UID: \"77f8ae4c-96ea-49a3-ac6d-93df2beaa785\") " pod="openshift-multus/multus-sz95c" Jul 21 01:36:18.762552 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.762302 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/77f8ae4c-96ea-49a3-ac6d-93df2beaa785-host-run-k8s-cni-cncf-io\") pod \"multus-sz95c\" (UID: \"77f8ae4c-96ea-49a3-ac6d-93df2beaa785\") " pod="openshift-multus/multus-sz95c" Jul 21 01:36:18.762552 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.762334 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a8276e8-8761-4775-950c-742e4384c570-etc-openvswitch\") pod \"ovnkube-node-5m9tn\" (UID: \"8a8276e8-8761-4775-950c-742e4384c570\") " pod="openshift-ovn-kubernetes/ovnkube-node-5m9tn" Jul 21 01:36:18.762552 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.762367 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8a8276e8-8761-4775-950c-742e4384c570-host-cni-netd\") pod \"ovnkube-node-5m9tn\" (UID: \"8a8276e8-8761-4775-950c-742e4384c570\") " pod="openshift-ovn-kubernetes/ovnkube-node-5m9tn" Jul 21 01:36:18.762552 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.762392 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/77f8ae4c-96ea-49a3-ac6d-93df2beaa785-cnibin\") pod \"multus-sz95c\" (UID: \"77f8ae4c-96ea-49a3-ac6d-93df2beaa785\") " pod="openshift-multus/multus-sz95c" Jul 21 01:36:18.762552 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.762441 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/77f8ae4c-96ea-49a3-ac6d-93df2beaa785-host-run-netns\") pod \"multus-sz95c\" (UID: \"77f8ae4c-96ea-49a3-ac6d-93df2beaa785\") " pod="openshift-multus/multus-sz95c" Jul 21 01:36:18.762552 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.762470 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/77f8ae4c-96ea-49a3-ac6d-93df2beaa785-host-var-lib-cni-bin\") pod \"multus-sz95c\" (UID: \"77f8ae4c-96ea-49a3-ac6d-93df2beaa785\") " pod="openshift-multus/multus-sz95c" Jul 21 01:36:18.762552 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.762499 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/77f8ae4c-96ea-49a3-ac6d-93df2beaa785-hostroot\") pod \"multus-sz95c\" (UID: \"77f8ae4c-96ea-49a3-ac6d-93df2beaa785\") " pod="openshift-multus/multus-sz95c" Jul 21 01:36:18.762913 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.762569 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/77f8ae4c-96ea-49a3-ac6d-93df2beaa785-multus-daemon-config\") pod \"multus-sz95c\" (UID: \"77f8ae4c-96ea-49a3-ac6d-93df2beaa785\") " pod="openshift-multus/multus-sz95c" Jul 21 01:36:18.762913 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.762603 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a8276e8-8761-4775-950c-742e4384c570-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5m9tn\" (UID: \"8a8276e8-8761-4775-950c-742e4384c570\") " pod="openshift-ovn-kubernetes/ovnkube-node-5m9tn" Jul 21 01:36:18.762913 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.762630 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zmrl\" (UniqueName: \"kubernetes.io/projected/2b1299d5-1664-4e47-b8af-381bc1a7d7fe-kube-api-access-6zmrl\") pod \"iptables-alerter-8688j\" (UID: \"2b1299d5-1664-4e47-b8af-381bc1a7d7fe\") " pod="openshift-network-operator/iptables-alerter-8688j" Jul 21 01:36:18.762913 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.762737 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e548f5b6-6507-4431-b027-88917b43191f-agent-certs\") pod \"konnectivity-agent-smxdt\" (UID: \"e548f5b6-6507-4431-b027-88917b43191f\") " pod="kube-system/konnectivity-agent-smxdt" Jul 21 01:36:18.762913 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.762784 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/77f8ae4c-96ea-49a3-ac6d-93df2beaa785-multus-socket-dir-parent\") pod \"multus-sz95c\" (UID: \"77f8ae4c-96ea-49a3-ac6d-93df2beaa785\") " pod="openshift-multus/multus-sz95c" Jul 21 01:36:18.762913 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.762812 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/77f8ae4c-96ea-49a3-ac6d-93df2beaa785-host-var-lib-kubelet\") pod \"multus-sz95c\" (UID: \"77f8ae4c-96ea-49a3-ac6d-93df2beaa785\") " pod="openshift-multus/multus-sz95c" Jul 21 01:36:18.762913 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.762873 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d79t9\" (UniqueName: \"kubernetes.io/projected/77f8ae4c-96ea-49a3-ac6d-93df2beaa785-kube-api-access-d79t9\") pod \"multus-sz95c\" (UID: \"77f8ae4c-96ea-49a3-ac6d-93df2beaa785\") " pod="openshift-multus/multus-sz95c" Jul 21 01:36:18.762913 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.762900 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2b1299d5-1664-4e47-b8af-381bc1a7d7fe-iptables-alerter-script\") pod \"iptables-alerter-8688j\" (UID: \"2b1299d5-1664-4e47-b8af-381bc1a7d7fe\") " pod="openshift-network-operator/iptables-alerter-8688j" Jul 21 01:36:18.763332 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.762941 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a8276e8-8761-4775-950c-742e4384c570-host-kubelet\") pod \"ovnkube-node-5m9tn\" (UID: \"8a8276e8-8761-4775-950c-742e4384c570\") " pod="openshift-ovn-kubernetes/ovnkube-node-5m9tn" Jul 21 01:36:18.763332 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.763002 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a8276e8-8761-4775-950c-742e4384c570-var-lib-openvswitch\") pod \"ovnkube-node-5m9tn\" (UID: \"8a8276e8-8761-4775-950c-742e4384c570\") " pod="openshift-ovn-kubernetes/ovnkube-node-5m9tn" Jul 21 01:36:18.763332 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.763033 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a8276e8-8761-4775-950c-742e4384c570-run-openvswitch\") pod \"ovnkube-node-5m9tn\" (UID: \"8a8276e8-8761-4775-950c-742e4384c570\") " pod="openshift-ovn-kubernetes/ovnkube-node-5m9tn" Jul 21 01:36:18.763332 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.763068 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8a8276e8-8761-4775-950c-742e4384c570-node-log\") pod \"ovnkube-node-5m9tn\" (UID: \"8a8276e8-8761-4775-950c-742e4384c570\") " pod="openshift-ovn-kubernetes/ovnkube-node-5m9tn" Jul 21 01:36:18.763332 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.763095 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a8276e8-8761-4775-950c-742e4384c570-host-run-ovn-kubernetes\") pod \"ovnkube-node-5m9tn\" (UID: \"8a8276e8-8761-4775-950c-742e4384c570\") " pod="openshift-ovn-kubernetes/ovnkube-node-5m9tn" Jul 21 01:36:18.763332 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.763152 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8a8276e8-8761-4775-950c-742e4384c570-host-cni-bin\") pod \"ovnkube-node-5m9tn\" (UID: \"8a8276e8-8761-4775-950c-742e4384c570\") " pod="openshift-ovn-kubernetes/ovnkube-node-5m9tn" Jul 21 01:36:18.763332 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.763220 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8a8276e8-8761-4775-950c-742e4384c570-run-systemd\") pod \"ovnkube-node-5m9tn\" (UID: \"8a8276e8-8761-4775-950c-742e4384c570\") " pod="openshift-ovn-kubernetes/ovnkube-node-5m9tn" Jul 21 01:36:18.764702 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.764679 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5n4nl" Jul 21 01:36:18.765018 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.764993 2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 21 01:36:18.765090 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.765053 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 21 01:36:18.768199 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.768148 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 21 01:36:18.768276 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.768196 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-p5lx4\"" Jul 21 01:36:18.768670 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.768441 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 21 01:36:18.768670 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.768503 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 21 01:36:18.768670 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.768570 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 21 01:36:18.769728 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.769567 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-sm5jf" Jul 21 01:36:18.770192 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.770169 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-z8wv2\"" Jul 21 01:36:18.771556 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.771531 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 21 01:36:18.772136 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.772119 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 21 01:36:18.772230 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.772183 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-g855s\"" Jul 21 01:36:18.772799 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.772780 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vdpxq" Jul 21 01:36:18.772880 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:18.772860 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vdpxq" podUID="7346c638-55dd-4d75-9d69-cead4209e04c" Jul 21 01:36:18.786434 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.786404 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-5pr77" Jul 21 01:36:18.796643 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.796624 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-5pr77" Jul 21 01:36:18.846879 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.846851 2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 21 01:36:18.863630 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.863597 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a8276e8-8761-4775-950c-742e4384c570-host-kubelet\") pod \"ovnkube-node-5m9tn\" (UID: \"8a8276e8-8761-4775-950c-742e4384c570\") " pod="openshift-ovn-kubernetes/ovnkube-node-5m9tn" Jul 21 01:36:18.863823 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.863637 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a8276e8-8761-4775-950c-742e4384c570-var-lib-openvswitch\") pod \"ovnkube-node-5m9tn\" (UID: \"8a8276e8-8761-4775-950c-742e4384c570\") " pod="openshift-ovn-kubernetes/ovnkube-node-5m9tn" Jul 21 01:36:18.863823 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.863667 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a8276e8-8761-4775-950c-742e4384c570-run-openvswitch\") pod \"ovnkube-node-5m9tn\" (UID: \"8a8276e8-8761-4775-950c-742e4384c570\") " pod="openshift-ovn-kubernetes/ovnkube-node-5m9tn" Jul 21 01:36:18.863823 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.863723 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a8276e8-8761-4775-950c-742e4384c570-run-openvswitch\") pod \"ovnkube-node-5m9tn\" (UID: \"8a8276e8-8761-4775-950c-742e4384c570\") " pod="openshift-ovn-kubernetes/ovnkube-node-5m9tn" Jul 21 01:36:18.863823 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.863733 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a8276e8-8761-4775-950c-742e4384c570-host-kubelet\") pod \"ovnkube-node-5m9tn\" (UID: \"8a8276e8-8761-4775-950c-742e4384c570\") " pod="openshift-ovn-kubernetes/ovnkube-node-5m9tn" Jul 21 01:36:18.863823 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.863774 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a8276e8-8761-4775-950c-742e4384c570-var-lib-openvswitch\") pod \"ovnkube-node-5m9tn\" (UID: \"8a8276e8-8761-4775-950c-742e4384c570\") " pod="openshift-ovn-kubernetes/ovnkube-node-5m9tn" Jul 21 01:36:18.863823 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.863784 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8a8276e8-8761-4775-950c-742e4384c570-node-log\") pod \"ovnkube-node-5m9tn\" (UID: \"8a8276e8-8761-4775-950c-742e4384c570\") " pod="openshift-ovn-kubernetes/ovnkube-node-5m9tn" Jul 21 01:36:18.863823 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.863806 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8a8276e8-8761-4775-950c-742e4384c570-host-cni-bin\") pod \"ovnkube-node-5m9tn\" (UID: \"8a8276e8-8761-4775-950c-742e4384c570\") " pod="openshift-ovn-kubernetes/ovnkube-node-5m9tn" Jul 21 01:36:18.864131 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.863836 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-st25h\" (UniqueName: \"kubernetes.io/projected/a1ca57f9-11f8-48c3-9ff6-a7b2561fede0-kube-api-access-st25h\") pod \"node-resolver-5n4nl\" (UID: \"a1ca57f9-11f8-48c3-9ff6-a7b2561fede0\") " pod="openshift-dns/node-resolver-5n4nl" Jul 21 01:36:18.864131 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.863868 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8a8276e8-8761-4775-950c-742e4384c570-node-log\") pod \"ovnkube-node-5m9tn\" (UID: \"8a8276e8-8761-4775-950c-742e4384c570\") " pod="openshift-ovn-kubernetes/ovnkube-node-5m9tn" Jul 21 01:36:18.864131 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.863871 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1489d404-17a6-4bf1-9a85-9d3787aece54-host\") pod \"node-ca-zxcjm\" (UID: \"1489d404-17a6-4bf1-9a85-9d3787aece54\") " pod="openshift-image-registry/node-ca-zxcjm" Jul 21 01:36:18.864131 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.863897 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8a8276e8-8761-4775-950c-742e4384c570-host-cni-bin\") pod \"ovnkube-node-5m9tn\" (UID: \"8a8276e8-8761-4775-950c-742e4384c570\") " pod="openshift-ovn-kubernetes/ovnkube-node-5m9tn" Jul 21 01:36:18.864131 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.863917 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e9b5af18-33c8-4f64-bf52-548a457165ec-var-lib-kubelet\") pod \"tuned-fm8n4\" (UID: \"e9b5af18-33c8-4f64-bf52-548a457165ec\") " pod="openshift-cluster-node-tuning-operator/tuned-fm8n4" Jul 21 01:36:18.864131 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.863946 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8a8276e8-8761-4775-950c-742e4384c570-run-systemd\") pod \"ovnkube-node-5m9tn\" (UID: \"8a8276e8-8761-4775-950c-742e4384c570\") " pod="openshift-ovn-kubernetes/ovnkube-node-5m9tn" Jul 21 01:36:18.864131 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.863974 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/77f8ae4c-96ea-49a3-ac6d-93df2beaa785-os-release\") pod \"multus-sz95c\" (UID: \"77f8ae4c-96ea-49a3-ac6d-93df2beaa785\") " pod="openshift-multus/multus-sz95c" Jul 21 01:36:18.864131 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.864000 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1ca57f9-11f8-48c3-9ff6-a7b2561fede0-tmp-dir\") pod \"node-resolver-5n4nl\" (UID: \"a1ca57f9-11f8-48c3-9ff6-a7b2561fede0\") " pod="openshift-dns/node-resolver-5n4nl" Jul 21 01:36:18.864131 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.864021 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e9b5af18-33c8-4f64-bf52-548a457165ec-lib-modules\") pod \"tuned-fm8n4\" (UID: \"e9b5af18-33c8-4f64-bf52-548a457165ec\") " pod="openshift-cluster-node-tuning-operator/tuned-fm8n4" Jul 21 01:36:18.864131 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.864044 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0a21e01e-3c16-429c-85bb-df4c0e096756-registration-dir\") pod \"aws-ebs-csi-driver-node-bs276\" (UID: \"0a21e01e-3c16-429c-85bb-df4c0e096756\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bs276" Jul 21 01:36:18.864131 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.864069 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0a21e01e-3c16-429c-85bb-df4c0e096756-etc-selinux\") pod \"aws-ebs-csi-driver-node-bs276\" (UID: \"0a21e01e-3c16-429c-85bb-df4c0e096756\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bs276" Jul 21 01:36:18.864131 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.864094 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx2d7\" (UniqueName: \"kubernetes.io/projected/0a21e01e-3c16-429c-85bb-df4c0e096756-kube-api-access-dx2d7\") pod \"aws-ebs-csi-driver-node-bs276\" (UID: \"0a21e01e-3c16-429c-85bb-df4c0e096756\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bs276" Jul 21 01:36:18.864131 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.864116 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/64ab7299-1b1f-4d7e-8c26-998adf4499db-system-cni-dir\") pod \"multus-additional-cni-plugins-sm5jf\" (UID: \"64ab7299-1b1f-4d7e-8c26-998adf4499db\") " pod="openshift-multus/multus-additional-cni-plugins-sm5jf" Jul 21 01:36:18.864774 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.864149 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/77f8ae4c-96ea-49a3-ac6d-93df2beaa785-os-release\") pod \"multus-sz95c\" (UID: \"77f8ae4c-96ea-49a3-ac6d-93df2beaa785\") " pod="openshift-multus/multus-sz95c" Jul 21 01:36:18.864774 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.864145 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8a8276e8-8761-4775-950c-742e4384c570-ovn-node-metrics-cert\") pod \"ovnkube-node-5m9tn\" (UID: \"8a8276e8-8761-4775-950c-742e4384c570\") " pod="openshift-ovn-kubernetes/ovnkube-node-5m9tn" Jul 21 01:36:18.864774 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.864068 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8a8276e8-8761-4775-950c-742e4384c570-run-systemd\") pod \"ovnkube-node-5m9tn\" (UID: \"8a8276e8-8761-4775-950c-742e4384c570\") " pod="openshift-ovn-kubernetes/ovnkube-node-5m9tn" Jul 21 01:36:18.864774 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.864212 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2b1299d5-1664-4e47-b8af-381bc1a7d7fe-host-slash\") pod \"iptables-alerter-8688j\" (UID: \"2b1299d5-1664-4e47-b8af-381bc1a7d7fe\") " pod="openshift-network-operator/iptables-alerter-8688j" Jul 21 01:36:18.864774 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.864346 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/77f8ae4c-96ea-49a3-ac6d-93df2beaa785-etc-kubernetes\") pod \"multus-sz95c\" (UID: \"77f8ae4c-96ea-49a3-ac6d-93df2beaa785\") " pod="openshift-multus/multus-sz95c" Jul 21 01:36:18.864774 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.864355 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2b1299d5-1664-4e47-b8af-381bc1a7d7fe-host-slash\") pod \"iptables-alerter-8688j\" (UID: \"2b1299d5-1664-4e47-b8af-381bc1a7d7fe\") " pod="openshift-network-operator/iptables-alerter-8688j" Jul 21 01:36:18.864774 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.864396 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e548f5b6-6507-4431-b027-88917b43191f-konnectivity-ca\") pod \"konnectivity-agent-smxdt\" (UID: \"e548f5b6-6507-4431-b027-88917b43191f\") " pod="kube-system/konnectivity-agent-smxdt" Jul 21 01:36:18.864774 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.864439 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlprc\" (UniqueName: \"kubernetes.io/projected/1527759a-5529-4c49-a9e5-cdc780b0b21a-kube-api-access-mlprc\") pod \"network-check-target-pczgw\" (UID: \"1527759a-5529-4c49-a9e5-cdc780b0b21a\") " pod="openshift-network-diagnostics/network-check-target-pczgw" Jul 21 01:36:18.864774 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.864507 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8a8276e8-8761-4775-950c-742e4384c570-log-socket\") pod \"ovnkube-node-5m9tn\" (UID: \"8a8276e8-8761-4775-950c-742e4384c570\") " pod="openshift-ovn-kubernetes/ovnkube-node-5m9tn" Jul 21 01:36:18.864774 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.864592 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8a8276e8-8761-4775-950c-742e4384c570-log-socket\") pod \"ovnkube-node-5m9tn\" (UID: \"8a8276e8-8761-4775-950c-742e4384c570\") " pod="openshift-ovn-kubernetes/ovnkube-node-5m9tn" Jul 21 01:36:18.864774 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.864604 2580 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 21 01:36:18.864774 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.864668 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8a8276e8-8761-4775-950c-742e4384c570-env-overrides\") pod \"ovnkube-node-5m9tn\" (UID: \"8a8276e8-8761-4775-950c-742e4384c570\") " pod="openshift-ovn-kubernetes/ovnkube-node-5m9tn" Jul 21 01:36:18.865424 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.864927 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvkr6\" (UniqueName: \"kubernetes.io/projected/8a8276e8-8761-4775-950c-742e4384c570-kube-api-access-gvkr6\") pod \"ovnkube-node-5m9tn\" (UID: \"8a8276e8-8761-4775-950c-742e4384c570\") " pod="openshift-ovn-kubernetes/ovnkube-node-5m9tn" Jul 21 01:36:18.865424 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.864964 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/17bdceb8-8016-4d0b-8b3c-f36507279428-dbus\") pod \"global-pull-secret-syncer-24nj2\" (UID: \"17bdceb8-8016-4d0b-8b3c-f36507279428\") " pod="kube-system/global-pull-secret-syncer-24nj2" Jul 21 01:36:18.865424 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.864992 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e9b5af18-33c8-4f64-bf52-548a457165ec-tmp\") pod \"tuned-fm8n4\" (UID: \"e9b5af18-33c8-4f64-bf52-548a457165ec\") " pod="openshift-cluster-node-tuning-operator/tuned-fm8n4" Jul 21 01:36:18.865424 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.865026 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx7sr\" (UniqueName: \"kubernetes.io/projected/64ab7299-1b1f-4d7e-8c26-998adf4499db-kube-api-access-vx7sr\") pod \"multus-additional-cni-plugins-sm5jf\" (UID: \"64ab7299-1b1f-4d7e-8c26-998adf4499db\") " pod="openshift-multus/multus-additional-cni-plugins-sm5jf" Jul 21 01:36:18.865424 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.865071 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8a8276e8-8761-4775-950c-742e4384c570-ovnkube-config\") pod \"ovnkube-node-5m9tn\" (UID: \"8a8276e8-8761-4775-950c-742e4384c570\") " pod="openshift-ovn-kubernetes/ovnkube-node-5m9tn" Jul 21 01:36:18.865424 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.865098 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8a8276e8-8761-4775-950c-742e4384c570-ovnkube-script-lib\") pod \"ovnkube-node-5m9tn\" (UID: \"8a8276e8-8761-4775-950c-742e4384c570\") " pod="openshift-ovn-kubernetes/ovnkube-node-5m9tn" Jul 21 01:36:18.865424 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.865125 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/77f8ae4c-96ea-49a3-ac6d-93df2beaa785-multus-cni-dir\") pod \"multus-sz95c\" (UID: \"77f8ae4c-96ea-49a3-ac6d-93df2beaa785\") " pod="openshift-multus/multus-sz95c" Jul 21 01:36:18.865424 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.865139 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e548f5b6-6507-4431-b027-88917b43191f-konnectivity-ca\") pod \"konnectivity-agent-smxdt\" (UID: \"e548f5b6-6507-4431-b027-88917b43191f\") " pod="kube-system/konnectivity-agent-smxdt" Jul 21 01:36:18.865424 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.865151 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e9b5af18-33c8-4f64-bf52-548a457165ec-run\") pod \"tuned-fm8n4\" (UID: \"e9b5af18-33c8-4f64-bf52-548a457165ec\") " pod="openshift-cluster-node-tuning-operator/tuned-fm8n4" Jul 21 01:36:18.865424 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.865179 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a8276e8-8761-4775-950c-742e4384c570-etc-openvswitch\") pod \"ovnkube-node-5m9tn\" (UID: \"8a8276e8-8761-4775-950c-742e4384c570\") " pod="openshift-ovn-kubernetes/ovnkube-node-5m9tn" Jul 21 01:36:18.865424 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.865197 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8a8276e8-8761-4775-950c-742e4384c570-env-overrides\") pod \"ovnkube-node-5m9tn\" (UID: \"8a8276e8-8761-4775-950c-742e4384c570\") " pod="openshift-ovn-kubernetes/ovnkube-node-5m9tn" Jul 21 01:36:18.865424 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.865214 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8a8276e8-8761-4775-950c-742e4384c570-host-cni-netd\") pod \"ovnkube-node-5m9tn\" (UID: \"8a8276e8-8761-4775-950c-742e4384c570\") " pod="openshift-ovn-kubernetes/ovnkube-node-5m9tn" Jul 21 01:36:18.865424 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.865241 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/77f8ae4c-96ea-49a3-ac6d-93df2beaa785-cnibin\") pod \"multus-sz95c\" (UID: \"77f8ae4c-96ea-49a3-ac6d-93df2beaa785\") " pod="openshift-multus/multus-sz95c" Jul 21 01:36:18.865424 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.865266 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/77f8ae4c-96ea-49a3-ac6d-93df2beaa785-host-var-lib-cni-bin\") pod \"multus-sz95c\" (UID: \"77f8ae4c-96ea-49a3-ac6d-93df2beaa785\") " pod="openshift-multus/multus-sz95c" Jul 21 01:36:18.865424 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.865315 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/77f8ae4c-96ea-49a3-ac6d-93df2beaa785-hostroot\") pod \"multus-sz95c\" (UID: \"77f8ae4c-96ea-49a3-ac6d-93df2beaa785\") " pod="openshift-multus/multus-sz95c" Jul 21 01:36:18.865424 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.865340 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/77f8ae4c-96ea-49a3-ac6d-93df2beaa785-multus-daemon-config\") pod \"multus-sz95c\" (UID: \"77f8ae4c-96ea-49a3-ac6d-93df2beaa785\") " pod="openshift-multus/multus-sz95c" Jul 21 01:36:18.865424 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.865368 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chjdn\" (UniqueName: \"kubernetes.io/projected/7346c638-55dd-4d75-9d69-cead4209e04c-kube-api-access-chjdn\") pod \"network-metrics-daemon-vdpxq\" (UID: \"7346c638-55dd-4d75-9d69-cead4209e04c\") " pod="openshift-multus/network-metrics-daemon-vdpxq" Jul 21 01:36:18.866345 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.865394 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e9b5af18-33c8-4f64-bf52-548a457165ec-host\") pod \"tuned-fm8n4\" (UID: \"e9b5af18-33c8-4f64-bf52-548a457165ec\") " pod="openshift-cluster-node-tuning-operator/tuned-fm8n4" Jul 21 01:36:18.866345 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.865417 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/77f8ae4c-96ea-49a3-ac6d-93df2beaa785-multus-cni-dir\") pod \"multus-sz95c\" (UID: \"77f8ae4c-96ea-49a3-ac6d-93df2beaa785\") " pod="openshift-multus/multus-sz95c" Jul 21 01:36:18.866345 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.865475 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a8276e8-8761-4775-950c-742e4384c570-etc-openvswitch\") pod \"ovnkube-node-5m9tn\" (UID: \"8a8276e8-8761-4775-950c-742e4384c570\") " pod="openshift-ovn-kubernetes/ovnkube-node-5m9tn" Jul 21 01:36:18.866345 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.865495 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a8276e8-8761-4775-950c-742e4384c570-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5m9tn\" (UID: \"8a8276e8-8761-4775-950c-742e4384c570\") " pod="openshift-ovn-kubernetes/ovnkube-node-5m9tn" Jul 21 01:36:18.866345 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.865552 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8a8276e8-8761-4775-950c-742e4384c570-host-cni-netd\") pod \"ovnkube-node-5m9tn\" (UID: \"8a8276e8-8761-4775-950c-742e4384c570\") " pod="openshift-ovn-kubernetes/ovnkube-node-5m9tn" Jul 21 01:36:18.866345 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.865564 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zmrl\" (UniqueName: \"kubernetes.io/projected/2b1299d5-1664-4e47-b8af-381bc1a7d7fe-kube-api-access-6zmrl\") pod \"iptables-alerter-8688j\" (UID: \"2b1299d5-1664-4e47-b8af-381bc1a7d7fe\") " pod="openshift-network-operator/iptables-alerter-8688j" Jul 21 01:36:18.866345 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.865595 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e548f5b6-6507-4431-b027-88917b43191f-agent-certs\") pod \"konnectivity-agent-smxdt\" (UID: \"e548f5b6-6507-4431-b027-88917b43191f\") " pod="kube-system/konnectivity-agent-smxdt" Jul 21 01:36:18.866345 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.865601 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/77f8ae4c-96ea-49a3-ac6d-93df2beaa785-hostroot\") pod \"multus-sz95c\" (UID: \"77f8ae4c-96ea-49a3-ac6d-93df2beaa785\") " pod="openshift-multus/multus-sz95c" Jul 21 01:36:18.866345 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.865680 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d79t9\" (UniqueName: \"kubernetes.io/projected/77f8ae4c-96ea-49a3-ac6d-93df2beaa785-kube-api-access-d79t9\") pod \"multus-sz95c\" (UID: \"77f8ae4c-96ea-49a3-ac6d-93df2beaa785\") " pod="openshift-multus/multus-sz95c" Jul 21 01:36:18.866345 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.865687 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a8276e8-8761-4775-950c-742e4384c570-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5m9tn\" (UID: \"8a8276e8-8761-4775-950c-742e4384c570\") " pod="openshift-ovn-kubernetes/ovnkube-node-5m9tn" Jul 21 01:36:18.866345 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.865711 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0a21e01e-3c16-429c-85bb-df4c0e096756-device-dir\") pod \"aws-ebs-csi-driver-node-bs276\" (UID: \"0a21e01e-3c16-429c-85bb-df4c0e096756\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bs276" Jul 21 01:36:18.866345 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.865740 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2b1299d5-1664-4e47-b8af-381bc1a7d7fe-iptables-alerter-script\") pod \"iptables-alerter-8688j\" (UID: \"2b1299d5-1664-4e47-b8af-381bc1a7d7fe\") " pod="openshift-network-operator/iptables-alerter-8688j" Jul 21 01:36:18.866345 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.865781 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a8276e8-8761-4775-950c-742e4384c570-host-run-ovn-kubernetes\") pod \"ovnkube-node-5m9tn\" (UID: \"8a8276e8-8761-4775-950c-742e4384c570\") " pod="openshift-ovn-kubernetes/ovnkube-node-5m9tn" Jul 21 01:36:18.866345 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.864630 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/77f8ae4c-96ea-49a3-ac6d-93df2beaa785-etc-kubernetes\") pod \"multus-sz95c\" (UID: \"77f8ae4c-96ea-49a3-ac6d-93df2beaa785\") " pod="openshift-multus/multus-sz95c" Jul 21 01:36:18.866345 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.865887 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74pgb\" (UniqueName: \"kubernetes.io/projected/1489d404-17a6-4bf1-9a85-9d3787aece54-kube-api-access-74pgb\") pod \"node-ca-zxcjm\" (UID: \"1489d404-17a6-4bf1-9a85-9d3787aece54\") " pod="openshift-image-registry/node-ca-zxcjm" Jul 21 01:36:18.866345 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.865936 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e9b5af18-33c8-4f64-bf52-548a457165ec-etc-sysconfig\") pod \"tuned-fm8n4\" (UID: \"e9b5af18-33c8-4f64-bf52-548a457165ec\") " pod="openshift-cluster-node-tuning-operator/tuned-fm8n4" Jul 21 01:36:18.866345 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.865961 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e9b5af18-33c8-4f64-bf52-548a457165ec-etc-systemd\") pod \"tuned-fm8n4\" (UID: \"e9b5af18-33c8-4f64-bf52-548a457165ec\") " pod="openshift-cluster-node-tuning-operator/tuned-fm8n4" Jul 21 01:36:18.867221 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.866002 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e9b5af18-33c8-4f64-bf52-548a457165ec-etc-tuned\") pod \"tuned-fm8n4\" (UID: \"e9b5af18-33c8-4f64-bf52-548a457165ec\") " pod="openshift-cluster-node-tuning-operator/tuned-fm8n4" Jul 21 01:36:18.867221 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.866022 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8a8276e8-8761-4775-950c-742e4384c570-ovnkube-config\") pod \"ovnkube-node-5m9tn\" (UID: \"8a8276e8-8761-4775-950c-742e4384c570\") " pod="openshift-ovn-kubernetes/ovnkube-node-5m9tn" Jul 21 01:36:18.867221 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.866029 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qqbs\" (UniqueName: \"kubernetes.io/projected/e9b5af18-33c8-4f64-bf52-548a457165ec-kube-api-access-8qqbs\") pod \"tuned-fm8n4\" (UID: \"e9b5af18-33c8-4f64-bf52-548a457165ec\") " pod="openshift-cluster-node-tuning-operator/tuned-fm8n4" Jul 21 01:36:18.867221 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.866081 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/77f8ae4c-96ea-49a3-ac6d-93df2beaa785-cni-binary-copy\") pod \"multus-sz95c\" (UID: \"77f8ae4c-96ea-49a3-ac6d-93df2beaa785\") " pod="openshift-multus/multus-sz95c" Jul 21 01:36:18.867221 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.866125 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a1ca57f9-11f8-48c3-9ff6-a7b2561fede0-hosts-file\") pod \"node-resolver-5n4nl\" (UID: \"a1ca57f9-11f8-48c3-9ff6-a7b2561fede0\") " pod="openshift-dns/node-resolver-5n4nl" Jul 21 01:36:18.867221 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.866144 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/77f8ae4c-96ea-49a3-ac6d-93df2beaa785-cnibin\") pod \"multus-sz95c\" (UID: \"77f8ae4c-96ea-49a3-ac6d-93df2beaa785\") " pod="openshift-multus/multus-sz95c" Jul 21 01:36:18.867221 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.866151 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1489d404-17a6-4bf1-9a85-9d3787aece54-serviceca\") pod \"node-ca-zxcjm\" (UID: \"1489d404-17a6-4bf1-9a85-9d3787aece54\") " pod="openshift-image-registry/node-ca-zxcjm" Jul 21 01:36:18.867221 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.866191 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8a8276e8-8761-4775-950c-742e4384c570-host-slash\") pod \"ovnkube-node-5m9tn\" (UID: \"8a8276e8-8761-4775-950c-742e4384c570\") " pod="openshift-ovn-kubernetes/ovnkube-node-5m9tn" Jul 21 01:36:18.867221 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.866256 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8a8276e8-8761-4775-950c-742e4384c570-run-ovn\") pod \"ovnkube-node-5m9tn\" (UID: \"8a8276e8-8761-4775-950c-742e4384c570\") " pod="openshift-ovn-kubernetes/ovnkube-node-5m9tn" Jul 21 01:36:18.867221 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.866284 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/77f8ae4c-96ea-49a3-ac6d-93df2beaa785-host-var-lib-cni-multus\") pod \"multus-sz95c\" (UID: \"77f8ae4c-96ea-49a3-ac6d-93df2beaa785\") " pod="openshift-multus/multus-sz95c" Jul 21 01:36:18.867221 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.866303 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/77f8ae4c-96ea-49a3-ac6d-93df2beaa785-host-var-lib-cni-bin\") pod \"multus-sz95c\" (UID: \"77f8ae4c-96ea-49a3-ac6d-93df2beaa785\") " pod="openshift-multus/multus-sz95c" Jul 21 01:36:18.867221 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.866309 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/77f8ae4c-96ea-49a3-ac6d-93df2beaa785-multus-daemon-config\") pod \"multus-sz95c\" (UID: \"77f8ae4c-96ea-49a3-ac6d-93df2beaa785\") " pod="openshift-multus/multus-sz95c" Jul 21 01:36:18.867221 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.866315 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/17bdceb8-8016-4d0b-8b3c-f36507279428-kubelet-config\") pod \"global-pull-secret-syncer-24nj2\" (UID: \"17bdceb8-8016-4d0b-8b3c-f36507279428\") " pod="kube-system/global-pull-secret-syncer-24nj2" Jul 21 01:36:18.867221 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.866354 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8a8276e8-8761-4775-950c-742e4384c570-host-slash\") pod \"ovnkube-node-5m9tn\" (UID: \"8a8276e8-8761-4775-950c-742e4384c570\") " pod="openshift-ovn-kubernetes/ovnkube-node-5m9tn" Jul 21 01:36:18.867221 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.866361 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e9b5af18-33c8-4f64-bf52-548a457165ec-etc-kubernetes\") pod \"tuned-fm8n4\" (UID: \"e9b5af18-33c8-4f64-bf52-548a457165ec\") " pod="openshift-cluster-node-tuning-operator/tuned-fm8n4" Jul 21 01:36:18.867221 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.866401 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8a8276e8-8761-4775-950c-742e4384c570-run-ovn\") pod \"ovnkube-node-5m9tn\" (UID: \"8a8276e8-8761-4775-950c-742e4384c570\") " pod="openshift-ovn-kubernetes/ovnkube-node-5m9tn" Jul 21 01:36:18.867221 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.866456 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e9b5af18-33c8-4f64-bf52-548a457165ec-sys\") pod \"tuned-fm8n4\" (UID: \"e9b5af18-33c8-4f64-bf52-548a457165ec\") " pod="openshift-cluster-node-tuning-operator/tuned-fm8n4" Jul 21 01:36:18.867221 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.866460 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a8276e8-8761-4775-950c-742e4384c570-host-run-ovn-kubernetes\") pod \"ovnkube-node-5m9tn\" (UID: \"8a8276e8-8761-4775-950c-742e4384c570\") " pod="openshift-ovn-kubernetes/ovnkube-node-5m9tn" Jul 21 01:36:18.868140 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.866580 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/64ab7299-1b1f-4d7e-8c26-998adf4499db-os-release\") pod \"multus-additional-cni-plugins-sm5jf\" (UID: \"64ab7299-1b1f-4d7e-8c26-998adf4499db\") " pod="openshift-multus/multus-additional-cni-plugins-sm5jf" Jul 21 01:36:18.868140 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.866619 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/77f8ae4c-96ea-49a3-ac6d-93df2beaa785-host-var-lib-cni-multus\") pod \"multus-sz95c\" (UID: \"77f8ae4c-96ea-49a3-ac6d-93df2beaa785\") " pod="openshift-multus/multus-sz95c" Jul 21 01:36:18.868140 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.866693 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/64ab7299-1b1f-4d7e-8c26-998adf4499db-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sm5jf\" (UID: \"64ab7299-1b1f-4d7e-8c26-998adf4499db\") " pod="openshift-multus/multus-additional-cni-plugins-sm5jf" Jul 21 01:36:18.868140 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.866738 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8a8276e8-8761-4775-950c-742e4384c570-host-run-netns\") pod \"ovnkube-node-5m9tn\" (UID: \"8a8276e8-8761-4775-950c-742e4384c570\") " pod="openshift-ovn-kubernetes/ovnkube-node-5m9tn" Jul 21 01:36:18.868140 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.866787 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8a8276e8-8761-4775-950c-742e4384c570-host-run-netns\") pod \"ovnkube-node-5m9tn\" (UID: \"8a8276e8-8761-4775-950c-742e4384c570\") " pod="openshift-ovn-kubernetes/ovnkube-node-5m9tn" Jul 21 01:36:18.868140 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.866808 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/77f8ae4c-96ea-49a3-ac6d-93df2beaa785-system-cni-dir\") pod \"multus-sz95c\" (UID: \"77f8ae4c-96ea-49a3-ac6d-93df2beaa785\") " pod="openshift-multus/multus-sz95c" Jul 21 01:36:18.868140 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.866846 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/77f8ae4c-96ea-49a3-ac6d-93df2beaa785-cni-binary-copy\") pod \"multus-sz95c\" (UID: \"77f8ae4c-96ea-49a3-ac6d-93df2beaa785\") " pod="openshift-multus/multus-sz95c" Jul 21 01:36:18.868140 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.866875 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/77f8ae4c-96ea-49a3-ac6d-93df2beaa785-host-run-multus-certs\") pod \"multus-sz95c\" (UID: \"77f8ae4c-96ea-49a3-ac6d-93df2beaa785\") " pod="openshift-multus/multus-sz95c" Jul 21 01:36:18.868140 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.866915 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/77f8ae4c-96ea-49a3-ac6d-93df2beaa785-system-cni-dir\") pod \"multus-sz95c\" (UID: \"77f8ae4c-96ea-49a3-ac6d-93df2beaa785\") " pod="openshift-multus/multus-sz95c" Jul 21 01:36:18.868140 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.866959 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e9b5af18-33c8-4f64-bf52-548a457165ec-etc-sysctl-d\") pod \"tuned-fm8n4\" (UID: \"e9b5af18-33c8-4f64-bf52-548a457165ec\") " pod="openshift-cluster-node-tuning-operator/tuned-fm8n4" Jul 21 01:36:18.868140 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.866967 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/77f8ae4c-96ea-49a3-ac6d-93df2beaa785-host-run-multus-certs\") pod \"multus-sz95c\" (UID: \"77f8ae4c-96ea-49a3-ac6d-93df2beaa785\") " pod="openshift-multus/multus-sz95c" Jul 21 01:36:18.868140 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.867006 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e9b5af18-33c8-4f64-bf52-548a457165ec-etc-sysctl-conf\") pod \"tuned-fm8n4\" (UID: \"e9b5af18-33c8-4f64-bf52-548a457165ec\") " pod="openshift-cluster-node-tuning-operator/tuned-fm8n4" Jul 21 01:36:18.868140 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.867005 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8a8276e8-8761-4775-950c-742e4384c570-ovnkube-script-lib\") pod \"ovnkube-node-5m9tn\" (UID: \"8a8276e8-8761-4775-950c-742e4384c570\") " pod="openshift-ovn-kubernetes/ovnkube-node-5m9tn" Jul 21 01:36:18.868140 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.867070 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8a8276e8-8761-4775-950c-742e4384c570-systemd-units\") pod \"ovnkube-node-5m9tn\" (UID: \"8a8276e8-8761-4775-950c-742e4384c570\") " pod="openshift-ovn-kubernetes/ovnkube-node-5m9tn" Jul 21 01:36:18.868140 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.867103 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/77f8ae4c-96ea-49a3-ac6d-93df2beaa785-multus-conf-dir\") pod \"multus-sz95c\" (UID: \"77f8ae4c-96ea-49a3-ac6d-93df2beaa785\") " pod="openshift-multus/multus-sz95c" Jul 21 01:36:18.868140 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.867137 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8a8276e8-8761-4775-950c-742e4384c570-systemd-units\") pod \"ovnkube-node-5m9tn\" (UID: \"8a8276e8-8761-4775-950c-742e4384c570\") " pod="openshift-ovn-kubernetes/ovnkube-node-5m9tn" Jul 21 01:36:18.868140 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.867186 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7346c638-55dd-4d75-9d69-cead4209e04c-metrics-certs\") pod \"network-metrics-daemon-vdpxq\" (UID: \"7346c638-55dd-4d75-9d69-cead4209e04c\") " pod="openshift-multus/network-metrics-daemon-vdpxq" Jul 21 01:36:18.869301 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.867216 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0a21e01e-3c16-429c-85bb-df4c0e096756-socket-dir\") pod \"aws-ebs-csi-driver-node-bs276\" (UID: \"0a21e01e-3c16-429c-85bb-df4c0e096756\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bs276" Jul 21 01:36:18.869301 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.867256 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/77f8ae4c-96ea-49a3-ac6d-93df2beaa785-host-run-k8s-cni-cncf-io\") pod \"multus-sz95c\" (UID: \"77f8ae4c-96ea-49a3-ac6d-93df2beaa785\") " pod="openshift-multus/multus-sz95c" Jul 21 01:36:18.869301 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.867253 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/77f8ae4c-96ea-49a3-ac6d-93df2beaa785-multus-conf-dir\") pod \"multus-sz95c\" (UID: \"77f8ae4c-96ea-49a3-ac6d-93df2beaa785\") " pod="openshift-multus/multus-sz95c" Jul 21 01:36:18.869301 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.867302 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0a21e01e-3c16-429c-85bb-df4c0e096756-sys-fs\") pod \"aws-ebs-csi-driver-node-bs276\" (UID: \"0a21e01e-3c16-429c-85bb-df4c0e096756\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bs276" Jul 21 01:36:18.869301 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.867312 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/77f8ae4c-96ea-49a3-ac6d-93df2beaa785-host-run-k8s-cni-cncf-io\") pod \"multus-sz95c\" (UID: \"77f8ae4c-96ea-49a3-ac6d-93df2beaa785\") " pod="openshift-multus/multus-sz95c" Jul 21 01:36:18.869301 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.867352 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/64ab7299-1b1f-4d7e-8c26-998adf4499db-cnibin\") pod \"multus-additional-cni-plugins-sm5jf\" (UID: \"64ab7299-1b1f-4d7e-8c26-998adf4499db\") " pod="openshift-multus/multus-additional-cni-plugins-sm5jf" Jul 21 01:36:18.869301 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.867382 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/64ab7299-1b1f-4d7e-8c26-998adf4499db-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sm5jf\" (UID: \"64ab7299-1b1f-4d7e-8c26-998adf4499db\") " pod="openshift-multus/multus-additional-cni-plugins-sm5jf" Jul 21 01:36:18.869301 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.867413 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/77f8ae4c-96ea-49a3-ac6d-93df2beaa785-host-run-netns\") pod \"multus-sz95c\" (UID: \"77f8ae4c-96ea-49a3-ac6d-93df2beaa785\") " pod="openshift-multus/multus-sz95c" Jul 21 01:36:18.869301 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.867441 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2b1299d5-1664-4e47-b8af-381bc1a7d7fe-iptables-alerter-script\") pod \"iptables-alerter-8688j\" (UID: \"2b1299d5-1664-4e47-b8af-381bc1a7d7fe\") " pod="openshift-network-operator/iptables-alerter-8688j" Jul 21 01:36:18.869301 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.867454 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0a21e01e-3c16-429c-85bb-df4c0e096756-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bs276\" (UID: \"0a21e01e-3c16-429c-85bb-df4c0e096756\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bs276" Jul 21 01:36:18.869301 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.867517 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/77f8ae4c-96ea-49a3-ac6d-93df2beaa785-host-run-netns\") pod \"multus-sz95c\" (UID: \"77f8ae4c-96ea-49a3-ac6d-93df2beaa785\") " pod="openshift-multus/multus-sz95c" Jul 21 01:36:18.869301 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.867529 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/64ab7299-1b1f-4d7e-8c26-998adf4499db-cni-binary-copy\") pod \"multus-additional-cni-plugins-sm5jf\" (UID: \"64ab7299-1b1f-4d7e-8c26-998adf4499db\") " pod="openshift-multus/multus-additional-cni-plugins-sm5jf" Jul 21 01:36:18.869301 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.867559 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/77f8ae4c-96ea-49a3-ac6d-93df2beaa785-multus-socket-dir-parent\") pod \"multus-sz95c\" (UID: \"77f8ae4c-96ea-49a3-ac6d-93df2beaa785\") " pod="openshift-multus/multus-sz95c" Jul 21 01:36:18.869301 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.867595 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/77f8ae4c-96ea-49a3-ac6d-93df2beaa785-host-var-lib-kubelet\") pod \"multus-sz95c\" (UID: \"77f8ae4c-96ea-49a3-ac6d-93df2beaa785\") " pod="openshift-multus/multus-sz95c" Jul 21 01:36:18.869301 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.867651 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/77f8ae4c-96ea-49a3-ac6d-93df2beaa785-host-var-lib-kubelet\") pod \"multus-sz95c\" (UID: \"77f8ae4c-96ea-49a3-ac6d-93df2beaa785\") " pod="openshift-multus/multus-sz95c" Jul 21 01:36:18.869301 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.867692 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/77f8ae4c-96ea-49a3-ac6d-93df2beaa785-multus-socket-dir-parent\") pod \"multus-sz95c\" (UID: \"77f8ae4c-96ea-49a3-ac6d-93df2beaa785\") " pod="openshift-multus/multus-sz95c" Jul 21 01:36:18.869301 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.867714 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17bdceb8-8016-4d0b-8b3c-f36507279428-original-pull-secret\") pod \"global-pull-secret-syncer-24nj2\" (UID: \"17bdceb8-8016-4d0b-8b3c-f36507279428\") " pod="kube-system/global-pull-secret-syncer-24nj2" Jul 21 01:36:18.870123 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.867748 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e9b5af18-33c8-4f64-bf52-548a457165ec-etc-modprobe-d\") pod \"tuned-fm8n4\" (UID: \"e9b5af18-33c8-4f64-bf52-548a457165ec\") " pod="openshift-cluster-node-tuning-operator/tuned-fm8n4" Jul 21 01:36:18.870123 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.867790 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/64ab7299-1b1f-4d7e-8c26-998adf4499db-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sm5jf\" (UID: \"64ab7299-1b1f-4d7e-8c26-998adf4499db\") " pod="openshift-multus/multus-additional-cni-plugins-sm5jf" Jul 21 01:36:18.870123 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.869572 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8a8276e8-8761-4775-950c-742e4384c570-ovn-node-metrics-cert\") pod \"ovnkube-node-5m9tn\" (UID: \"8a8276e8-8761-4775-950c-742e4384c570\") " pod="openshift-ovn-kubernetes/ovnkube-node-5m9tn" Jul 21 01:36:18.870322 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.870300 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e548f5b6-6507-4431-b027-88917b43191f-agent-certs\") pod \"konnectivity-agent-smxdt\" (UID: \"e548f5b6-6507-4431-b027-88917b43191f\") " pod="kube-system/konnectivity-agent-smxdt" Jul 21 01:36:18.874863 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:18.874222 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 01:36:18.874863 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:18.874254 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 01:36:18.874863 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:18.874268 2580 projected.go:196] Error preparing data for projected volume kube-api-access-mlprc for pod openshift-network-diagnostics/network-check-target-pczgw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 01:36:18.875116 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:18.874961 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1527759a-5529-4c49-a9e5-cdc780b0b21a-kube-api-access-mlprc podName:1527759a-5529-4c49-a9e5-cdc780b0b21a nodeName:}" failed. No retries permitted until 2026-07-21 01:36:19.374902401 +0000 UTC m=+2.962068453 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-mlprc" (UniqueName: "kubernetes.io/projected/1527759a-5529-4c49-a9e5-cdc780b0b21a-kube-api-access-mlprc") pod "network-check-target-pczgw" (UID: "1527759a-5529-4c49-a9e5-cdc780b0b21a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 01:36:18.880934 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.880309 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d79t9\" (UniqueName: \"kubernetes.io/projected/77f8ae4c-96ea-49a3-ac6d-93df2beaa785-kube-api-access-d79t9\") pod \"multus-sz95c\" (UID: \"77f8ae4c-96ea-49a3-ac6d-93df2beaa785\") " pod="openshift-multus/multus-sz95c" Jul 21 01:36:18.880934 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.880479 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvkr6\" (UniqueName: \"kubernetes.io/projected/8a8276e8-8761-4775-950c-742e4384c570-kube-api-access-gvkr6\") pod \"ovnkube-node-5m9tn\" (UID: \"8a8276e8-8761-4775-950c-742e4384c570\") " pod="openshift-ovn-kubernetes/ovnkube-node-5m9tn" Jul 21 01:36:18.882468 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.882435 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zmrl\" (UniqueName: \"kubernetes.io/projected/2b1299d5-1664-4e47-b8af-381bc1a7d7fe-kube-api-access-6zmrl\") pod \"iptables-alerter-8688j\" (UID: \"2b1299d5-1664-4e47-b8af-381bc1a7d7fe\") " pod="openshift-network-operator/iptables-alerter-8688j" Jul 21 01:36:18.886273 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.886215 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-183.ec2.internal" event={"ID":"fb35f72ee7f4b217fd778d25ec504876","Type":"ContainerStarted","Data":"67f63760309118c1b822e73805d908735ad169ebcb32701b0ea13ea99b7d8c61"} Jul 21 01:36:18.886380 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.886297 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-183.ec2.internal" event={"ID":"b7aa145ad43083f158fe9d117b8cccf9","Type":"ContainerStarted","Data":"720120c260d26e3163487207db7cfd53e425359d347cb03aca3eafdae3064a29"} Jul 21 01:36:18.968815 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.968778 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/64ab7299-1b1f-4d7e-8c26-998adf4499db-cni-binary-copy\") pod \"multus-additional-cni-plugins-sm5jf\" (UID: \"64ab7299-1b1f-4d7e-8c26-998adf4499db\") " pod="openshift-multus/multus-additional-cni-plugins-sm5jf" Jul 21 01:36:18.968815 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.968817 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17bdceb8-8016-4d0b-8b3c-f36507279428-original-pull-secret\") pod \"global-pull-secret-syncer-24nj2\" (UID: \"17bdceb8-8016-4d0b-8b3c-f36507279428\") " pod="kube-system/global-pull-secret-syncer-24nj2" Jul 21 01:36:18.969006 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.968834 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e9b5af18-33c8-4f64-bf52-548a457165ec-etc-modprobe-d\") pod \"tuned-fm8n4\" (UID: \"e9b5af18-33c8-4f64-bf52-548a457165ec\") " pod="openshift-cluster-node-tuning-operator/tuned-fm8n4" Jul 21 01:36:18.969006 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.968852 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/64ab7299-1b1f-4d7e-8c26-998adf4499db-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sm5jf\" (UID: \"64ab7299-1b1f-4d7e-8c26-998adf4499db\") " pod="openshift-multus/multus-additional-cni-plugins-sm5jf" Jul 21 01:36:18.969006 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.968871 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-st25h\" (UniqueName: \"kubernetes.io/projected/a1ca57f9-11f8-48c3-9ff6-a7b2561fede0-kube-api-access-st25h\") pod \"node-resolver-5n4nl\" (UID: \"a1ca57f9-11f8-48c3-9ff6-a7b2561fede0\") " pod="openshift-dns/node-resolver-5n4nl" Jul 21 01:36:18.969006 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.968915 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1489d404-17a6-4bf1-9a85-9d3787aece54-host\") pod \"node-ca-zxcjm\" (UID: \"1489d404-17a6-4bf1-9a85-9d3787aece54\") " pod="openshift-image-registry/node-ca-zxcjm" Jul 21 01:36:18.969006 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.968932 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e9b5af18-33c8-4f64-bf52-548a457165ec-var-lib-kubelet\") pod \"tuned-fm8n4\" (UID: \"e9b5af18-33c8-4f64-bf52-548a457165ec\") " pod="openshift-cluster-node-tuning-operator/tuned-fm8n4" Jul 21 01:36:18.969006 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.968951 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1ca57f9-11f8-48c3-9ff6-a7b2561fede0-tmp-dir\") pod \"node-resolver-5n4nl\" (UID: \"a1ca57f9-11f8-48c3-9ff6-a7b2561fede0\") " pod="openshift-dns/node-resolver-5n4nl" Jul 21 01:36:18.969006 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:18.968959 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 01:36:18.969298 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:18.969056 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17bdceb8-8016-4d0b-8b3c-f36507279428-original-pull-secret podName:17bdceb8-8016-4d0b-8b3c-f36507279428 nodeName:}" failed. No retries permitted until 2026-07-21 01:36:19.469033177 +0000 UTC m=+3.056199155 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/17bdceb8-8016-4d0b-8b3c-f36507279428-original-pull-secret") pod "global-pull-secret-syncer-24nj2" (UID: "17bdceb8-8016-4d0b-8b3c-f36507279428") : object "kube-system"/"original-pull-secret" not registered Jul 21 01:36:18.969298 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.969060 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/64ab7299-1b1f-4d7e-8c26-998adf4499db-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sm5jf\" (UID: \"64ab7299-1b1f-4d7e-8c26-998adf4499db\") " pod="openshift-multus/multus-additional-cni-plugins-sm5jf" Jul 21 01:36:18.969298 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.969073 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e9b5af18-33c8-4f64-bf52-548a457165ec-etc-modprobe-d\") pod \"tuned-fm8n4\" (UID: \"e9b5af18-33c8-4f64-bf52-548a457165ec\") " pod="openshift-cluster-node-tuning-operator/tuned-fm8n4" Jul 21 01:36:18.969298 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.969091 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1489d404-17a6-4bf1-9a85-9d3787aece54-host\") pod \"node-ca-zxcjm\" (UID: \"1489d404-17a6-4bf1-9a85-9d3787aece54\") " pod="openshift-image-registry/node-ca-zxcjm" Jul 21 01:36:18.969298 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.969071 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e9b5af18-33c8-4f64-bf52-548a457165ec-lib-modules\") pod \"tuned-fm8n4\" (UID: \"e9b5af18-33c8-4f64-bf52-548a457165ec\") " pod="openshift-cluster-node-tuning-operator/tuned-fm8n4" Jul 21 01:36:18.969298 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.968972 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e9b5af18-33c8-4f64-bf52-548a457165ec-lib-modules\") pod \"tuned-fm8n4\" (UID: \"e9b5af18-33c8-4f64-bf52-548a457165ec\") " pod="openshift-cluster-node-tuning-operator/tuned-fm8n4" Jul 21 01:36:18.969298 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.969139 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e9b5af18-33c8-4f64-bf52-548a457165ec-var-lib-kubelet\") pod \"tuned-fm8n4\" (UID: \"e9b5af18-33c8-4f64-bf52-548a457165ec\") " pod="openshift-cluster-node-tuning-operator/tuned-fm8n4" Jul 21 01:36:18.969298 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.969142 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0a21e01e-3c16-429c-85bb-df4c0e096756-registration-dir\") pod \"aws-ebs-csi-driver-node-bs276\" (UID: \"0a21e01e-3c16-429c-85bb-df4c0e096756\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bs276" Jul 21 01:36:18.969298 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.969173 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0a21e01e-3c16-429c-85bb-df4c0e096756-etc-selinux\") pod \"aws-ebs-csi-driver-node-bs276\" (UID: \"0a21e01e-3c16-429c-85bb-df4c0e096756\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bs276" Jul 21 01:36:18.969298 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.969203 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx2d7\" (UniqueName: \"kubernetes.io/projected/0a21e01e-3c16-429c-85bb-df4c0e096756-kube-api-access-dx2d7\") pod \"aws-ebs-csi-driver-node-bs276\" (UID: \"0a21e01e-3c16-429c-85bb-df4c0e096756\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bs276" Jul 21 01:36:18.969298 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.969232 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/64ab7299-1b1f-4d7e-8c26-998adf4499db-system-cni-dir\") pod \"multus-additional-cni-plugins-sm5jf\" (UID: \"64ab7299-1b1f-4d7e-8c26-998adf4499db\") " pod="openshift-multus/multus-additional-cni-plugins-sm5jf" Jul 21 01:36:18.969298 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.969282 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/17bdceb8-8016-4d0b-8b3c-f36507279428-dbus\") pod \"global-pull-secret-syncer-24nj2\" (UID: \"17bdceb8-8016-4d0b-8b3c-f36507279428\") " pod="kube-system/global-pull-secret-syncer-24nj2" Jul 21 01:36:18.970000 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.969310 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e9b5af18-33c8-4f64-bf52-548a457165ec-tmp\") pod \"tuned-fm8n4\" (UID: \"e9b5af18-33c8-4f64-bf52-548a457165ec\") " pod="openshift-cluster-node-tuning-operator/tuned-fm8n4" Jul 21 01:36:18.970000 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.969337 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1ca57f9-11f8-48c3-9ff6-a7b2561fede0-tmp-dir\") pod \"node-resolver-5n4nl\" (UID: \"a1ca57f9-11f8-48c3-9ff6-a7b2561fede0\") " pod="openshift-dns/node-resolver-5n4nl" Jul 21 01:36:18.970000 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.969347 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0a21e01e-3c16-429c-85bb-df4c0e096756-etc-selinux\") pod \"aws-ebs-csi-driver-node-bs276\" (UID: \"0a21e01e-3c16-429c-85bb-df4c0e096756\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bs276" Jul 21 01:36:18.970000 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.969337 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx7sr\" (UniqueName: \"kubernetes.io/projected/64ab7299-1b1f-4d7e-8c26-998adf4499db-kube-api-access-vx7sr\") pod \"multus-additional-cni-plugins-sm5jf\" (UID: \"64ab7299-1b1f-4d7e-8c26-998adf4499db\") " pod="openshift-multus/multus-additional-cni-plugins-sm5jf" Jul 21 01:36:18.970000 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.969363 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/64ab7299-1b1f-4d7e-8c26-998adf4499db-system-cni-dir\") pod \"multus-additional-cni-plugins-sm5jf\" (UID: \"64ab7299-1b1f-4d7e-8c26-998adf4499db\") " pod="openshift-multus/multus-additional-cni-plugins-sm5jf" Jul 21 01:36:18.970000 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.969397 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0a21e01e-3c16-429c-85bb-df4c0e096756-registration-dir\") pod \"aws-ebs-csi-driver-node-bs276\" (UID: \"0a21e01e-3c16-429c-85bb-df4c0e096756\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bs276" Jul 21 01:36:18.970000 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.969449 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/64ab7299-1b1f-4d7e-8c26-998adf4499db-cni-binary-copy\") pod \"multus-additional-cni-plugins-sm5jf\" (UID: \"64ab7299-1b1f-4d7e-8c26-998adf4499db\") " pod="openshift-multus/multus-additional-cni-plugins-sm5jf" Jul 21 01:36:18.970000 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.969485 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e9b5af18-33c8-4f64-bf52-548a457165ec-run\") pod \"tuned-fm8n4\" (UID: \"e9b5af18-33c8-4f64-bf52-548a457165ec\") " pod="openshift-cluster-node-tuning-operator/tuned-fm8n4" Jul 21 01:36:18.970000 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.969553 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e9b5af18-33c8-4f64-bf52-548a457165ec-run\") pod \"tuned-fm8n4\" (UID: \"e9b5af18-33c8-4f64-bf52-548a457165ec\") " pod="openshift-cluster-node-tuning-operator/tuned-fm8n4" Jul 21 01:36:18.970000 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.969583 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chjdn\" (UniqueName: \"kubernetes.io/projected/7346c638-55dd-4d75-9d69-cead4209e04c-kube-api-access-chjdn\") pod \"network-metrics-daemon-vdpxq\" (UID: \"7346c638-55dd-4d75-9d69-cead4209e04c\") " pod="openshift-multus/network-metrics-daemon-vdpxq" Jul 21 01:36:18.970000 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.969613 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e9b5af18-33c8-4f64-bf52-548a457165ec-host\") pod \"tuned-fm8n4\" (UID: \"e9b5af18-33c8-4f64-bf52-548a457165ec\") " pod="openshift-cluster-node-tuning-operator/tuned-fm8n4" Jul 21 01:36:18.970000 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.969613 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/17bdceb8-8016-4d0b-8b3c-f36507279428-dbus\") pod \"global-pull-secret-syncer-24nj2\" (UID: \"17bdceb8-8016-4d0b-8b3c-f36507279428\") " pod="kube-system/global-pull-secret-syncer-24nj2" Jul 21 01:36:18.970000 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.969635 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0a21e01e-3c16-429c-85bb-df4c0e096756-device-dir\") pod \"aws-ebs-csi-driver-node-bs276\" (UID: \"0a21e01e-3c16-429c-85bb-df4c0e096756\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bs276" Jul 21 01:36:18.970000 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.969657 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74pgb\" (UniqueName: \"kubernetes.io/projected/1489d404-17a6-4bf1-9a85-9d3787aece54-kube-api-access-74pgb\") pod \"node-ca-zxcjm\" (UID: \"1489d404-17a6-4bf1-9a85-9d3787aece54\") " pod="openshift-image-registry/node-ca-zxcjm" Jul 21 01:36:18.970000 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.969683 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e9b5af18-33c8-4f64-bf52-548a457165ec-etc-sysconfig\") pod \"tuned-fm8n4\" (UID: \"e9b5af18-33c8-4f64-bf52-548a457165ec\") " pod="openshift-cluster-node-tuning-operator/tuned-fm8n4" Jul 21 01:36:18.970000 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.969711 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e9b5af18-33c8-4f64-bf52-548a457165ec-host\") pod \"tuned-fm8n4\" (UID: \"e9b5af18-33c8-4f64-bf52-548a457165ec\") " pod="openshift-cluster-node-tuning-operator/tuned-fm8n4" Jul 21 01:36:18.970000 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.969738 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e9b5af18-33c8-4f64-bf52-548a457165ec-etc-sysconfig\") pod \"tuned-fm8n4\" (UID: \"e9b5af18-33c8-4f64-bf52-548a457165ec\") " pod="openshift-cluster-node-tuning-operator/tuned-fm8n4" Jul 21 01:36:18.970835 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.969760 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0a21e01e-3c16-429c-85bb-df4c0e096756-device-dir\") pod \"aws-ebs-csi-driver-node-bs276\" (UID: \"0a21e01e-3c16-429c-85bb-df4c0e096756\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bs276" Jul 21 01:36:18.970835 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.969790 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e9b5af18-33c8-4f64-bf52-548a457165ec-etc-systemd\") pod \"tuned-fm8n4\" (UID: \"e9b5af18-33c8-4f64-bf52-548a457165ec\") " pod="openshift-cluster-node-tuning-operator/tuned-fm8n4" Jul 21 01:36:18.970835 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.969823 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e9b5af18-33c8-4f64-bf52-548a457165ec-etc-tuned\") pod \"tuned-fm8n4\" (UID: \"e9b5af18-33c8-4f64-bf52-548a457165ec\") " pod="openshift-cluster-node-tuning-operator/tuned-fm8n4" Jul 21 01:36:18.970835 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.969853 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qqbs\" (UniqueName: \"kubernetes.io/projected/e9b5af18-33c8-4f64-bf52-548a457165ec-kube-api-access-8qqbs\") pod \"tuned-fm8n4\" (UID: \"e9b5af18-33c8-4f64-bf52-548a457165ec\") " pod="openshift-cluster-node-tuning-operator/tuned-fm8n4" Jul 21 01:36:18.970835 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.969859 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e9b5af18-33c8-4f64-bf52-548a457165ec-etc-systemd\") pod \"tuned-fm8n4\" (UID: \"e9b5af18-33c8-4f64-bf52-548a457165ec\") " pod="openshift-cluster-node-tuning-operator/tuned-fm8n4" Jul 21 01:36:18.970835 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.969879 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a1ca57f9-11f8-48c3-9ff6-a7b2561fede0-hosts-file\") pod \"node-resolver-5n4nl\" (UID: \"a1ca57f9-11f8-48c3-9ff6-a7b2561fede0\") " pod="openshift-dns/node-resolver-5n4nl" Jul 21 01:36:18.970835 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.969933 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a1ca57f9-11f8-48c3-9ff6-a7b2561fede0-hosts-file\") pod \"node-resolver-5n4nl\" (UID: \"a1ca57f9-11f8-48c3-9ff6-a7b2561fede0\") " pod="openshift-dns/node-resolver-5n4nl" Jul 21 01:36:18.970835 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.969956 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1489d404-17a6-4bf1-9a85-9d3787aece54-serviceca\") pod \"node-ca-zxcjm\" (UID: \"1489d404-17a6-4bf1-9a85-9d3787aece54\") " pod="openshift-image-registry/node-ca-zxcjm" Jul 21 01:36:18.970835 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.969983 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/17bdceb8-8016-4d0b-8b3c-f36507279428-kubelet-config\") pod \"global-pull-secret-syncer-24nj2\" (UID: \"17bdceb8-8016-4d0b-8b3c-f36507279428\") " pod="kube-system/global-pull-secret-syncer-24nj2" Jul 21 01:36:18.970835 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.970001 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e9b5af18-33c8-4f64-bf52-548a457165ec-etc-kubernetes\") pod \"tuned-fm8n4\" (UID: \"e9b5af18-33c8-4f64-bf52-548a457165ec\") " pod="openshift-cluster-node-tuning-operator/tuned-fm8n4" Jul 21 01:36:18.970835 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.970017 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e9b5af18-33c8-4f64-bf52-548a457165ec-sys\") pod \"tuned-fm8n4\" (UID: \"e9b5af18-33c8-4f64-bf52-548a457165ec\") " pod="openshift-cluster-node-tuning-operator/tuned-fm8n4" Jul 21 01:36:18.970835 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.970033 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/64ab7299-1b1f-4d7e-8c26-998adf4499db-os-release\") pod \"multus-additional-cni-plugins-sm5jf\" (UID: \"64ab7299-1b1f-4d7e-8c26-998adf4499db\") " pod="openshift-multus/multus-additional-cni-plugins-sm5jf" Jul 21 01:36:18.970835 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.970055 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/64ab7299-1b1f-4d7e-8c26-998adf4499db-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sm5jf\" (UID: \"64ab7299-1b1f-4d7e-8c26-998adf4499db\") " pod="openshift-multus/multus-additional-cni-plugins-sm5jf" Jul 21 01:36:18.970835 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.970076 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e9b5af18-33c8-4f64-bf52-548a457165ec-etc-sysctl-d\") pod \"tuned-fm8n4\" (UID: \"e9b5af18-33c8-4f64-bf52-548a457165ec\") " pod="openshift-cluster-node-tuning-operator/tuned-fm8n4" Jul 21 01:36:18.970835 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.970091 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e9b5af18-33c8-4f64-bf52-548a457165ec-etc-sysctl-conf\") pod \"tuned-fm8n4\" (UID: \"e9b5af18-33c8-4f64-bf52-548a457165ec\") " pod="openshift-cluster-node-tuning-operator/tuned-fm8n4" Jul 21 01:36:18.970835 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.970103 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e9b5af18-33c8-4f64-bf52-548a457165ec-etc-kubernetes\") pod \"tuned-fm8n4\" (UID: \"e9b5af18-33c8-4f64-bf52-548a457165ec\") " pod="openshift-cluster-node-tuning-operator/tuned-fm8n4" Jul 21 01:36:18.970835 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.970110 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7346c638-55dd-4d75-9d69-cead4209e04c-metrics-certs\") pod \"network-metrics-daemon-vdpxq\" (UID: \"7346c638-55dd-4d75-9d69-cead4209e04c\") " pod="openshift-multus/network-metrics-daemon-vdpxq" Jul 21 01:36:18.971437 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.970148 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0a21e01e-3c16-429c-85bb-df4c0e096756-socket-dir\") pod \"aws-ebs-csi-driver-node-bs276\" (UID: \"0a21e01e-3c16-429c-85bb-df4c0e096756\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bs276" Jul 21 01:36:18.971437 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.970152 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/17bdceb8-8016-4d0b-8b3c-f36507279428-kubelet-config\") pod \"global-pull-secret-syncer-24nj2\" (UID: \"17bdceb8-8016-4d0b-8b3c-f36507279428\") " pod="kube-system/global-pull-secret-syncer-24nj2" Jul 21 01:36:18.971437 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.970183 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0a21e01e-3c16-429c-85bb-df4c0e096756-sys-fs\") pod \"aws-ebs-csi-driver-node-bs276\" (UID: \"0a21e01e-3c16-429c-85bb-df4c0e096756\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bs276" Jul 21 01:36:18.971437 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.970209 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/64ab7299-1b1f-4d7e-8c26-998adf4499db-cnibin\") pod \"multus-additional-cni-plugins-sm5jf\" (UID: \"64ab7299-1b1f-4d7e-8c26-998adf4499db\") " pod="openshift-multus/multus-additional-cni-plugins-sm5jf" Jul 21 01:36:18.971437 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:18.970211 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 01:36:18.971437 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.970236 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/64ab7299-1b1f-4d7e-8c26-998adf4499db-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sm5jf\" (UID: \"64ab7299-1b1f-4d7e-8c26-998adf4499db\") " pod="openshift-multus/multus-additional-cni-plugins-sm5jf" Jul 21 01:36:18.971437 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:18.970264 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7346c638-55dd-4d75-9d69-cead4209e04c-metrics-certs podName:7346c638-55dd-4d75-9d69-cead4209e04c nodeName:}" failed. No retries permitted until 2026-07-21 01:36:19.470246993 +0000 UTC m=+3.057412955 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7346c638-55dd-4d75-9d69-cead4209e04c-metrics-certs") pod "network-metrics-daemon-vdpxq" (UID: "7346c638-55dd-4d75-9d69-cead4209e04c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 01:36:18.971437 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.970295 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0a21e01e-3c16-429c-85bb-df4c0e096756-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bs276\" (UID: \"0a21e01e-3c16-429c-85bb-df4c0e096756\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bs276" Jul 21 01:36:18.971437 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.970316 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/64ab7299-1b1f-4d7e-8c26-998adf4499db-cnibin\") pod \"multus-additional-cni-plugins-sm5jf\" (UID: \"64ab7299-1b1f-4d7e-8c26-998adf4499db\") " pod="openshift-multus/multus-additional-cni-plugins-sm5jf" Jul 21 01:36:18.971437 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.970315 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0a21e01e-3c16-429c-85bb-df4c0e096756-sys-fs\") pod \"aws-ebs-csi-driver-node-bs276\" (UID: \"0a21e01e-3c16-429c-85bb-df4c0e096756\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bs276" Jul 21 01:36:18.971437 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.970351 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1489d404-17a6-4bf1-9a85-9d3787aece54-serviceca\") pod \"node-ca-zxcjm\" (UID: \"1489d404-17a6-4bf1-9a85-9d3787aece54\") " pod="openshift-image-registry/node-ca-zxcjm" Jul 21 01:36:18.971437 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.970430 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0a21e01e-3c16-429c-85bb-df4c0e096756-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bs276\" (UID: \"0a21e01e-3c16-429c-85bb-df4c0e096756\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bs276" Jul 21 01:36:18.971437 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.970432 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0a21e01e-3c16-429c-85bb-df4c0e096756-socket-dir\") pod \"aws-ebs-csi-driver-node-bs276\" (UID: \"0a21e01e-3c16-429c-85bb-df4c0e096756\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bs276" Jul 21 01:36:18.971437 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.970468 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e9b5af18-33c8-4f64-bf52-548a457165ec-etc-sysctl-d\") pod \"tuned-fm8n4\" (UID: \"e9b5af18-33c8-4f64-bf52-548a457165ec\") " pod="openshift-cluster-node-tuning-operator/tuned-fm8n4" Jul 21 01:36:18.971437 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.970494 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e9b5af18-33c8-4f64-bf52-548a457165ec-sys\") pod \"tuned-fm8n4\" (UID: \"e9b5af18-33c8-4f64-bf52-548a457165ec\") " pod="openshift-cluster-node-tuning-operator/tuned-fm8n4" Jul 21 01:36:18.971437 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.970516 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/64ab7299-1b1f-4d7e-8c26-998adf4499db-os-release\") pod \"multus-additional-cni-plugins-sm5jf\" (UID: \"64ab7299-1b1f-4d7e-8c26-998adf4499db\") " pod="openshift-multus/multus-additional-cni-plugins-sm5jf" Jul 21 01:36:18.971437 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.970535 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e9b5af18-33c8-4f64-bf52-548a457165ec-etc-sysctl-conf\") pod \"tuned-fm8n4\" (UID: \"e9b5af18-33c8-4f64-bf52-548a457165ec\") " pod="openshift-cluster-node-tuning-operator/tuned-fm8n4" Jul 21 01:36:18.972141 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.970908 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/64ab7299-1b1f-4d7e-8c26-998adf4499db-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sm5jf\" (UID: \"64ab7299-1b1f-4d7e-8c26-998adf4499db\") " pod="openshift-multus/multus-additional-cni-plugins-sm5jf" Jul 21 01:36:18.972141 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.971008 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/64ab7299-1b1f-4d7e-8c26-998adf4499db-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sm5jf\" (UID: \"64ab7299-1b1f-4d7e-8c26-998adf4499db\") " pod="openshift-multus/multus-additional-cni-plugins-sm5jf" Jul 21 01:36:18.972141 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.971904 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e9b5af18-33c8-4f64-bf52-548a457165ec-tmp\") pod \"tuned-fm8n4\" (UID: \"e9b5af18-33c8-4f64-bf52-548a457165ec\") " pod="openshift-cluster-node-tuning-operator/tuned-fm8n4" Jul 21 01:36:18.972141 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.972133 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e9b5af18-33c8-4f64-bf52-548a457165ec-etc-tuned\") pod \"tuned-fm8n4\" (UID: \"e9b5af18-33c8-4f64-bf52-548a457165ec\") " pod="openshift-cluster-node-tuning-operator/tuned-fm8n4" Jul 21 01:36:18.979278 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.979211 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-st25h\" (UniqueName: \"kubernetes.io/projected/a1ca57f9-11f8-48c3-9ff6-a7b2561fede0-kube-api-access-st25h\") pod \"node-resolver-5n4nl\" (UID: \"a1ca57f9-11f8-48c3-9ff6-a7b2561fede0\") " pod="openshift-dns/node-resolver-5n4nl" Jul 21 01:36:18.979407 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.979295 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx7sr\" (UniqueName: \"kubernetes.io/projected/64ab7299-1b1f-4d7e-8c26-998adf4499db-kube-api-access-vx7sr\") pod \"multus-additional-cni-plugins-sm5jf\" (UID: \"64ab7299-1b1f-4d7e-8c26-998adf4499db\") " pod="openshift-multus/multus-additional-cni-plugins-sm5jf" Jul 21 01:36:18.980091 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.980062 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx2d7\" (UniqueName: \"kubernetes.io/projected/0a21e01e-3c16-429c-85bb-df4c0e096756-kube-api-access-dx2d7\") pod \"aws-ebs-csi-driver-node-bs276\" (UID: \"0a21e01e-3c16-429c-85bb-df4c0e096756\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bs276" Jul 21 01:36:18.980738 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.980709 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74pgb\" (UniqueName: \"kubernetes.io/projected/1489d404-17a6-4bf1-9a85-9d3787aece54-kube-api-access-74pgb\") pod \"node-ca-zxcjm\" (UID: \"1489d404-17a6-4bf1-9a85-9d3787aece54\") " pod="openshift-image-registry/node-ca-zxcjm" Jul 21 01:36:18.981049 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.981026 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qqbs\" (UniqueName: \"kubernetes.io/projected/e9b5af18-33c8-4f64-bf52-548a457165ec-kube-api-access-8qqbs\") pod \"tuned-fm8n4\" (UID: \"e9b5af18-33c8-4f64-bf52-548a457165ec\") " pod="openshift-cluster-node-tuning-operator/tuned-fm8n4" Jul 21 01:36:18.981160 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:18.981142 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chjdn\" (UniqueName: \"kubernetes.io/projected/7346c638-55dd-4d75-9d69-cead4209e04c-kube-api-access-chjdn\") pod \"network-metrics-daemon-vdpxq\" (UID: \"7346c638-55dd-4d75-9d69-cead4209e04c\") " pod="openshift-multus/network-metrics-daemon-vdpxq" Jul 21 01:36:19.047695 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:19.047655 2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 01:36:19.063290 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:19.063251 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5m9tn" Jul 21 01:36:19.073161 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:19.073135 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8688j" Jul 21 01:36:19.084437 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:19.084414 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-smxdt" Jul 21 01:36:19.091384 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:19.091355 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-sz95c" Jul 21 01:36:19.100257 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:19.100227 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zxcjm" Jul 21 01:36:19.108384 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:19.108348 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fm8n4" Jul 21 01:36:19.116465 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:19.116442 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bs276" Jul 21 01:36:19.124686 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:19.124664 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5n4nl" Jul 21 01:36:19.131729 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:19.131703 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-sm5jf" Jul 21 01:36:19.475107 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:19.475015 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlprc\" (UniqueName: \"kubernetes.io/projected/1527759a-5529-4c49-a9e5-cdc780b0b21a-kube-api-access-mlprc\") pod \"network-check-target-pczgw\" (UID: \"1527759a-5529-4c49-a9e5-cdc780b0b21a\") " pod="openshift-network-diagnostics/network-check-target-pczgw" Jul 21 01:36:19.475107 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:19.475082 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7346c638-55dd-4d75-9d69-cead4209e04c-metrics-certs\") pod \"network-metrics-daemon-vdpxq\" (UID: \"7346c638-55dd-4d75-9d69-cead4209e04c\") " pod="openshift-multus/network-metrics-daemon-vdpxq" Jul 21 01:36:19.475353 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:19.475198 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 01:36:19.475353 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:19.475221 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 01:36:19.475353 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:19.475254 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 01:36:19.475353 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:19.475269 2580 projected.go:196] Error preparing data for projected volume kube-api-access-mlprc for pod openshift-network-diagnostics/network-check-target-pczgw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 01:36:19.475353 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:19.475294 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7346c638-55dd-4d75-9d69-cead4209e04c-metrics-certs podName:7346c638-55dd-4d75-9d69-cead4209e04c nodeName:}" failed. No retries permitted until 2026-07-21 01:36:20.475273997 +0000 UTC m=+4.062439968 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7346c638-55dd-4d75-9d69-cead4209e04c-metrics-certs") pod "network-metrics-daemon-vdpxq" (UID: "7346c638-55dd-4d75-9d69-cead4209e04c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 01:36:19.475353 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:19.475308 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 01:36:19.475667 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:19.475358 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17bdceb8-8016-4d0b-8b3c-f36507279428-original-pull-secret podName:17bdceb8-8016-4d0b-8b3c-f36507279428 nodeName:}" failed. No retries permitted until 2026-07-21 01:36:20.475335754 +0000 UTC m=+4.062501727 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/17bdceb8-8016-4d0b-8b3c-f36507279428-original-pull-secret") pod "global-pull-secret-syncer-24nj2" (UID: "17bdceb8-8016-4d0b-8b3c-f36507279428") : object "kube-system"/"original-pull-secret" not registered Jul 21 01:36:19.475667 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:19.475264 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17bdceb8-8016-4d0b-8b3c-f36507279428-original-pull-secret\") pod \"global-pull-secret-syncer-24nj2\" (UID: \"17bdceb8-8016-4d0b-8b3c-f36507279428\") " pod="kube-system/global-pull-secret-syncer-24nj2" Jul 21 01:36:19.475667 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:19.475441 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1527759a-5529-4c49-a9e5-cdc780b0b21a-kube-api-access-mlprc podName:1527759a-5529-4c49-a9e5-cdc780b0b21a nodeName:}" failed. No retries permitted until 2026-07-21 01:36:20.475421828 +0000 UTC m=+4.062587804 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-mlprc" (UniqueName: "kubernetes.io/projected/1527759a-5529-4c49-a9e5-cdc780b0b21a-kube-api-access-mlprc") pod "network-check-target-pczgw" (UID: "1527759a-5529-4c49-a9e5-cdc780b0b21a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 01:36:19.800308 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:19.800214 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-20 01:31:19 +0000 UTC" deadline="2028-03-05 14:32:23.186147679 +0000 UTC" Jul 21 01:36:19.800308 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:19.800262 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14244h56m3.385906143s" Jul 21 01:36:19.871064 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:19.871026 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vdpxq" Jul 21 01:36:19.871250 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:19.871079 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-24nj2" Jul 21 01:36:19.871250 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:19.871195 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vdpxq" podUID="7346c638-55dd-4d75-9d69-cead4209e04c" Jul 21 01:36:19.871378 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:19.871279 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-24nj2" podUID="17bdceb8-8016-4d0b-8b3c-f36507279428" Jul 21 01:36:20.018054 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:20.018020 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a8276e8_8761_4775_950c_742e4384c570.slice/crio-9217ca7f7819329bf0c1023fa29e770e7b51ee72025545ae4c7599ac2ef62c76 WatchSource:0}: Error finding container 9217ca7f7819329bf0c1023fa29e770e7b51ee72025545ae4c7599ac2ef62c76: Status 404 returned error can't find the container with id 9217ca7f7819329bf0c1023fa29e770e7b51ee72025545ae4c7599ac2ef62c76 Jul 21 01:36:20.041406 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:20.041210 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1489d404_17a6_4bf1_9a85_9d3787aece54.slice/crio-82e638b49469a558940ebc82a41d64952e2e3080c353498a3735f72855747b06 WatchSource:0}: Error finding container 82e638b49469a558940ebc82a41d64952e2e3080c353498a3735f72855747b06: Status 404 returned error can't find the container with id 82e638b49469a558940ebc82a41d64952e2e3080c353498a3735f72855747b06 Jul 21 01:36:20.041612 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:20.041559 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode548f5b6_6507_4431_b027_88917b43191f.slice/crio-3a10244543309a654fce77712a507c4680f943f78c242dc3112c5f63248c30d9 WatchSource:0}: Error finding container 3a10244543309a654fce77712a507c4680f943f78c242dc3112c5f63248c30d9: Status 404 returned error can't find the container with id 3a10244543309a654fce77712a507c4680f943f78c242dc3112c5f63248c30d9 Jul 21 01:36:20.049731 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:20.049707 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a21e01e_3c16_429c_85bb_df4c0e096756.slice/crio-a85a6e0faf2387dee7f69c229e4ff461138e8cf3d09248e40d4dcd88887b0967 WatchSource:0}: Error finding container a85a6e0faf2387dee7f69c229e4ff461138e8cf3d09248e40d4dcd88887b0967: Status 404 returned error can't find the container with id a85a6e0faf2387dee7f69c229e4ff461138e8cf3d09248e40d4dcd88887b0967 Jul 21 01:36:20.051039 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:20.051010 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9b5af18_33c8_4f64_bf52_548a457165ec.slice/crio-c011f661dc33c39a86bd9496e3da4790bd6ace9c87f8b70b263af61f8a328082 WatchSource:0}: Error finding container c011f661dc33c39a86bd9496e3da4790bd6ace9c87f8b70b263af61f8a328082: Status 404 returned error can't find the container with id c011f661dc33c39a86bd9496e3da4790bd6ace9c87f8b70b263af61f8a328082 Jul 21 01:36:20.051817 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:20.051793 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77f8ae4c_96ea_49a3_ac6d_93df2beaa785.slice/crio-b0735f6e65fbe6ad70529ac5d75987232a182dd4edae54a1e43dd750a1620b61 WatchSource:0}: Error finding container b0735f6e65fbe6ad70529ac5d75987232a182dd4edae54a1e43dd750a1620b61: Status 404 returned error can't find the container with id b0735f6e65fbe6ad70529ac5d75987232a182dd4edae54a1e43dd750a1620b61 Jul 21 01:36:20.053872 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:20.053654 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64ab7299_1b1f_4d7e_8c26_998adf4499db.slice/crio-125e46e9dd98b6711933acdb124e41c8bff6752c70efc6964ae7c4aa9b7ac135 WatchSource:0}: Error finding container 125e46e9dd98b6711933acdb124e41c8bff6752c70efc6964ae7c4aa9b7ac135: Status 404 returned error can't find the container with id 125e46e9dd98b6711933acdb124e41c8bff6752c70efc6964ae7c4aa9b7ac135 Jul 21 01:36:20.056290 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:20.056009 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b1299d5_1664_4e47_b8af_381bc1a7d7fe.slice/crio-1ae1109953204dff1c87175e5334b586d0ddc421fc3eb0e2d07669b5ce58b0fe WatchSource:0}: Error finding container 1ae1109953204dff1c87175e5334b586d0ddc421fc3eb0e2d07669b5ce58b0fe: Status 404 returned error can't find the container with id 1ae1109953204dff1c87175e5334b586d0ddc421fc3eb0e2d07669b5ce58b0fe Jul 21 01:36:20.056495 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:20.056474 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1ca57f9_11f8_48c3_9ff6_a7b2561fede0.slice/crio-016036ff39490544ee456e67e77e0744949a01e7bbfac54bc92c2a068f94b73d WatchSource:0}: Error finding container 016036ff39490544ee456e67e77e0744949a01e7bbfac54bc92c2a068f94b73d: Status 404 returned error can't find the container with id 016036ff39490544ee456e67e77e0744949a01e7bbfac54bc92c2a068f94b73d Jul 21 01:36:20.483624 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:20.483515 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7346c638-55dd-4d75-9d69-cead4209e04c-metrics-certs\") pod \"network-metrics-daemon-vdpxq\" (UID: \"7346c638-55dd-4d75-9d69-cead4209e04c\") " pod="openshift-multus/network-metrics-daemon-vdpxq" Jul 21 01:36:20.483624 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:20.483577 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17bdceb8-8016-4d0b-8b3c-f36507279428-original-pull-secret\") pod \"global-pull-secret-syncer-24nj2\" (UID: \"17bdceb8-8016-4d0b-8b3c-f36507279428\") " pod="kube-system/global-pull-secret-syncer-24nj2" Jul 21 01:36:20.483624 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:20.483621 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlprc\" (UniqueName: \"kubernetes.io/projected/1527759a-5529-4c49-a9e5-cdc780b0b21a-kube-api-access-mlprc\") pod \"network-check-target-pczgw\" (UID: \"1527759a-5529-4c49-a9e5-cdc780b0b21a\") " pod="openshift-network-diagnostics/network-check-target-pczgw" Jul 21 01:36:20.483904 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:20.483779 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 01:36:20.483904 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:20.483812 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 01:36:20.483904 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:20.483852 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17bdceb8-8016-4d0b-8b3c-f36507279428-original-pull-secret podName:17bdceb8-8016-4d0b-8b3c-f36507279428 nodeName:}" failed. No retries permitted until 2026-07-21 01:36:22.483832375 +0000 UTC m=+6.070998337 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/17bdceb8-8016-4d0b-8b3c-f36507279428-original-pull-secret") pod "global-pull-secret-syncer-24nj2" (UID: "17bdceb8-8016-4d0b-8b3c-f36507279428") : object "kube-system"/"original-pull-secret" not registered Jul 21 01:36:20.483904 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:20.483781 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 01:36:20.483904 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:20.483902 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 01:36:20.484131 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:20.483916 2580 projected.go:196] Error preparing data for projected volume kube-api-access-mlprc for pod openshift-network-diagnostics/network-check-target-pczgw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 01:36:20.484131 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:20.483884 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7346c638-55dd-4d75-9d69-cead4209e04c-metrics-certs podName:7346c638-55dd-4d75-9d69-cead4209e04c nodeName:}" failed. No retries permitted until 2026-07-21 01:36:22.483864815 +0000 UTC m=+6.071030794 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7346c638-55dd-4d75-9d69-cead4209e04c-metrics-certs") pod "network-metrics-daemon-vdpxq" (UID: "7346c638-55dd-4d75-9d69-cead4209e04c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 01:36:20.484131 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:20.483980 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1527759a-5529-4c49-a9e5-cdc780b0b21a-kube-api-access-mlprc podName:1527759a-5529-4c49-a9e5-cdc780b0b21a nodeName:}" failed. No retries permitted until 2026-07-21 01:36:22.483966718 +0000 UTC m=+6.071132676 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-mlprc" (UniqueName: "kubernetes.io/projected/1527759a-5529-4c49-a9e5-cdc780b0b21a-kube-api-access-mlprc") pod "network-check-target-pczgw" (UID: "1527759a-5529-4c49-a9e5-cdc780b0b21a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 01:36:20.800844 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:20.800482 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-20 01:31:19 +0000 UTC" deadline="2027-12-30 23:24:07.279578837 +0000 UTC" Jul 21 01:36:20.800844 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:20.800795 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12669h47m46.478791348s" Jul 21 01:36:20.874191 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:20.873763 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pczgw" Jul 21 01:36:20.874191 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:20.873903 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pczgw" podUID="1527759a-5529-4c49-a9e5-cdc780b0b21a" Jul 21 01:36:20.898069 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:20.898025 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bs276" event={"ID":"0a21e01e-3c16-429c-85bb-df4c0e096756","Type":"ContainerStarted","Data":"a85a6e0faf2387dee7f69c229e4ff461138e8cf3d09248e40d4dcd88887b0967"} Jul 21 01:36:20.901231 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:20.901190 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-smxdt" event={"ID":"e548f5b6-6507-4431-b027-88917b43191f","Type":"ContainerStarted","Data":"3a10244543309a654fce77712a507c4680f943f78c242dc3112c5f63248c30d9"} Jul 21 01:36:20.904910 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:20.904882 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5m9tn" event={"ID":"8a8276e8-8761-4775-950c-742e4384c570","Type":"ContainerStarted","Data":"9217ca7f7819329bf0c1023fa29e770e7b51ee72025545ae4c7599ac2ef62c76"} Jul 21 01:36:20.919551 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:20.919266 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-183.ec2.internal" event={"ID":"fb35f72ee7f4b217fd778d25ec504876","Type":"ContainerStarted","Data":"2d2f67972a7f74719885bb636ae37624e40ef8a84ed221dcb6f4326bb5ee9ebd"} Jul 21 01:36:20.930734 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:20.930696 2580 generic.go:356] "Generic (PLEG): container finished" podID="b7aa145ad43083f158fe9d117b8cccf9" containerID="b987760f9ab8bc424c3690f4215748a0e556a2fe1998031b6231dd019bb368b4" exitCode=0 Jul 21 01:36:20.931380 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:20.931350 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-183.ec2.internal" event={"ID":"b7aa145ad43083f158fe9d117b8cccf9","Type":"ContainerDied","Data":"b987760f9ab8bc424c3690f4215748a0e556a2fe1998031b6231dd019bb368b4"} Jul 21 01:36:20.944814 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:20.944755 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5n4nl" event={"ID":"a1ca57f9-11f8-48c3-9ff6-a7b2561fede0","Type":"ContainerStarted","Data":"016036ff39490544ee456e67e77e0744949a01e7bbfac54bc92c2a068f94b73d"} Jul 21 01:36:20.952375 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:20.952317 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-183.ec2.internal" podStartSLOduration=2.952296413 podStartE2EDuration="2.952296413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 01:36:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:36:20.936593763 +0000 UTC m=+4.523759766" watchObservedRunningTime="2026-07-21 01:36:20.952296413 +0000 UTC m=+4.539462397" Jul 21 01:36:20.952821 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:20.952780 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fm8n4" event={"ID":"e9b5af18-33c8-4f64-bf52-548a457165ec","Type":"ContainerStarted","Data":"c011f661dc33c39a86bd9496e3da4790bd6ace9c87f8b70b263af61f8a328082"} Jul 21 01:36:20.970774 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:20.970741 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zxcjm" event={"ID":"1489d404-17a6-4bf1-9a85-9d3787aece54","Type":"ContainerStarted","Data":"82e638b49469a558940ebc82a41d64952e2e3080c353498a3735f72855747b06"} Jul 21 01:36:20.984393 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:20.984361 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8688j" event={"ID":"2b1299d5-1664-4e47-b8af-381bc1a7d7fe","Type":"ContainerStarted","Data":"1ae1109953204dff1c87175e5334b586d0ddc421fc3eb0e2d07669b5ce58b0fe"} Jul 21 01:36:20.992805 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:20.992775 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sm5jf" event={"ID":"64ab7299-1b1f-4d7e-8c26-998adf4499db","Type":"ContainerStarted","Data":"125e46e9dd98b6711933acdb124e41c8bff6752c70efc6964ae7c4aa9b7ac135"} Jul 21 01:36:20.997961 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:20.997933 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-sz95c" event={"ID":"77f8ae4c-96ea-49a3-ac6d-93df2beaa785","Type":"ContainerStarted","Data":"b0735f6e65fbe6ad70529ac5d75987232a182dd4edae54a1e43dd750a1620b61"} Jul 21 01:36:21.870582 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:21.870492 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vdpxq" Jul 21 01:36:21.871081 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:21.870656 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vdpxq" podUID="7346c638-55dd-4d75-9d69-cead4209e04c" Jul 21 01:36:21.871081 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:21.871065 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-24nj2" Jul 21 01:36:21.871225 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:21.871167 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-24nj2" podUID="17bdceb8-8016-4d0b-8b3c-f36507279428" Jul 21 01:36:22.030637 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:22.030593 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-183.ec2.internal" event={"ID":"b7aa145ad43083f158fe9d117b8cccf9","Type":"ContainerStarted","Data":"b1f97fdb0d7d8efe9ae82d7c8181b7f2d46de50e057d88b5925064590f95580d"} Jul 21 01:36:22.048445 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:22.048385 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-183.ec2.internal" podStartSLOduration=4.048365006 podStartE2EDuration="4.048365006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 01:36:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:36:22.047940315 +0000 UTC m=+5.635106298" watchObservedRunningTime="2026-07-21 01:36:22.048365006 +0000 UTC m=+5.635530987" Jul 21 01:36:22.500455 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:22.500227 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlprc\" (UniqueName: \"kubernetes.io/projected/1527759a-5529-4c49-a9e5-cdc780b0b21a-kube-api-access-mlprc\") pod \"network-check-target-pczgw\" (UID: \"1527759a-5529-4c49-a9e5-cdc780b0b21a\") " pod="openshift-network-diagnostics/network-check-target-pczgw" Jul 21 01:36:22.500455 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:22.500332 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7346c638-55dd-4d75-9d69-cead4209e04c-metrics-certs\") pod \"network-metrics-daemon-vdpxq\" (UID: \"7346c638-55dd-4d75-9d69-cead4209e04c\") " pod="openshift-multus/network-metrics-daemon-vdpxq" Jul 21 01:36:22.500455 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:22.500367 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17bdceb8-8016-4d0b-8b3c-f36507279428-original-pull-secret\") pod \"global-pull-secret-syncer-24nj2\" (UID: \"17bdceb8-8016-4d0b-8b3c-f36507279428\") " pod="kube-system/global-pull-secret-syncer-24nj2" Jul 21 01:36:22.500455 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:22.500431 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 01:36:22.500455 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:22.500474 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 01:36:22.500944 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:22.500489 2580 projected.go:196] Error preparing data for projected volume kube-api-access-mlprc for pod openshift-network-diagnostics/network-check-target-pczgw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 01:36:22.500944 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:22.500554 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 01:36:22.500944 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:22.500598 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 01:36:22.500944 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:22.500558 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1527759a-5529-4c49-a9e5-cdc780b0b21a-kube-api-access-mlprc podName:1527759a-5529-4c49-a9e5-cdc780b0b21a nodeName:}" failed. No retries permitted until 2026-07-21 01:36:26.500535947 +0000 UTC m=+10.087701910 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-mlprc" (UniqueName: "kubernetes.io/projected/1527759a-5529-4c49-a9e5-cdc780b0b21a-kube-api-access-mlprc") pod "network-check-target-pczgw" (UID: "1527759a-5529-4c49-a9e5-cdc780b0b21a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 01:36:22.500944 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:22.500638 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7346c638-55dd-4d75-9d69-cead4209e04c-metrics-certs podName:7346c638-55dd-4d75-9d69-cead4209e04c nodeName:}" failed. No retries permitted until 2026-07-21 01:36:26.500619249 +0000 UTC m=+10.087785224 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7346c638-55dd-4d75-9d69-cead4209e04c-metrics-certs") pod "network-metrics-daemon-vdpxq" (UID: "7346c638-55dd-4d75-9d69-cead4209e04c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 01:36:22.500944 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:22.500678 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17bdceb8-8016-4d0b-8b3c-f36507279428-original-pull-secret podName:17bdceb8-8016-4d0b-8b3c-f36507279428 nodeName:}" failed. No retries permitted until 2026-07-21 01:36:26.500661202 +0000 UTC m=+10.087827179 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/17bdceb8-8016-4d0b-8b3c-f36507279428-original-pull-secret") pod "global-pull-secret-syncer-24nj2" (UID: "17bdceb8-8016-4d0b-8b3c-f36507279428") : object "kube-system"/"original-pull-secret" not registered Jul 21 01:36:22.875689 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:22.875607 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pczgw" Jul 21 01:36:22.876178 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:22.875744 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pczgw" podUID="1527759a-5529-4c49-a9e5-cdc780b0b21a" Jul 21 01:36:23.870793 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:23.870755 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vdpxq" Jul 21 01:36:23.871014 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:23.870915 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vdpxq" podUID="7346c638-55dd-4d75-9d69-cead4209e04c" Jul 21 01:36:23.871014 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:23.870977 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-24nj2" Jul 21 01:36:23.871139 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:23.871065 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-24nj2" podUID="17bdceb8-8016-4d0b-8b3c-f36507279428" Jul 21 01:36:24.871963 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:24.871347 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pczgw" Jul 21 01:36:24.871963 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:24.871504 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pczgw" podUID="1527759a-5529-4c49-a9e5-cdc780b0b21a" Jul 21 01:36:25.870960 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:25.870372 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vdpxq" Jul 21 01:36:25.870960 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:25.870552 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vdpxq" podUID="7346c638-55dd-4d75-9d69-cead4209e04c" Jul 21 01:36:25.870960 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:25.870782 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-24nj2" Jul 21 01:36:25.870960 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:25.870940 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-24nj2" podUID="17bdceb8-8016-4d0b-8b3c-f36507279428" Jul 21 01:36:26.533552 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:26.533504 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlprc\" (UniqueName: \"kubernetes.io/projected/1527759a-5529-4c49-a9e5-cdc780b0b21a-kube-api-access-mlprc\") pod \"network-check-target-pczgw\" (UID: \"1527759a-5529-4c49-a9e5-cdc780b0b21a\") " pod="openshift-network-diagnostics/network-check-target-pczgw" Jul 21 01:36:26.534074 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:26.533602 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7346c638-55dd-4d75-9d69-cead4209e04c-metrics-certs\") pod \"network-metrics-daemon-vdpxq\" (UID: \"7346c638-55dd-4d75-9d69-cead4209e04c\") " pod="openshift-multus/network-metrics-daemon-vdpxq" Jul 21 01:36:26.534074 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:26.533639 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17bdceb8-8016-4d0b-8b3c-f36507279428-original-pull-secret\") pod \"global-pull-secret-syncer-24nj2\" (UID: \"17bdceb8-8016-4d0b-8b3c-f36507279428\") " pod="kube-system/global-pull-secret-syncer-24nj2" Jul 21 01:36:26.534074 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:26.533778 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 01:36:26.534074 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:26.533852 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17bdceb8-8016-4d0b-8b3c-f36507279428-original-pull-secret podName:17bdceb8-8016-4d0b-8b3c-f36507279428 nodeName:}" failed. No retries permitted until 2026-07-21 01:36:34.533831225 +0000 UTC m=+18.120997229 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/17bdceb8-8016-4d0b-8b3c-f36507279428-original-pull-secret") pod "global-pull-secret-syncer-24nj2" (UID: "17bdceb8-8016-4d0b-8b3c-f36507279428") : object "kube-system"/"original-pull-secret" not registered Jul 21 01:36:26.534397 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:26.534373 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 01:36:26.534479 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:26.534399 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 01:36:26.534479 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:26.534424 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 01:36:26.534479 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:26.534438 2580 projected.go:196] Error preparing data for projected volume kube-api-access-mlprc for pod openshift-network-diagnostics/network-check-target-pczgw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 01:36:26.534479 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:26.534461 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7346c638-55dd-4d75-9d69-cead4209e04c-metrics-certs podName:7346c638-55dd-4d75-9d69-cead4209e04c nodeName:}" failed. No retries permitted until 2026-07-21 01:36:34.534430641 +0000 UTC m=+18.121596623 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7346c638-55dd-4d75-9d69-cead4209e04c-metrics-certs") pod "network-metrics-daemon-vdpxq" (UID: "7346c638-55dd-4d75-9d69-cead4209e04c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 01:36:26.534743 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:26.534499 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1527759a-5529-4c49-a9e5-cdc780b0b21a-kube-api-access-mlprc podName:1527759a-5529-4c49-a9e5-cdc780b0b21a nodeName:}" failed. No retries permitted until 2026-07-21 01:36:34.534481996 +0000 UTC m=+18.121647974 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-mlprc" (UniqueName: "kubernetes.io/projected/1527759a-5529-4c49-a9e5-cdc780b0b21a-kube-api-access-mlprc") pod "network-check-target-pczgw" (UID: "1527759a-5529-4c49-a9e5-cdc780b0b21a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 01:36:26.871731 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:26.871634 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pczgw" Jul 21 01:36:26.871910 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:26.871757 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pczgw" podUID="1527759a-5529-4c49-a9e5-cdc780b0b21a" Jul 21 01:36:27.871362 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:27.871317 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vdpxq" Jul 21 01:36:27.871362 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:27.871356 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-24nj2" Jul 21 01:36:27.871974 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:27.871512 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vdpxq" podUID="7346c638-55dd-4d75-9d69-cead4209e04c" Jul 21 01:36:27.871974 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:27.871651 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-24nj2" podUID="17bdceb8-8016-4d0b-8b3c-f36507279428" Jul 21 01:36:28.871425 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:28.871337 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pczgw" Jul 21 01:36:28.871884 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:28.871509 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pczgw" podUID="1527759a-5529-4c49-a9e5-cdc780b0b21a" Jul 21 01:36:29.870661 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:29.870600 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vdpxq" Jul 21 01:36:29.870661 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:29.870626 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-24nj2" Jul 21 01:36:29.870903 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:29.870734 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-24nj2" podUID="17bdceb8-8016-4d0b-8b3c-f36507279428" Jul 21 01:36:29.870959 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:29.870888 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vdpxq" podUID="7346c638-55dd-4d75-9d69-cead4209e04c" Jul 21 01:36:30.870441 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:30.870408 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pczgw" Jul 21 01:36:30.870911 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:30.870561 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pczgw" podUID="1527759a-5529-4c49-a9e5-cdc780b0b21a" Jul 21 01:36:31.871046 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:31.871008 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vdpxq" Jul 21 01:36:31.871542 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:31.871053 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-24nj2" Jul 21 01:36:31.871542 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:31.871151 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vdpxq" podUID="7346c638-55dd-4d75-9d69-cead4209e04c" Jul 21 01:36:31.871542 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:31.871269 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-24nj2" podUID="17bdceb8-8016-4d0b-8b3c-f36507279428" Jul 21 01:36:32.870679 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:32.870645 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pczgw" Jul 21 01:36:32.870852 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:32.870751 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pczgw" podUID="1527759a-5529-4c49-a9e5-cdc780b0b21a" Jul 21 01:36:33.870914 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:33.870876 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vdpxq" Jul 21 01:36:33.871383 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:33.870925 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-24nj2" Jul 21 01:36:33.871383 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:33.871007 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vdpxq" podUID="7346c638-55dd-4d75-9d69-cead4209e04c" Jul 21 01:36:33.871383 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:33.871128 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-24nj2" podUID="17bdceb8-8016-4d0b-8b3c-f36507279428" Jul 21 01:36:34.595707 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:34.595666 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7346c638-55dd-4d75-9d69-cead4209e04c-metrics-certs\") pod \"network-metrics-daemon-vdpxq\" (UID: \"7346c638-55dd-4d75-9d69-cead4209e04c\") " pod="openshift-multus/network-metrics-daemon-vdpxq" Jul 21 01:36:34.595909 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:34.595721 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17bdceb8-8016-4d0b-8b3c-f36507279428-original-pull-secret\") pod \"global-pull-secret-syncer-24nj2\" (UID: \"17bdceb8-8016-4d0b-8b3c-f36507279428\") " pod="kube-system/global-pull-secret-syncer-24nj2" Jul 21 01:36:34.595909 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:34.595762 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlprc\" (UniqueName: \"kubernetes.io/projected/1527759a-5529-4c49-a9e5-cdc780b0b21a-kube-api-access-mlprc\") pod \"network-check-target-pczgw\" (UID: \"1527759a-5529-4c49-a9e5-cdc780b0b21a\") " pod="openshift-network-diagnostics/network-check-target-pczgw" Jul 21 01:36:34.595909 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:34.595810 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 01:36:34.595909 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:34.595872 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7346c638-55dd-4d75-9d69-cead4209e04c-metrics-certs podName:7346c638-55dd-4d75-9d69-cead4209e04c nodeName:}" failed. No retries permitted until 2026-07-21 01:36:50.595851121 +0000 UTC m=+34.183017093 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7346c638-55dd-4d75-9d69-cead4209e04c-metrics-certs") pod "network-metrics-daemon-vdpxq" (UID: "7346c638-55dd-4d75-9d69-cead4209e04c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 01:36:34.595909 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:34.595884 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 01:36:34.595909 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:34.595891 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 01:36:34.595909 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:34.595909 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 01:36:34.596257 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:34.595919 2580 projected.go:196] Error preparing data for projected volume kube-api-access-mlprc for pod openshift-network-diagnostics/network-check-target-pczgw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 01:36:34.596257 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:34.595942 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17bdceb8-8016-4d0b-8b3c-f36507279428-original-pull-secret podName:17bdceb8-8016-4d0b-8b3c-f36507279428 nodeName:}" failed. No retries permitted until 2026-07-21 01:36:50.595925518 +0000 UTC m=+34.183091488 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/17bdceb8-8016-4d0b-8b3c-f36507279428-original-pull-secret") pod "global-pull-secret-syncer-24nj2" (UID: "17bdceb8-8016-4d0b-8b3c-f36507279428") : object "kube-system"/"original-pull-secret" not registered Jul 21 01:36:34.596257 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:34.595972 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1527759a-5529-4c49-a9e5-cdc780b0b21a-kube-api-access-mlprc podName:1527759a-5529-4c49-a9e5-cdc780b0b21a nodeName:}" failed. No retries permitted until 2026-07-21 01:36:50.595962921 +0000 UTC m=+34.183128877 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-mlprc" (UniqueName: "kubernetes.io/projected/1527759a-5529-4c49-a9e5-cdc780b0b21a-kube-api-access-mlprc") pod "network-check-target-pczgw" (UID: "1527759a-5529-4c49-a9e5-cdc780b0b21a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 01:36:34.871067 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:34.870993 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pczgw" Jul 21 01:36:34.871470 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:34.871104 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pczgw" podUID="1527759a-5529-4c49-a9e5-cdc780b0b21a" Jul 21 01:36:35.870782 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:35.870743 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vdpxq" Jul 21 01:36:35.870944 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:35.870765 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-24nj2" Jul 21 01:36:35.870944 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:35.870902 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vdpxq" podUID="7346c638-55dd-4d75-9d69-cead4209e04c" Jul 21 01:36:35.871053 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:35.871027 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-24nj2" podUID="17bdceb8-8016-4d0b-8b3c-f36507279428" Jul 21 01:36:36.871247 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:36.871213 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pczgw" Jul 21 01:36:36.871756 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:36.871322 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pczgw" podUID="1527759a-5529-4c49-a9e5-cdc780b0b21a" Jul 21 01:36:37.870874 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:37.870824 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vdpxq" Jul 21 01:36:37.870874 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:37.870869 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-24nj2" Jul 21 01:36:37.871102 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:37.870971 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vdpxq" podUID="7346c638-55dd-4d75-9d69-cead4209e04c" Jul 21 01:36:37.871163 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:37.871121 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-24nj2" podUID="17bdceb8-8016-4d0b-8b3c-f36507279428" Jul 21 01:36:38.870779 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:38.870752 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pczgw" Jul 21 01:36:38.871333 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:38.870890 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pczgw" podUID="1527759a-5529-4c49-a9e5-cdc780b0b21a" Jul 21 01:36:39.870596 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:39.870275 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vdpxq" Jul 21 01:36:39.870762 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:39.870326 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-24nj2" Jul 21 01:36:39.870762 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:39.870689 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vdpxq" podUID="7346c638-55dd-4d75-9d69-cead4209e04c" Jul 21 01:36:39.870762 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:39.870744 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-24nj2" podUID="17bdceb8-8016-4d0b-8b3c-f36507279428" Jul 21 01:36:40.064606 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:40.064560 2580 generic.go:356] "Generic (PLEG): container finished" podID="64ab7299-1b1f-4d7e-8c26-998adf4499db" containerID="be082cf750ad3e823ee54c930846ec27f7c582780a6cd476242b62f92ddd0ad4" exitCode=0 Jul 21 01:36:40.064867 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:40.064653 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sm5jf" event={"ID":"64ab7299-1b1f-4d7e-8c26-998adf4499db","Type":"ContainerDied","Data":"be082cf750ad3e823ee54c930846ec27f7c582780a6cd476242b62f92ddd0ad4"} Jul 21 01:36:40.066555 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:40.066345 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-sz95c" event={"ID":"77f8ae4c-96ea-49a3-ac6d-93df2beaa785","Type":"ContainerStarted","Data":"f4e51d8fd1ae02672735bae867083b7bffcbe0bd11f99f07f39c26c3669524f6"} Jul 21 01:36:40.068309 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:40.068279 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bs276" event={"ID":"0a21e01e-3c16-429c-85bb-df4c0e096756","Type":"ContainerStarted","Data":"5d2471f0de7abb9655d8af5c153503c5e97090863ceee587e4e90723f4e4a0cf"} Jul 21 01:36:40.070128 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:40.069835 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-smxdt" event={"ID":"e548f5b6-6507-4431-b027-88917b43191f","Type":"ContainerStarted","Data":"750d527186d004dee29342123fc9cf5cef2f41fad6b280f3a847ac607997d1fa"} Jul 21 01:36:40.073785 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:40.073761 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5m9tn" event={"ID":"8a8276e8-8761-4775-950c-742e4384c570","Type":"ContainerStarted","Data":"5d21de07f80a81e1c35c7a667bc39eff24352cb6aa978f93145182c4a9dfb894"} Jul 21 01:36:40.073880 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:40.073837 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5m9tn" event={"ID":"8a8276e8-8761-4775-950c-742e4384c570","Type":"ContainerStarted","Data":"afb386fa2639ab44021aa767593cc615d31679871aaa86a585b8924b2b187aa5"} Jul 21 01:36:40.073932 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:40.073885 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5m9tn" event={"ID":"8a8276e8-8761-4775-950c-742e4384c570","Type":"ContainerStarted","Data":"ca16612ebc011d684338ff27e237c1281da9480590c3d04b987ab08628bbdb90"} Jul 21 01:36:40.074264 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:40.074242 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5m9tn" event={"ID":"8a8276e8-8761-4775-950c-742e4384c570","Type":"ContainerStarted","Data":"4549295a1d4c50e4bb05ea05915963eeae7c7de884561ca25dca02bc79c87f5d"} Jul 21 01:36:40.074349 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:40.074326 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5m9tn" event={"ID":"8a8276e8-8761-4775-950c-742e4384c570","Type":"ContainerStarted","Data":"0faa8f0c5317bd8f7b230f7b3b6f00773d0e0555fc8b2afedf2c509981ba5862"} Jul 21 01:36:40.074420 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:40.074399 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5m9tn" event={"ID":"8a8276e8-8761-4775-950c-742e4384c570","Type":"ContainerStarted","Data":"ce83dd0321566456238efa7a9853593bad4bb8d7cbdfff958414838a2c14076f"} Jul 21 01:36:40.077526 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:40.076074 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5n4nl" event={"ID":"a1ca57f9-11f8-48c3-9ff6-a7b2561fede0","Type":"ContainerStarted","Data":"c1abcebe600fb7da434ab2bb9f039bae00395c0bd7d4088795eb1566d733a329"} Jul 21 01:36:40.078378 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:40.078340 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fm8n4" event={"ID":"e9b5af18-33c8-4f64-bf52-548a457165ec","Type":"ContainerStarted","Data":"8aa4f848f0715f9410e2b44c89dff3f041d4829fbe20c9639ea8e99dfab42095"} Jul 21 01:36:40.079989 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:40.079967 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zxcjm" event={"ID":"1489d404-17a6-4bf1-9a85-9d3787aece54","Type":"ContainerStarted","Data":"b7c07bf1a60e088ad3f088f3f148537cf4af54ad48ad056cc696b2eeaffe35a8"} Jul 21 01:36:40.101866 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:40.101811 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-sz95c" podStartSLOduration=2.941790984 podStartE2EDuration="22.101790192s" totalImagesPullingTime="19.159999208s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 01:36:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:36:40.10126708 +0000 UTC m=+23.688433058" watchObservedRunningTime="2026-07-21 01:36:40.101790192 +0000 UTC m=+23.688956173" Jul 21 01:36:40.115422 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:40.115366 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-5n4nl" podStartSLOduration=3.022336975 podStartE2EDuration="22.115351403s" totalImagesPullingTime="19.093014428s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 01:36:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:36:40.114618817 +0000 UTC m=+23.701784795" watchObservedRunningTime="2026-07-21 01:36:40.115351403 +0000 UTC m=+23.702517381" Jul 21 01:36:40.128372 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:40.128326 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-smxdt" podStartSLOduration=4.047877094 podStartE2EDuration="23.128310942s" totalImagesPullingTime="19.080433848s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 01:36:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:36:40.127754797 +0000 UTC m=+23.714920775" watchObservedRunningTime="2026-07-21 01:36:40.128310942 +0000 UTC m=+23.715476919" Jul 21 01:36:40.141694 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:40.141652 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-zxcjm" podStartSLOduration=3.038161734 podStartE2EDuration="22.141635688s" totalImagesPullingTime="19.103473954s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 01:36:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:36:40.141210528 +0000 UTC m=+23.728376506" watchObservedRunningTime="2026-07-21 01:36:40.141635688 +0000 UTC m=+23.728801665" Jul 21 01:36:40.156253 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:40.156206 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-fm8n4" podStartSLOduration=3.041806967 podStartE2EDuration="22.156186714s" totalImagesPullingTime="19.114379747s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 01:36:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:36:40.155965813 +0000 UTC m=+23.743131817" watchObservedRunningTime="2026-07-21 01:36:40.156186714 +0000 UTC m=+23.743352694" Jul 21 01:36:40.870965 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:40.870936 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pczgw" Jul 21 01:36:40.871303 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:40.871068 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pczgw" podUID="1527759a-5529-4c49-a9e5-cdc780b0b21a" Jul 21 01:36:40.903085 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:40.903044 2580 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 21 01:36:41.083890 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:41.083806 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8688j" event={"ID":"2b1299d5-1664-4e47-b8af-381bc1a7d7fe","Type":"ContainerStarted","Data":"7dd09cb718b73589b8e178e7242f855ce04e18f1e50632d4b43cf5b8286e23bf"} Jul 21 01:36:41.085868 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:41.085841 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bs276" event={"ID":"0a21e01e-3c16-429c-85bb-df4c0e096756","Type":"ContainerStarted","Data":"462fed7d9894604bbb457715ffa99747184d0cb4a879d5de524266f8f8bfc035"} Jul 21 01:36:41.097558 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:41.097519 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-8688j" podStartSLOduration=4.997154149 podStartE2EDuration="24.09750473s" totalImagesPullingTime="19.100350581s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 01:36:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:36:41.097383553 +0000 UTC m=+24.684549543" watchObservedRunningTime="2026-07-21 01:36:41.09750473 +0000 UTC m=+24.684670708" Jul 21 01:36:41.827437 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:41.827336 2580 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-21T01:36:40.903064505Z","UUID":"1f79c8db-a262-4efe-92a4-3d1400112242","Handler":null,"Name":"","Endpoint":""} Jul 21 01:36:41.828853 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:41.828831 2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 21 01:36:41.828979 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:41.828869 2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 21 01:36:41.871236 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:41.871196 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vdpxq" Jul 21 01:36:41.871785 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:41.871330 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vdpxq" podUID="7346c638-55dd-4d75-9d69-cead4209e04c" Jul 21 01:36:41.871892 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:41.871227 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-24nj2" Jul 21 01:36:41.871940 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:41.871890 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-24nj2" podUID="17bdceb8-8016-4d0b-8b3c-f36507279428" Jul 21 01:36:42.832818 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:42.832773 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-smxdt" Jul 21 01:36:42.833350 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:42.833327 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-smxdt" Jul 21 01:36:42.870594 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:42.870563 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pczgw" Jul 21 01:36:42.870756 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:42.870666 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pczgw" podUID="1527759a-5529-4c49-a9e5-cdc780b0b21a" Jul 21 01:36:43.094127 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:43.094043 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bs276" event={"ID":"0a21e01e-3c16-429c-85bb-df4c0e096756","Type":"ContainerStarted","Data":"bc2fbceba9f46cc18b58adf16c35ddfecca185c8adcf6c055976fab2ac8e984a"} Jul 21 01:36:43.097375 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:43.097349 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5m9tn" event={"ID":"8a8276e8-8761-4775-950c-742e4384c570","Type":"ContainerStarted","Data":"860eaff3b29eeacad6354e5628e6abfc81e667d04ed42ca998843f47da61f532"} Jul 21 01:36:43.097548 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:43.097415 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-smxdt" Jul 21 01:36:43.097973 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:43.097957 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-smxdt" Jul 21 01:36:43.110706 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:43.110665 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bs276" podStartSLOduration=3.102733793 podStartE2EDuration="25.110652916s" totalImagesPullingTime="22.007919123s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 01:36:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:36:43.110134223 +0000 UTC m=+26.697300201" watchObservedRunningTime="2026-07-21 01:36:43.110652916 +0000 UTC m=+26.697818894" Jul 21 01:36:43.871250 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:43.871211 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vdpxq" Jul 21 01:36:43.871442 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:43.871228 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-24nj2" Jul 21 01:36:43.871442 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:43.871341 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vdpxq" podUID="7346c638-55dd-4d75-9d69-cead4209e04c" Jul 21 01:36:43.871442 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:43.871390 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-24nj2" podUID="17bdceb8-8016-4d0b-8b3c-f36507279428" Jul 21 01:36:44.870850 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:44.870604 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pczgw" Jul 21 01:36:44.871311 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:44.870951 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pczgw" podUID="1527759a-5529-4c49-a9e5-cdc780b0b21a" Jul 21 01:36:45.871139 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:45.871064 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vdpxq" Jul 21 01:36:45.871139 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:45.871091 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-24nj2" Jul 21 01:36:45.871871 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:45.871182 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vdpxq" podUID="7346c638-55dd-4d75-9d69-cead4209e04c" Jul 21 01:36:45.871871 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:45.871261 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-24nj2" podUID="17bdceb8-8016-4d0b-8b3c-f36507279428" Jul 21 01:36:46.106122 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:46.105892 2580 generic.go:356] "Generic (PLEG): container finished" podID="64ab7299-1b1f-4d7e-8c26-998adf4499db" containerID="8e992d61bac0e0c22e0fcd6258e69c9b7780e8ab6c468cccc9b6d47fe7917ac8" exitCode=0 Jul 21 01:36:46.106276 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:46.105969 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sm5jf" event={"ID":"64ab7299-1b1f-4d7e-8c26-998adf4499db","Type":"ContainerDied","Data":"8e992d61bac0e0c22e0fcd6258e69c9b7780e8ab6c468cccc9b6d47fe7917ac8"} Jul 21 01:36:46.109527 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:46.109477 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5m9tn" event={"ID":"8a8276e8-8761-4775-950c-742e4384c570","Type":"ContainerStarted","Data":"86abd7157f7b756582b59d089a06c2ee3a79c26f05c9fe0c719ee46608ec5886"} Jul 21 01:36:46.109737 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:46.109721 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5m9tn" Jul 21 01:36:46.124580 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:46.124499 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5m9tn" Jul 21 01:36:46.149168 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:46.149114 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-5m9tn" podStartSLOduration=9.96192261 podStartE2EDuration="29.14909717s" totalImagesPullingTime="19.18717456s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 01:36:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:36:46.148693551 +0000 UTC m=+29.735859559" watchObservedRunningTime="2026-07-21 01:36:46.14909717 +0000 UTC m=+29.736263148" Jul 21 01:36:46.871805 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:46.871763 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pczgw" Jul 21 01:36:46.872413 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:46.871904 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pczgw" podUID="1527759a-5529-4c49-a9e5-cdc780b0b21a" Jul 21 01:36:47.111829 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:47.111796 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5m9tn" Jul 21 01:36:47.111996 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:47.111886 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5m9tn" Jul 21 01:36:47.126620 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:47.126541 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5m9tn" Jul 21 01:36:47.871333 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:47.871294 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-24nj2" Jul 21 01:36:47.871524 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:47.871298 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vdpxq" Jul 21 01:36:47.871524 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:47.871391 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-24nj2" podUID="17bdceb8-8016-4d0b-8b3c-f36507279428" Jul 21 01:36:47.871524 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:47.871511 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vdpxq" podUID="7346c638-55dd-4d75-9d69-cead4209e04c" Jul 21 01:36:47.904374 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:47.904341 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-24nj2"] Jul 21 01:36:47.907620 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:47.907591 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vdpxq"] Jul 21 01:36:47.908116 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:47.908096 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pczgw"] Jul 21 01:36:47.908199 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:47.908191 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pczgw" Jul 21 01:36:47.908310 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:47.908291 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pczgw" podUID="1527759a-5529-4c49-a9e5-cdc780b0b21a" Jul 21 01:36:48.115482 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:48.115433 2580 generic.go:356] "Generic (PLEG): container finished" podID="64ab7299-1b1f-4d7e-8c26-998adf4499db" containerID="2d3f5e4d241e4895fa32bfefca5d58478b62aae3986bb98e128ce6da37e4a4e7" exitCode=0 Jul 21 01:36:48.115678 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:48.115543 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sm5jf" event={"ID":"64ab7299-1b1f-4d7e-8c26-998adf4499db","Type":"ContainerDied","Data":"2d3f5e4d241e4895fa32bfefca5d58478b62aae3986bb98e128ce6da37e4a4e7"} Jul 21 01:36:48.115678 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:48.115556 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vdpxq" Jul 21 01:36:48.115787 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:48.115687 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-24nj2" Jul 21 01:36:48.115787 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:48.115702 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vdpxq" podUID="7346c638-55dd-4d75-9d69-cead4209e04c" Jul 21 01:36:48.115889 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:48.115821 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-24nj2" podUID="17bdceb8-8016-4d0b-8b3c-f36507279428" Jul 21 01:36:49.120572 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:49.120327 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sm5jf" event={"ID":"64ab7299-1b1f-4d7e-8c26-998adf4499db","Type":"ContainerStarted","Data":"f1eca2952dccc4b32ed88de730c68e2f9695a2bda7c9cf5a6890d033a5f77e48"} Jul 21 01:36:49.870874 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:49.870835 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vdpxq" Jul 21 01:36:49.870874 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:49.870874 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pczgw" Jul 21 01:36:49.871069 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:49.870878 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-24nj2" Jul 21 01:36:49.871069 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:49.870958 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vdpxq" podUID="7346c638-55dd-4d75-9d69-cead4209e04c" Jul 21 01:36:49.871069 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:49.871023 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pczgw" podUID="1527759a-5529-4c49-a9e5-cdc780b0b21a" Jul 21 01:36:49.871165 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:49.871074 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-24nj2" podUID="17bdceb8-8016-4d0b-8b3c-f36507279428" Jul 21 01:36:50.124506 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:50.124467 2580 generic.go:356] "Generic (PLEG): container finished" podID="64ab7299-1b1f-4d7e-8c26-998adf4499db" containerID="f1eca2952dccc4b32ed88de730c68e2f9695a2bda7c9cf5a6890d033a5f77e48" exitCode=0 Jul 21 01:36:50.124907 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:50.124481 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sm5jf" event={"ID":"64ab7299-1b1f-4d7e-8c26-998adf4499db","Type":"ContainerDied","Data":"f1eca2952dccc4b32ed88de730c68e2f9695a2bda7c9cf5a6890d033a5f77e48"} Jul 21 01:36:50.619771 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:50.619740 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17bdceb8-8016-4d0b-8b3c-f36507279428-original-pull-secret\") pod \"global-pull-secret-syncer-24nj2\" (UID: \"17bdceb8-8016-4d0b-8b3c-f36507279428\") " pod="kube-system/global-pull-secret-syncer-24nj2" Jul 21 01:36:50.619947 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:50.619790 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlprc\" (UniqueName: \"kubernetes.io/projected/1527759a-5529-4c49-a9e5-cdc780b0b21a-kube-api-access-mlprc\") pod \"network-check-target-pczgw\" (UID: \"1527759a-5529-4c49-a9e5-cdc780b0b21a\") " pod="openshift-network-diagnostics/network-check-target-pczgw" Jul 21 01:36:50.619947 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:50.619827 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7346c638-55dd-4d75-9d69-cead4209e04c-metrics-certs\") pod \"network-metrics-daemon-vdpxq\" (UID: \"7346c638-55dd-4d75-9d69-cead4209e04c\") " pod="openshift-multus/network-metrics-daemon-vdpxq" Jul 21 01:36:50.619947 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:50.619905 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 01:36:50.620101 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:50.619956 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 01:36:50.620101 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:50.619972 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 01:36:50.620101 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:50.619983 2580 projected.go:196] Error preparing data for projected volume kube-api-access-mlprc for pod openshift-network-diagnostics/network-check-target-pczgw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 01:36:50.620101 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:50.619985 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17bdceb8-8016-4d0b-8b3c-f36507279428-original-pull-secret podName:17bdceb8-8016-4d0b-8b3c-f36507279428 nodeName:}" failed. No retries permitted until 2026-07-21 01:37:22.619964341 +0000 UTC m=+66.207130321 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/17bdceb8-8016-4d0b-8b3c-f36507279428-original-pull-secret") pod "global-pull-secret-syncer-24nj2" (UID: "17bdceb8-8016-4d0b-8b3c-f36507279428") : object "kube-system"/"original-pull-secret" not registered Jul 21 01:36:50.620101 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:50.619914 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 01:36:50.620101 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:50.620022 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1527759a-5529-4c49-a9e5-cdc780b0b21a-kube-api-access-mlprc podName:1527759a-5529-4c49-a9e5-cdc780b0b21a nodeName:}" failed. No retries permitted until 2026-07-21 01:37:22.620010519 +0000 UTC m=+66.207176510 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-mlprc" (UniqueName: "kubernetes.io/projected/1527759a-5529-4c49-a9e5-cdc780b0b21a-kube-api-access-mlprc") pod "network-check-target-pczgw" (UID: "1527759a-5529-4c49-a9e5-cdc780b0b21a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 01:36:50.620101 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:50.620036 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7346c638-55dd-4d75-9d69-cead4209e04c-metrics-certs podName:7346c638-55dd-4d75-9d69-cead4209e04c nodeName:}" failed. No retries permitted until 2026-07-21 01:37:22.620029712 +0000 UTC m=+66.207195667 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7346c638-55dd-4d75-9d69-cead4209e04c-metrics-certs") pod "network-metrics-daemon-vdpxq" (UID: "7346c638-55dd-4d75-9d69-cead4209e04c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 01:36:51.870900 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:51.870864 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vdpxq" Jul 21 01:36:51.870900 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:51.870888 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pczgw" Jul 21 01:36:51.871357 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:51.870996 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vdpxq" podUID="7346c638-55dd-4d75-9d69-cead4209e04c" Jul 21 01:36:51.871357 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:51.871042 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pczgw" podUID="1527759a-5529-4c49-a9e5-cdc780b0b21a" Jul 21 01:36:51.871357 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:51.871096 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-24nj2" Jul 21 01:36:51.871357 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:51.871173 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-24nj2" podUID="17bdceb8-8016-4d0b-8b3c-f36507279428" Jul 21 01:36:52.776709 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.776676 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-183.ec2.internal" event="NodeReady" Jul 21 01:36:52.776895 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.776823 2580 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 21 01:36:52.817146 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.817109 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-b47d488dd-h2wcn"] Jul 21 01:36:52.818995 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.818979 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b47d488dd-h2wcn" Jul 21 01:36:52.821097 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.821061 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-r74bp\"" Jul 21 01:36:52.821283 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.821271 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 21 01:36:52.821594 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.821578 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 21 01:36:52.821702 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.821682 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 21 01:36:52.825061 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.825037 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-555d9d7885-wnjpr"] Jul 21 01:36:52.827102 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.826861 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-vxvqz"] Jul 21 01:36:52.827327 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.827306 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-wnjpr" Jul 21 01:36:52.828766 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.828748 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-6f7df7d7cb-gcx5w"] Jul 21 01:36:52.828902 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.828882 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-vxvqz" Jul 21 01:36:52.829675 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.829650 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-wxnxj\"" Jul 21 01:36:52.829802 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.829748 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 21 01:36:52.830520 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.830468 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-4mtp7"] Jul 21 01:36:52.830606 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.830565 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6f7df7d7cb-gcx5w" Jul 21 01:36:52.831238 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.831213 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 21 01:36:52.831742 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.831723 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 21 01:36:52.832074 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.832055 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 21 01:36:52.834161 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.832991 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 21 01:36:52.835611 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.835593 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-9ctn2"] Jul 21 01:36:52.835805 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.835735 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-4mtp7" Jul 21 01:36:52.836415 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.836335 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 21 01:36:52.836415 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.836358 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 21 01:36:52.836648 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.836628 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 21 01:36:52.836747 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.836700 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 21 01:36:52.836810 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.836768 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-tbvlf\"" Jul 21 01:36:52.837107 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.836969 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-qdcfq\"" Jul 21 01:36:52.837107 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.837009 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 21 01:36:52.837107 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.837040 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 21 01:36:52.837365 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.837271 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 21 01:36:52.837647 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.837629 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 21 01:36:52.837922 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.837907 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-9h5pm"] Jul 21 01:36:52.838042 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.838007 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-9ctn2" Jul 21 01:36:52.838831 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.838809 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 21 01:36:52.840144 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.840081 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-g86ww"] Jul 21 01:36:52.840241 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.840149 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-9h5pm" Jul 21 01:36:52.842576 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.842558 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 21 01:36:52.842901 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.842882 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-g86ww" Jul 21 01:36:52.843649 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.843609 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 21 01:36:52.843649 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.843621 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 21 01:36:52.844022 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.844005 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 21 01:36:52.844293 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.844253 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 21 01:36:52.844293 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.844266 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 21 01:36:52.845879 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.845425 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 21 01:36:52.845879 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.845502 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 21 01:36:52.845879 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.845557 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 21 01:36:52.845879 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.845557 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-727qm\"" Jul 21 01:36:52.845879 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.845604 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-fd6ng\"" Jul 21 01:36:52.849676 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.849652 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 21 01:36:52.851487 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.851435 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 21 01:36:52.853119 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.853080 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 21 01:36:52.853357 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.853308 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 21 01:36:52.854088 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.853521 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 21 01:36:52.854088 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.853759 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-d9brp\"" Jul 21 01:36:52.854088 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.854046 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-tsgng\"" Jul 21 01:36:52.854301 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.854285 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 21 01:36:52.855901 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.855877 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-b47d488dd-h2wcn"] Jul 21 01:36:52.856518 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.856500 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 21 01:36:52.856830 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.856809 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-wnjpr"] Jul 21 01:36:52.857644 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.857603 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-vxvqz"] Jul 21 01:36:52.858636 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.858621 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6f7df7d7cb-gcx5w"] Jul 21 01:36:52.860485 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.859506 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-4mtp7"] Jul 21 01:36:52.862208 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.861030 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-v899s"] Jul 21 01:36:52.864100 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.864076 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-2pscn"] Jul 21 01:36:52.865374 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.865354 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-v899s" Jul 21 01:36:52.867030 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.866968 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-bmgfc"] Jul 21 01:36:52.867238 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.867222 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-2pscn" Jul 21 01:36:52.868987 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.868626 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 21 01:36:52.868987 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.868867 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 21 01:36:52.872244 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.872225 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wcxpb\"" Jul 21 01:36:52.874126 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.873296 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 21 01:36:52.874126 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.873642 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 21 01:36:52.874126 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.873657 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 21 01:36:52.874126 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.873826 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-qtdww\"" Jul 21 01:36:52.874126 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.872376 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 21 01:36:52.877290 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.877266 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-dprcn"] Jul 21 01:36:52.877945 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.877929 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bmgfc" Jul 21 01:36:52.881588 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.881543 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-dprcn" Jul 21 01:36:52.886019 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.885788 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-wrwpn\"" Jul 21 01:36:52.886019 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.885866 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 21 01:36:52.886019 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.885911 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 21 01:36:52.886019 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.886010 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 21 01:36:52.886262 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.886053 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-g86ww"] Jul 21 01:36:52.886262 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.886060 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 21 01:36:52.886262 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.886098 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-9h5pm"] Jul 21 01:36:52.886262 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.886119 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-9ctn2"] Jul 21 01:36:52.886262 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.886126 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-dhmtc\"" Jul 21 01:36:52.886262 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.886138 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bmgfc"] Jul 21 01:36:52.886262 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.886166 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-v899s"] Jul 21 01:36:52.886262 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.886190 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-dprcn"] Jul 21 01:36:52.886262 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.886211 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-2pscn"] Jul 21 01:36:52.886709 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.886304 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 21 01:36:52.936226 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.936203 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d67dd362-e3a6-4ad1-9f9f-8f628ffce9a2-tmp\") pod \"insights-operator-555d9d7885-wnjpr\" (UID: \"d67dd362-e3a6-4ad1-9f9f-8f628ffce9a2\") " pod="openshift-insights/insights-operator-555d9d7885-wnjpr" Jul 21 01:36:52.936346 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.936232 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fd13d7ca-7c61-4c40-a86e-a58c1a7d1390-image-registry-private-configuration\") pod \"image-registry-b47d488dd-h2wcn\" (UID: \"fd13d7ca-7c61-4c40-a86e-a58c1a7d1390\") " pod="openshift-image-registry/image-registry-b47d488dd-h2wcn" Jul 21 01:36:52.936346 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.936253 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd13d7ca-7c61-4c40-a86e-a58c1a7d1390-registry-tls\") pod \"image-registry-b47d488dd-h2wcn\" (UID: \"fd13d7ca-7c61-4c40-a86e-a58c1a7d1390\") " pod="openshift-image-registry/image-registry-b47d488dd-h2wcn" Jul 21 01:36:52.936346 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.936279 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfmgw\" (UniqueName: \"kubernetes.io/projected/ea46ff20-2a04-4b13-92ed-371a4eb06987-kube-api-access-pfmgw\") pod \"volume-data-source-validator-78cf4c6f6-vxvqz\" (UID: \"ea46ff20-2a04-4b13-92ed-371a4eb06987\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-vxvqz" Jul 21 01:36:52.936530 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.936343 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nz2n9\" (UniqueName: \"kubernetes.io/projected/d67dd362-e3a6-4ad1-9f9f-8f628ffce9a2-kube-api-access-nz2n9\") pod \"insights-operator-555d9d7885-wnjpr\" (UID: \"d67dd362-e3a6-4ad1-9f9f-8f628ffce9a2\") " pod="openshift-insights/insights-operator-555d9d7885-wnjpr" Jul 21 01:36:52.936530 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.936378 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b7843343-f8dd-4ea6-a8f4-e3153a0b3ab2-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-g86ww\" (UID: \"b7843343-f8dd-4ea6-a8f4-e3153a0b3ab2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-g86ww" Jul 21 01:36:52.936530 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.936418 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d67dd362-e3a6-4ad1-9f9f-8f628ffce9a2-service-ca-bundle\") pod \"insights-operator-555d9d7885-wnjpr\" (UID: \"d67dd362-e3a6-4ad1-9f9f-8f628ffce9a2\") " pod="openshift-insights/insights-operator-555d9d7885-wnjpr" Jul 21 01:36:52.936530 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.936445 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/e6d9baf0-743b-48f5-9374-4443c65ea048-stats-auth\") pod \"router-default-6f7df7d7cb-gcx5w\" (UID: \"e6d9baf0-743b-48f5-9374-4443c65ea048\") " pod="openshift-ingress/router-default-6f7df7d7cb-gcx5w" Jul 21 01:36:52.936530 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.936498 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1e930d67-d711-419c-af02-86b43027a95d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-9ctn2\" (UID: \"1e930d67-d711-419c-af02-86b43027a95d\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-9ctn2" Jul 21 01:36:52.936752 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.936520 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lv8mb\" (UniqueName: \"kubernetes.io/projected/1e930d67-d711-419c-af02-86b43027a95d-kube-api-access-lv8mb\") pod \"cluster-monitoring-operator-76cf8966f8-9ctn2\" (UID: \"1e930d67-d711-419c-af02-86b43027a95d\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-9ctn2" Jul 21 01:36:52.936752 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.936555 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9c380009-9bf5-4b73-803a-5a1ea9c1abf0-serving-cert\") pod \"console-operator-565b7b569d-4mtp7\" (UID: \"9c380009-9bf5-4b73-803a-5a1ea9c1abf0\") " pod="openshift-console-operator/console-operator-565b7b569d-4mtp7" Jul 21 01:36:52.936752 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.936574 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6e08e8c3-dd3e-4792-8096-cf150dbcb58b-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-9h5pm\" (UID: \"6e08e8c3-dd3e-4792-8096-cf150dbcb58b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-9h5pm" Jul 21 01:36:52.936752 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.936608 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6e08e8c3-dd3e-4792-8096-cf150dbcb58b-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-9h5pm\" (UID: \"6e08e8c3-dd3e-4792-8096-cf150dbcb58b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-9h5pm" Jul 21 01:36:52.936752 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.936640 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9c380009-9bf5-4b73-803a-5a1ea9c1abf0-trusted-ca\") pod \"console-operator-565b7b569d-4mtp7\" (UID: \"9c380009-9bf5-4b73-803a-5a1ea9c1abf0\") " pod="openshift-console-operator/console-operator-565b7b569d-4mtp7" Jul 21 01:36:52.936752 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.936700 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d67dd362-e3a6-4ad1-9f9f-8f628ffce9a2-serving-cert\") pod \"insights-operator-555d9d7885-wnjpr\" (UID: \"d67dd362-e3a6-4ad1-9f9f-8f628ffce9a2\") " pod="openshift-insights/insights-operator-555d9d7885-wnjpr" Jul 21 01:36:52.936752 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.936738 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fd13d7ca-7c61-4c40-a86e-a58c1a7d1390-registry-certificates\") pod \"image-registry-b47d488dd-h2wcn\" (UID: \"fd13d7ca-7c61-4c40-a86e-a58c1a7d1390\") " pod="openshift-image-registry/image-registry-b47d488dd-h2wcn" Jul 21 01:36:52.937053 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.936805 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/1e930d67-d711-419c-af02-86b43027a95d-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-9ctn2\" (UID: \"1e930d67-d711-419c-af02-86b43027a95d\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-9ctn2" Jul 21 01:36:52.937053 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.936833 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d67dd362-e3a6-4ad1-9f9f-8f628ffce9a2-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-wnjpr\" (UID: \"d67dd362-e3a6-4ad1-9f9f-8f628ffce9a2\") " pod="openshift-insights/insights-operator-555d9d7885-wnjpr" Jul 21 01:36:52.937053 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.936860 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6d9baf0-743b-48f5-9374-4443c65ea048-metrics-certs\") pod \"router-default-6f7df7d7cb-gcx5w\" (UID: \"e6d9baf0-743b-48f5-9374-4443c65ea048\") " pod="openshift-ingress/router-default-6f7df7d7cb-gcx5w" Jul 21 01:36:52.937053 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.936887 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/e6d9baf0-743b-48f5-9374-4443c65ea048-default-certificate\") pod \"router-default-6f7df7d7cb-gcx5w\" (UID: \"e6d9baf0-743b-48f5-9374-4443c65ea048\") " pod="openshift-ingress/router-default-6f7df7d7cb-gcx5w" Jul 21 01:36:52.937053 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.936903 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr9b4\" (UniqueName: \"kubernetes.io/projected/b7843343-f8dd-4ea6-a8f4-e3153a0b3ab2-kube-api-access-dr9b4\") pod \"cluster-samples-operator-79b7c55f87-g86ww\" (UID: \"b7843343-f8dd-4ea6-a8f4-e3153a0b3ab2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-g86ww" Jul 21 01:36:52.937053 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.936938 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fd13d7ca-7c61-4c40-a86e-a58c1a7d1390-trusted-ca\") pod \"image-registry-b47d488dd-h2wcn\" (UID: \"fd13d7ca-7c61-4c40-a86e-a58c1a7d1390\") " pod="openshift-image-registry/image-registry-b47d488dd-h2wcn" Jul 21 01:36:52.937053 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.936973 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fd13d7ca-7c61-4c40-a86e-a58c1a7d1390-ca-trust-extracted\") pod \"image-registry-b47d488dd-h2wcn\" (UID: \"fd13d7ca-7c61-4c40-a86e-a58c1a7d1390\") " pod="openshift-image-registry/image-registry-b47d488dd-h2wcn" Jul 21 01:36:52.937053 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.937006 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvsfg\" (UniqueName: \"kubernetes.io/projected/e6d9baf0-743b-48f5-9374-4443c65ea048-kube-api-access-fvsfg\") pod \"router-default-6f7df7d7cb-gcx5w\" (UID: \"e6d9baf0-743b-48f5-9374-4443c65ea048\") " pod="openshift-ingress/router-default-6f7df7d7cb-gcx5w" Jul 21 01:36:52.937053 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.937026 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fd13d7ca-7c61-4c40-a86e-a58c1a7d1390-installation-pull-secrets\") pod \"image-registry-b47d488dd-h2wcn\" (UID: \"fd13d7ca-7c61-4c40-a86e-a58c1a7d1390\") " pod="openshift-image-registry/image-registry-b47d488dd-h2wcn" Jul 21 01:36:52.937053 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.937045 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9c380009-9bf5-4b73-803a-5a1ea9c1abf0-config\") pod \"console-operator-565b7b569d-4mtp7\" (UID: \"9c380009-9bf5-4b73-803a-5a1ea9c1abf0\") " pod="openshift-console-operator/console-operator-565b7b569d-4mtp7" Jul 21 01:36:52.937512 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.937063 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d67dd362-e3a6-4ad1-9f9f-8f628ffce9a2-snapshots\") pod \"insights-operator-555d9d7885-wnjpr\" (UID: \"d67dd362-e3a6-4ad1-9f9f-8f628ffce9a2\") " pod="openshift-insights/insights-operator-555d9d7885-wnjpr" Jul 21 01:36:52.937512 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.937106 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6d9baf0-743b-48f5-9374-4443c65ea048-service-ca-bundle\") pod \"router-default-6f7df7d7cb-gcx5w\" (UID: \"e6d9baf0-743b-48f5-9374-4443c65ea048\") " pod="openshift-ingress/router-default-6f7df7d7cb-gcx5w" Jul 21 01:36:52.937512 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.937158 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w94h\" (UniqueName: \"kubernetes.io/projected/9c380009-9bf5-4b73-803a-5a1ea9c1abf0-kube-api-access-5w94h\") pod \"console-operator-565b7b569d-4mtp7\" (UID: \"9c380009-9bf5-4b73-803a-5a1ea9c1abf0\") " pod="openshift-console-operator/console-operator-565b7b569d-4mtp7" Jul 21 01:36:52.937512 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.937240 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fd13d7ca-7c61-4c40-a86e-a58c1a7d1390-bound-sa-token\") pod \"image-registry-b47d488dd-h2wcn\" (UID: \"fd13d7ca-7c61-4c40-a86e-a58c1a7d1390\") " pod="openshift-image-registry/image-registry-b47d488dd-h2wcn" Jul 21 01:36:52.937512 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.937276 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45nhc\" (UniqueName: \"kubernetes.io/projected/6e08e8c3-dd3e-4792-8096-cf150dbcb58b-kube-api-access-45nhc\") pod \"kube-storage-version-migrator-operator-7658d9b749-9h5pm\" (UID: \"6e08e8c3-dd3e-4792-8096-cf150dbcb58b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-9h5pm" Jul 21 01:36:52.937512 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:52.937310 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrb8x\" (UniqueName: \"kubernetes.io/projected/fd13d7ca-7c61-4c40-a86e-a58c1a7d1390-kube-api-access-mrb8x\") pod \"image-registry-b47d488dd-h2wcn\" (UID: \"fd13d7ca-7c61-4c40-a86e-a58c1a7d1390\") " pod="openshift-image-registry/image-registry-b47d488dd-h2wcn" Jul 21 01:36:53.037677 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.037646 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d67dd362-e3a6-4ad1-9f9f-8f628ffce9a2-service-ca-bundle\") pod \"insights-operator-555d9d7885-wnjpr\" (UID: \"d67dd362-e3a6-4ad1-9f9f-8f628ffce9a2\") " pod="openshift-insights/insights-operator-555d9d7885-wnjpr" Jul 21 01:36:53.037677 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.037682 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/e6d9baf0-743b-48f5-9374-4443c65ea048-stats-auth\") pod \"router-default-6f7df7d7cb-gcx5w\" (UID: \"e6d9baf0-743b-48f5-9374-4443c65ea048\") " pod="openshift-ingress/router-default-6f7df7d7cb-gcx5w" Jul 21 01:36:53.037933 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.037708 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1e930d67-d711-419c-af02-86b43027a95d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-9ctn2\" (UID: \"1e930d67-d711-419c-af02-86b43027a95d\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-9ctn2" Jul 21 01:36:53.037933 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.037734 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lv8mb\" (UniqueName: \"kubernetes.io/projected/1e930d67-d711-419c-af02-86b43027a95d-kube-api-access-lv8mb\") pod \"cluster-monitoring-operator-76cf8966f8-9ctn2\" (UID: \"1e930d67-d711-419c-af02-86b43027a95d\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-9ctn2" Jul 21 01:36:53.037933 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.037761 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9c380009-9bf5-4b73-803a-5a1ea9c1abf0-serving-cert\") pod \"console-operator-565b7b569d-4mtp7\" (UID: \"9c380009-9bf5-4b73-803a-5a1ea9c1abf0\") " pod="openshift-console-operator/console-operator-565b7b569d-4mtp7" Jul 21 01:36:53.037933 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.037781 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6e08e8c3-dd3e-4792-8096-cf150dbcb58b-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-9h5pm\" (UID: \"6e08e8c3-dd3e-4792-8096-cf150dbcb58b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-9h5pm" Jul 21 01:36:53.037933 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.037801 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0add4762-9fef-4072-a05a-cdca30f4cdcf-metrics-tls\") pod \"dns-default-v899s\" (UID: \"0add4762-9fef-4072-a05a-cdca30f4cdcf\") " pod="openshift-dns/dns-default-v899s" Jul 21 01:36:53.037933 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:53.037831 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 21 01:36:53.037933 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.037860 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0add4762-9fef-4072-a05a-cdca30f4cdcf-tmp-dir\") pod \"dns-default-v899s\" (UID: \"0add4762-9fef-4072-a05a-cdca30f4cdcf\") " pod="openshift-dns/dns-default-v899s" Jul 21 01:36:53.037933 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.037907 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/de3854b8-6d90-46d9-9ab9-60a83f7a58be-serving-cert\") pod \"service-ca-operator-5bcbc94c78-2pscn\" (UID: \"de3854b8-6d90-46d9-9ab9-60a83f7a58be\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-2pscn" Jul 21 01:36:53.037933 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.037931 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1d402931-b366-48df-acda-2e144e7c6c70-cert\") pod \"ingress-canary-bmgfc\" (UID: \"1d402931-b366-48df-acda-2e144e7c6c70\") " pod="openshift-ingress-canary/ingress-canary-bmgfc" Jul 21 01:36:53.038357 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.037958 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6e08e8c3-dd3e-4792-8096-cf150dbcb58b-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-9h5pm\" (UID: \"6e08e8c3-dd3e-4792-8096-cf150dbcb58b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-9h5pm" Jul 21 01:36:53.038357 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.037985 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9c380009-9bf5-4b73-803a-5a1ea9c1abf0-trusted-ca\") pod \"console-operator-565b7b569d-4mtp7\" (UID: \"9c380009-9bf5-4b73-803a-5a1ea9c1abf0\") " pod="openshift-console-operator/console-operator-565b7b569d-4mtp7" Jul 21 01:36:53.038357 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.038019 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d67dd362-e3a6-4ad1-9f9f-8f628ffce9a2-serving-cert\") pod \"insights-operator-555d9d7885-wnjpr\" (UID: \"d67dd362-e3a6-4ad1-9f9f-8f628ffce9a2\") " pod="openshift-insights/insights-operator-555d9d7885-wnjpr" Jul 21 01:36:53.038357 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:53.038058 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e930d67-d711-419c-af02-86b43027a95d-cluster-monitoring-operator-tls podName:1e930d67-d711-419c-af02-86b43027a95d nodeName:}" failed. No retries permitted until 2026-07-21 01:36:53.538035823 +0000 UTC m=+37.125201784 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1e930d67-d711-419c-af02-86b43027a95d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-9ctn2" (UID: "1e930d67-d711-419c-af02-86b43027a95d") : secret "cluster-monitoring-operator-tls" not found Jul 21 01:36:53.038357 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.038085 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fd13d7ca-7c61-4c40-a86e-a58c1a7d1390-registry-certificates\") pod \"image-registry-b47d488dd-h2wcn\" (UID: \"fd13d7ca-7c61-4c40-a86e-a58c1a7d1390\") " pod="openshift-image-registry/image-registry-b47d488dd-h2wcn" Jul 21 01:36:53.038357 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.038119 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq668\" (UniqueName: \"kubernetes.io/projected/0add4762-9fef-4072-a05a-cdca30f4cdcf-kube-api-access-cq668\") pod \"dns-default-v899s\" (UID: \"0add4762-9fef-4072-a05a-cdca30f4cdcf\") " pod="openshift-dns/dns-default-v899s" Jul 21 01:36:53.038357 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.038155 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/1e930d67-d711-419c-af02-86b43027a95d-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-9ctn2\" (UID: \"1e930d67-d711-419c-af02-86b43027a95d\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-9ctn2" Jul 21 01:36:53.038357 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.038182 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d67dd362-e3a6-4ad1-9f9f-8f628ffce9a2-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-wnjpr\" (UID: \"d67dd362-e3a6-4ad1-9f9f-8f628ffce9a2\") " pod="openshift-insights/insights-operator-555d9d7885-wnjpr" Jul 21 01:36:53.038357 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.038208 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6d9baf0-743b-48f5-9374-4443c65ea048-metrics-certs\") pod \"router-default-6f7df7d7cb-gcx5w\" (UID: \"e6d9baf0-743b-48f5-9374-4443c65ea048\") " pod="openshift-ingress/router-default-6f7df7d7cb-gcx5w" Jul 21 01:36:53.038357 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.038240 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/e6d9baf0-743b-48f5-9374-4443c65ea048-default-certificate\") pod \"router-default-6f7df7d7cb-gcx5w\" (UID: \"e6d9baf0-743b-48f5-9374-4443c65ea048\") " pod="openshift-ingress/router-default-6f7df7d7cb-gcx5w" Jul 21 01:36:53.038357 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.038268 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr9b4\" (UniqueName: \"kubernetes.io/projected/b7843343-f8dd-4ea6-a8f4-e3153a0b3ab2-kube-api-access-dr9b4\") pod \"cluster-samples-operator-79b7c55f87-g86ww\" (UID: \"b7843343-f8dd-4ea6-a8f4-e3153a0b3ab2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-g86ww" Jul 21 01:36:53.038357 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.038294 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fd13d7ca-7c61-4c40-a86e-a58c1a7d1390-trusted-ca\") pod \"image-registry-b47d488dd-h2wcn\" (UID: \"fd13d7ca-7c61-4c40-a86e-a58c1a7d1390\") " pod="openshift-image-registry/image-registry-b47d488dd-h2wcn" Jul 21 01:36:53.038357 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.038320 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snklz\" (UniqueName: \"kubernetes.io/projected/cbbe27be-311c-40ea-9bdd-80b5fb532422-kube-api-access-snklz\") pod \"network-check-source-7bc5b6bf85-dprcn\" (UID: \"cbbe27be-311c-40ea-9bdd-80b5fb532422\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-dprcn" Jul 21 01:36:53.038357 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.038352 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d67dd362-e3a6-4ad1-9f9f-8f628ffce9a2-service-ca-bundle\") pod \"insights-operator-555d9d7885-wnjpr\" (UID: \"d67dd362-e3a6-4ad1-9f9f-8f628ffce9a2\") " pod="openshift-insights/insights-operator-555d9d7885-wnjpr" Jul 21 01:36:53.039075 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.038359 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fd13d7ca-7c61-4c40-a86e-a58c1a7d1390-ca-trust-extracted\") pod \"image-registry-b47d488dd-h2wcn\" (UID: \"fd13d7ca-7c61-4c40-a86e-a58c1a7d1390\") " pod="openshift-image-registry/image-registry-b47d488dd-h2wcn" Jul 21 01:36:53.039075 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.038411 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvsfg\" (UniqueName: \"kubernetes.io/projected/e6d9baf0-743b-48f5-9374-4443c65ea048-kube-api-access-fvsfg\") pod \"router-default-6f7df7d7cb-gcx5w\" (UID: \"e6d9baf0-743b-48f5-9374-4443c65ea048\") " pod="openshift-ingress/router-default-6f7df7d7cb-gcx5w" Jul 21 01:36:53.039075 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.038437 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fd13d7ca-7c61-4c40-a86e-a58c1a7d1390-installation-pull-secrets\") pod \"image-registry-b47d488dd-h2wcn\" (UID: \"fd13d7ca-7c61-4c40-a86e-a58c1a7d1390\") " pod="openshift-image-registry/image-registry-b47d488dd-h2wcn" Jul 21 01:36:53.039075 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.038492 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9c380009-9bf5-4b73-803a-5a1ea9c1abf0-config\") pod \"console-operator-565b7b569d-4mtp7\" (UID: \"9c380009-9bf5-4b73-803a-5a1ea9c1abf0\") " pod="openshift-console-operator/console-operator-565b7b569d-4mtp7" Jul 21 01:36:53.039075 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.038518 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d67dd362-e3a6-4ad1-9f9f-8f628ffce9a2-snapshots\") pod \"insights-operator-555d9d7885-wnjpr\" (UID: \"d67dd362-e3a6-4ad1-9f9f-8f628ffce9a2\") " pod="openshift-insights/insights-operator-555d9d7885-wnjpr" Jul 21 01:36:53.039075 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.038543 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6d9baf0-743b-48f5-9374-4443c65ea048-service-ca-bundle\") pod \"router-default-6f7df7d7cb-gcx5w\" (UID: \"e6d9baf0-743b-48f5-9374-4443c65ea048\") " pod="openshift-ingress/router-default-6f7df7d7cb-gcx5w" Jul 21 01:36:53.039075 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.038571 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w94h\" (UniqueName: \"kubernetes.io/projected/9c380009-9bf5-4b73-803a-5a1ea9c1abf0-kube-api-access-5w94h\") pod \"console-operator-565b7b569d-4mtp7\" (UID: \"9c380009-9bf5-4b73-803a-5a1ea9c1abf0\") " pod="openshift-console-operator/console-operator-565b7b569d-4mtp7" Jul 21 01:36:53.039075 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.038595 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0add4762-9fef-4072-a05a-cdca30f4cdcf-config-volume\") pod \"dns-default-v899s\" (UID: \"0add4762-9fef-4072-a05a-cdca30f4cdcf\") " pod="openshift-dns/dns-default-v899s" Jul 21 01:36:53.039075 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.038599 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6e08e8c3-dd3e-4792-8096-cf150dbcb58b-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-9h5pm\" (UID: \"6e08e8c3-dd3e-4792-8096-cf150dbcb58b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-9h5pm" Jul 21 01:36:53.039075 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.038625 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzxhs\" (UniqueName: \"kubernetes.io/projected/1d402931-b366-48df-acda-2e144e7c6c70-kube-api-access-jzxhs\") pod \"ingress-canary-bmgfc\" (UID: \"1d402931-b366-48df-acda-2e144e7c6c70\") " pod="openshift-ingress-canary/ingress-canary-bmgfc" Jul 21 01:36:53.039075 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.038668 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fd13d7ca-7c61-4c40-a86e-a58c1a7d1390-bound-sa-token\") pod \"image-registry-b47d488dd-h2wcn\" (UID: \"fd13d7ca-7c61-4c40-a86e-a58c1a7d1390\") " pod="openshift-image-registry/image-registry-b47d488dd-h2wcn" Jul 21 01:36:53.039075 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.038686 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fd13d7ca-7c61-4c40-a86e-a58c1a7d1390-registry-certificates\") pod \"image-registry-b47d488dd-h2wcn\" (UID: \"fd13d7ca-7c61-4c40-a86e-a58c1a7d1390\") " pod="openshift-image-registry/image-registry-b47d488dd-h2wcn" Jul 21 01:36:53.039075 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.038726 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45nhc\" (UniqueName: \"kubernetes.io/projected/6e08e8c3-dd3e-4792-8096-cf150dbcb58b-kube-api-access-45nhc\") pod \"kube-storage-version-migrator-operator-7658d9b749-9h5pm\" (UID: \"6e08e8c3-dd3e-4792-8096-cf150dbcb58b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-9h5pm" Jul 21 01:36:53.039075 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.039058 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrb8x\" (UniqueName: \"kubernetes.io/projected/fd13d7ca-7c61-4c40-a86e-a58c1a7d1390-kube-api-access-mrb8x\") pod \"image-registry-b47d488dd-h2wcn\" (UID: \"fd13d7ca-7c61-4c40-a86e-a58c1a7d1390\") " pod="openshift-image-registry/image-registry-b47d488dd-h2wcn" Jul 21 01:36:53.039755 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.039090 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d67dd362-e3a6-4ad1-9f9f-8f628ffce9a2-tmp\") pod \"insights-operator-555d9d7885-wnjpr\" (UID: \"d67dd362-e3a6-4ad1-9f9f-8f628ffce9a2\") " pod="openshift-insights/insights-operator-555d9d7885-wnjpr" Jul 21 01:36:53.039755 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.039113 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9c380009-9bf5-4b73-803a-5a1ea9c1abf0-config\") pod \"console-operator-565b7b569d-4mtp7\" (UID: \"9c380009-9bf5-4b73-803a-5a1ea9c1abf0\") " pod="openshift-console-operator/console-operator-565b7b569d-4mtp7" Jul 21 01:36:53.039755 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.039117 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fd13d7ca-7c61-4c40-a86e-a58c1a7d1390-image-registry-private-configuration\") pod \"image-registry-b47d488dd-h2wcn\" (UID: \"fd13d7ca-7c61-4c40-a86e-a58c1a7d1390\") " pod="openshift-image-registry/image-registry-b47d488dd-h2wcn" Jul 21 01:36:53.039755 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.039157 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd13d7ca-7c61-4c40-a86e-a58c1a7d1390-registry-tls\") pod \"image-registry-b47d488dd-h2wcn\" (UID: \"fd13d7ca-7c61-4c40-a86e-a58c1a7d1390\") " pod="openshift-image-registry/image-registry-b47d488dd-h2wcn" Jul 21 01:36:53.039755 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:53.039186 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 21 01:36:53.039755 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:53.039235 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6d9baf0-743b-48f5-9374-4443c65ea048-metrics-certs podName:e6d9baf0-743b-48f5-9374-4443c65ea048 nodeName:}" failed. No retries permitted until 2026-07-21 01:36:53.539219137 +0000 UTC m=+37.126385108 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e6d9baf0-743b-48f5-9374-4443c65ea048-metrics-certs") pod "router-default-6f7df7d7cb-gcx5w" (UID: "e6d9baf0-743b-48f5-9374-4443c65ea048") : secret "router-metrics-certs-default" not found Jul 21 01:36:53.039755 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.039311 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fd13d7ca-7c61-4c40-a86e-a58c1a7d1390-trusted-ca\") pod \"image-registry-b47d488dd-h2wcn\" (UID: \"fd13d7ca-7c61-4c40-a86e-a58c1a7d1390\") " pod="openshift-image-registry/image-registry-b47d488dd-h2wcn" Jul 21 01:36:53.039755 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:53.039440 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 21 01:36:53.039755 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:53.039469 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b47d488dd-h2wcn: secret "image-registry-tls" not found Jul 21 01:36:53.039755 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:53.039536 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd13d7ca-7c61-4c40-a86e-a58c1a7d1390-registry-tls podName:fd13d7ca-7c61-4c40-a86e-a58c1a7d1390 nodeName:}" failed. No retries permitted until 2026-07-21 01:36:53.539523516 +0000 UTC m=+37.126689471 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fd13d7ca-7c61-4c40-a86e-a58c1a7d1390-registry-tls") pod "image-registry-b47d488dd-h2wcn" (UID: "fd13d7ca-7c61-4c40-a86e-a58c1a7d1390") : secret "image-registry-tls" not found Jul 21 01:36:53.039755 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.039311 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9c380009-9bf5-4b73-803a-5a1ea9c1abf0-trusted-ca\") pod \"console-operator-565b7b569d-4mtp7\" (UID: \"9c380009-9bf5-4b73-803a-5a1ea9c1abf0\") " pod="openshift-console-operator/console-operator-565b7b569d-4mtp7" Jul 21 01:36:53.040268 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.039802 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d67dd362-e3a6-4ad1-9f9f-8f628ffce9a2-tmp\") pod \"insights-operator-555d9d7885-wnjpr\" (UID: \"d67dd362-e3a6-4ad1-9f9f-8f628ffce9a2\") " pod="openshift-insights/insights-operator-555d9d7885-wnjpr" Jul 21 01:36:53.040268 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.039186 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfmgw\" (UniqueName: \"kubernetes.io/projected/ea46ff20-2a04-4b13-92ed-371a4eb06987-kube-api-access-pfmgw\") pod \"volume-data-source-validator-78cf4c6f6-vxvqz\" (UID: \"ea46ff20-2a04-4b13-92ed-371a4eb06987\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-vxvqz" Jul 21 01:36:53.040268 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.039863 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w5h6\" (UniqueName: \"kubernetes.io/projected/de3854b8-6d90-46d9-9ab9-60a83f7a58be-kube-api-access-7w5h6\") pod \"service-ca-operator-5bcbc94c78-2pscn\" (UID: \"de3854b8-6d90-46d9-9ab9-60a83f7a58be\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-2pscn" Jul 21 01:36:53.040268 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.039897 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz2n9\" (UniqueName: \"kubernetes.io/projected/d67dd362-e3a6-4ad1-9f9f-8f628ffce9a2-kube-api-access-nz2n9\") pod \"insights-operator-555d9d7885-wnjpr\" (UID: \"d67dd362-e3a6-4ad1-9f9f-8f628ffce9a2\") " pod="openshift-insights/insights-operator-555d9d7885-wnjpr" Jul 21 01:36:53.040268 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.039924 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b7843343-f8dd-4ea6-a8f4-e3153a0b3ab2-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-g86ww\" (UID: \"b7843343-f8dd-4ea6-a8f4-e3153a0b3ab2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-g86ww" Jul 21 01:36:53.040268 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.039970 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/1e930d67-d711-419c-af02-86b43027a95d-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-9ctn2\" (UID: \"1e930d67-d711-419c-af02-86b43027a95d\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-9ctn2" Jul 21 01:36:53.040268 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.040007 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/de3854b8-6d90-46d9-9ab9-60a83f7a58be-config\") pod \"service-ca-operator-5bcbc94c78-2pscn\" (UID: \"de3854b8-6d90-46d9-9ab9-60a83f7a58be\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-2pscn" Jul 21 01:36:53.040268 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:53.040067 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 21 01:36:53.040268 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.040079 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d67dd362-e3a6-4ad1-9f9f-8f628ffce9a2-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-wnjpr\" (UID: \"d67dd362-e3a6-4ad1-9f9f-8f628ffce9a2\") " pod="openshift-insights/insights-operator-555d9d7885-wnjpr" Jul 21 01:36:53.040268 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:53.040110 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7843343-f8dd-4ea6-a8f4-e3153a0b3ab2-samples-operator-tls podName:b7843343-f8dd-4ea6-a8f4-e3153a0b3ab2 nodeName:}" failed. No retries permitted until 2026-07-21 01:36:53.540096394 +0000 UTC m=+37.127262365 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b7843343-f8dd-4ea6-a8f4-e3153a0b3ab2-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-g86ww" (UID: "b7843343-f8dd-4ea6-a8f4-e3153a0b3ab2") : secret "samples-operator-tls" not found Jul 21 01:36:53.040813 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.040581 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d67dd362-e3a6-4ad1-9f9f-8f628ffce9a2-snapshots\") pod \"insights-operator-555d9d7885-wnjpr\" (UID: \"d67dd362-e3a6-4ad1-9f9f-8f628ffce9a2\") " pod="openshift-insights/insights-operator-555d9d7885-wnjpr" Jul 21 01:36:53.041316 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.041044 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fd13d7ca-7c61-4c40-a86e-a58c1a7d1390-ca-trust-extracted\") pod \"image-registry-b47d488dd-h2wcn\" (UID: \"fd13d7ca-7c61-4c40-a86e-a58c1a7d1390\") " pod="openshift-image-registry/image-registry-b47d488dd-h2wcn" Jul 21 01:36:53.041316 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:53.041306 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e6d9baf0-743b-48f5-9374-4443c65ea048-service-ca-bundle podName:e6d9baf0-743b-48f5-9374-4443c65ea048 nodeName:}" failed. No retries permitted until 2026-07-21 01:36:53.541289382 +0000 UTC m=+37.128455361 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e6d9baf0-743b-48f5-9374-4443c65ea048-service-ca-bundle") pod "router-default-6f7df7d7cb-gcx5w" (UID: "e6d9baf0-743b-48f5-9374-4443c65ea048") : configmap references non-existent config key: service-ca.crt Jul 21 01:36:53.043626 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.043576 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6e08e8c3-dd3e-4792-8096-cf150dbcb58b-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-9h5pm\" (UID: \"6e08e8c3-dd3e-4792-8096-cf150dbcb58b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-9h5pm" Jul 21 01:36:53.044151 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.043730 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fd13d7ca-7c61-4c40-a86e-a58c1a7d1390-installation-pull-secrets\") pod \"image-registry-b47d488dd-h2wcn\" (UID: \"fd13d7ca-7c61-4c40-a86e-a58c1a7d1390\") " pod="openshift-image-registry/image-registry-b47d488dd-h2wcn" Jul 21 01:36:53.044151 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.043800 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/e6d9baf0-743b-48f5-9374-4443c65ea048-default-certificate\") pod \"router-default-6f7df7d7cb-gcx5w\" (UID: \"e6d9baf0-743b-48f5-9374-4443c65ea048\") " pod="openshift-ingress/router-default-6f7df7d7cb-gcx5w" Jul 21 01:36:53.044151 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.044049 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9c380009-9bf5-4b73-803a-5a1ea9c1abf0-serving-cert\") pod \"console-operator-565b7b569d-4mtp7\" (UID: \"9c380009-9bf5-4b73-803a-5a1ea9c1abf0\") " pod="openshift-console-operator/console-operator-565b7b569d-4mtp7" Jul 21 01:36:53.044341 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.044184 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fd13d7ca-7c61-4c40-a86e-a58c1a7d1390-image-registry-private-configuration\") pod \"image-registry-b47d488dd-h2wcn\" (UID: \"fd13d7ca-7c61-4c40-a86e-a58c1a7d1390\") " pod="openshift-image-registry/image-registry-b47d488dd-h2wcn" Jul 21 01:36:53.044505 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.044484 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/e6d9baf0-743b-48f5-9374-4443c65ea048-stats-auth\") pod \"router-default-6f7df7d7cb-gcx5w\" (UID: \"e6d9baf0-743b-48f5-9374-4443c65ea048\") " pod="openshift-ingress/router-default-6f7df7d7cb-gcx5w" Jul 21 01:36:53.044670 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.044652 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d67dd362-e3a6-4ad1-9f9f-8f628ffce9a2-serving-cert\") pod \"insights-operator-555d9d7885-wnjpr\" (UID: \"d67dd362-e3a6-4ad1-9f9f-8f628ffce9a2\") " pod="openshift-insights/insights-operator-555d9d7885-wnjpr" Jul 21 01:36:53.046610 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.046585 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lv8mb\" (UniqueName: \"kubernetes.io/projected/1e930d67-d711-419c-af02-86b43027a95d-kube-api-access-lv8mb\") pod \"cluster-monitoring-operator-76cf8966f8-9ctn2\" (UID: \"1e930d67-d711-419c-af02-86b43027a95d\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-9ctn2" Jul 21 01:36:53.052678 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.052316 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvsfg\" (UniqueName: \"kubernetes.io/projected/e6d9baf0-743b-48f5-9374-4443c65ea048-kube-api-access-fvsfg\") pod \"router-default-6f7df7d7cb-gcx5w\" (UID: \"e6d9baf0-743b-48f5-9374-4443c65ea048\") " pod="openshift-ingress/router-default-6f7df7d7cb-gcx5w" Jul 21 01:36:53.052678 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.052529 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfmgw\" (UniqueName: \"kubernetes.io/projected/ea46ff20-2a04-4b13-92ed-371a4eb06987-kube-api-access-pfmgw\") pod \"volume-data-source-validator-78cf4c6f6-vxvqz\" (UID: \"ea46ff20-2a04-4b13-92ed-371a4eb06987\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-vxvqz" Jul 21 01:36:53.052678 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.052659 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr9b4\" (UniqueName: \"kubernetes.io/projected/b7843343-f8dd-4ea6-a8f4-e3153a0b3ab2-kube-api-access-dr9b4\") pod \"cluster-samples-operator-79b7c55f87-g86ww\" (UID: \"b7843343-f8dd-4ea6-a8f4-e3153a0b3ab2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-g86ww" Jul 21 01:36:53.052901 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.052690 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrb8x\" (UniqueName: \"kubernetes.io/projected/fd13d7ca-7c61-4c40-a86e-a58c1a7d1390-kube-api-access-mrb8x\") pod \"image-registry-b47d488dd-h2wcn\" (UID: \"fd13d7ca-7c61-4c40-a86e-a58c1a7d1390\") " pod="openshift-image-registry/image-registry-b47d488dd-h2wcn" Jul 21 01:36:53.052901 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.052790 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w94h\" (UniqueName: \"kubernetes.io/projected/9c380009-9bf5-4b73-803a-5a1ea9c1abf0-kube-api-access-5w94h\") pod \"console-operator-565b7b569d-4mtp7\" (UID: \"9c380009-9bf5-4b73-803a-5a1ea9c1abf0\") " pod="openshift-console-operator/console-operator-565b7b569d-4mtp7" Jul 21 01:36:53.053539 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.053506 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz2n9\" (UniqueName: \"kubernetes.io/projected/d67dd362-e3a6-4ad1-9f9f-8f628ffce9a2-kube-api-access-nz2n9\") pod \"insights-operator-555d9d7885-wnjpr\" (UID: \"d67dd362-e3a6-4ad1-9f9f-8f628ffce9a2\") " pod="openshift-insights/insights-operator-555d9d7885-wnjpr" Jul 21 01:36:53.054576 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.054553 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fd13d7ca-7c61-4c40-a86e-a58c1a7d1390-bound-sa-token\") pod \"image-registry-b47d488dd-h2wcn\" (UID: \"fd13d7ca-7c61-4c40-a86e-a58c1a7d1390\") " pod="openshift-image-registry/image-registry-b47d488dd-h2wcn" Jul 21 01:36:53.054831 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.054812 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45nhc\" (UniqueName: \"kubernetes.io/projected/6e08e8c3-dd3e-4792-8096-cf150dbcb58b-kube-api-access-45nhc\") pod \"kube-storage-version-migrator-operator-7658d9b749-9h5pm\" (UID: \"6e08e8c3-dd3e-4792-8096-cf150dbcb58b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-9h5pm" Jul 21 01:36:53.140472 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.140418 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w5h6\" (UniqueName: \"kubernetes.io/projected/de3854b8-6d90-46d9-9ab9-60a83f7a58be-kube-api-access-7w5h6\") pod \"service-ca-operator-5bcbc94c78-2pscn\" (UID: \"de3854b8-6d90-46d9-9ab9-60a83f7a58be\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-2pscn" Jul 21 01:36:53.140648 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.140511 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/de3854b8-6d90-46d9-9ab9-60a83f7a58be-config\") pod \"service-ca-operator-5bcbc94c78-2pscn\" (UID: \"de3854b8-6d90-46d9-9ab9-60a83f7a58be\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-2pscn" Jul 21 01:36:53.140648 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.140565 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0add4762-9fef-4072-a05a-cdca30f4cdcf-metrics-tls\") pod \"dns-default-v899s\" (UID: \"0add4762-9fef-4072-a05a-cdca30f4cdcf\") " pod="openshift-dns/dns-default-v899s" Jul 21 01:36:53.140648 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.140587 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0add4762-9fef-4072-a05a-cdca30f4cdcf-tmp-dir\") pod \"dns-default-v899s\" (UID: \"0add4762-9fef-4072-a05a-cdca30f4cdcf\") " pod="openshift-dns/dns-default-v899s" Jul 21 01:36:53.140648 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.140624 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/de3854b8-6d90-46d9-9ab9-60a83f7a58be-serving-cert\") pod \"service-ca-operator-5bcbc94c78-2pscn\" (UID: \"de3854b8-6d90-46d9-9ab9-60a83f7a58be\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-2pscn" Jul 21 01:36:53.140648 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.140645 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1d402931-b366-48df-acda-2e144e7c6c70-cert\") pod \"ingress-canary-bmgfc\" (UID: \"1d402931-b366-48df-acda-2e144e7c6c70\") " pod="openshift-ingress-canary/ingress-canary-bmgfc" Jul 21 01:36:53.140890 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.140680 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq668\" (UniqueName: \"kubernetes.io/projected/0add4762-9fef-4072-a05a-cdca30f4cdcf-kube-api-access-cq668\") pod \"dns-default-v899s\" (UID: \"0add4762-9fef-4072-a05a-cdca30f4cdcf\") " pod="openshift-dns/dns-default-v899s" Jul 21 01:36:53.140890 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:53.140698 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 01:36:53.140890 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.140723 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snklz\" (UniqueName: \"kubernetes.io/projected/cbbe27be-311c-40ea-9bdd-80b5fb532422-kube-api-access-snklz\") pod \"network-check-source-7bc5b6bf85-dprcn\" (UID: \"cbbe27be-311c-40ea-9bdd-80b5fb532422\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-dprcn" Jul 21 01:36:53.140890 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:53.140768 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0add4762-9fef-4072-a05a-cdca30f4cdcf-metrics-tls podName:0add4762-9fef-4072-a05a-cdca30f4cdcf nodeName:}" failed. No retries permitted until 2026-07-21 01:36:53.640746018 +0000 UTC m=+37.227911978 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0add4762-9fef-4072-a05a-cdca30f4cdcf-metrics-tls") pod "dns-default-v899s" (UID: "0add4762-9fef-4072-a05a-cdca30f4cdcf") : secret "dns-default-metrics-tls" not found Jul 21 01:36:53.140890 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.140818 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0add4762-9fef-4072-a05a-cdca30f4cdcf-config-volume\") pod \"dns-default-v899s\" (UID: \"0add4762-9fef-4072-a05a-cdca30f4cdcf\") " pod="openshift-dns/dns-default-v899s" Jul 21 01:36:53.140890 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.140847 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzxhs\" (UniqueName: \"kubernetes.io/projected/1d402931-b366-48df-acda-2e144e7c6c70-kube-api-access-jzxhs\") pod \"ingress-canary-bmgfc\" (UID: \"1d402931-b366-48df-acda-2e144e7c6c70\") " pod="openshift-ingress-canary/ingress-canary-bmgfc" Jul 21 01:36:53.141312 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.141246 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/de3854b8-6d90-46d9-9ab9-60a83f7a58be-config\") pod \"service-ca-operator-5bcbc94c78-2pscn\" (UID: \"de3854b8-6d90-46d9-9ab9-60a83f7a58be\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-2pscn" Jul 21 01:36:53.141312 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.141248 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0add4762-9fef-4072-a05a-cdca30f4cdcf-tmp-dir\") pod \"dns-default-v899s\" (UID: \"0add4762-9fef-4072-a05a-cdca30f4cdcf\") " pod="openshift-dns/dns-default-v899s" Jul 21 01:36:53.141509 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.141385 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0add4762-9fef-4072-a05a-cdca30f4cdcf-config-volume\") pod \"dns-default-v899s\" (UID: \"0add4762-9fef-4072-a05a-cdca30f4cdcf\") " pod="openshift-dns/dns-default-v899s" Jul 21 01:36:53.143609 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.143585 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/de3854b8-6d90-46d9-9ab9-60a83f7a58be-serving-cert\") pod \"service-ca-operator-5bcbc94c78-2pscn\" (UID: \"de3854b8-6d90-46d9-9ab9-60a83f7a58be\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-2pscn" Jul 21 01:36:53.143726 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.143704 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1d402931-b366-48df-acda-2e144e7c6c70-cert\") pod \"ingress-canary-bmgfc\" (UID: \"1d402931-b366-48df-acda-2e144e7c6c70\") " pod="openshift-ingress-canary/ingress-canary-bmgfc" Jul 21 01:36:53.143836 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.143823 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-wnjpr" Jul 21 01:36:53.149068 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.149030 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w5h6\" (UniqueName: \"kubernetes.io/projected/de3854b8-6d90-46d9-9ab9-60a83f7a58be-kube-api-access-7w5h6\") pod \"service-ca-operator-5bcbc94c78-2pscn\" (UID: \"de3854b8-6d90-46d9-9ab9-60a83f7a58be\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-2pscn" Jul 21 01:36:53.149607 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.149564 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq668\" (UniqueName: \"kubernetes.io/projected/0add4762-9fef-4072-a05a-cdca30f4cdcf-kube-api-access-cq668\") pod \"dns-default-v899s\" (UID: \"0add4762-9fef-4072-a05a-cdca30f4cdcf\") " pod="openshift-dns/dns-default-v899s" Jul 21 01:36:53.149987 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.149963 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzxhs\" (UniqueName: \"kubernetes.io/projected/1d402931-b366-48df-acda-2e144e7c6c70-kube-api-access-jzxhs\") pod \"ingress-canary-bmgfc\" (UID: \"1d402931-b366-48df-acda-2e144e7c6c70\") " pod="openshift-ingress-canary/ingress-canary-bmgfc" Jul 21 01:36:53.150254 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.150233 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-snklz\" (UniqueName: \"kubernetes.io/projected/cbbe27be-311c-40ea-9bdd-80b5fb532422-kube-api-access-snklz\") pod \"network-check-source-7bc5b6bf85-dprcn\" (UID: \"cbbe27be-311c-40ea-9bdd-80b5fb532422\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-dprcn" Jul 21 01:36:53.157740 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.157680 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-vxvqz" Jul 21 01:36:53.181854 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.181824 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-4mtp7" Jul 21 01:36:53.197768 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.197742 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-dprcn" Jul 21 01:36:53.205609 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.205586 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-9h5pm" Jul 21 01:36:53.225686 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.225658 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-2pscn" Jul 21 01:36:53.232504 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.232482 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bmgfc" Jul 21 01:36:53.543510 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.543473 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6d9baf0-743b-48f5-9374-4443c65ea048-metrics-certs\") pod \"router-default-6f7df7d7cb-gcx5w\" (UID: \"e6d9baf0-743b-48f5-9374-4443c65ea048\") " pod="openshift-ingress/router-default-6f7df7d7cb-gcx5w" Jul 21 01:36:53.543695 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.543532 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6d9baf0-743b-48f5-9374-4443c65ea048-service-ca-bundle\") pod \"router-default-6f7df7d7cb-gcx5w\" (UID: \"e6d9baf0-743b-48f5-9374-4443c65ea048\") " pod="openshift-ingress/router-default-6f7df7d7cb-gcx5w" Jul 21 01:36:53.543695 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.543577 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd13d7ca-7c61-4c40-a86e-a58c1a7d1390-registry-tls\") pod \"image-registry-b47d488dd-h2wcn\" (UID: \"fd13d7ca-7c61-4c40-a86e-a58c1a7d1390\") " pod="openshift-image-registry/image-registry-b47d488dd-h2wcn" Jul 21 01:36:53.543695 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.543612 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b7843343-f8dd-4ea6-a8f4-e3153a0b3ab2-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-g86ww\" (UID: \"b7843343-f8dd-4ea6-a8f4-e3153a0b3ab2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-g86ww" Jul 21 01:36:53.543695 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:53.543679 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e6d9baf0-743b-48f5-9374-4443c65ea048-service-ca-bundle podName:e6d9baf0-743b-48f5-9374-4443c65ea048 nodeName:}" failed. No retries permitted until 2026-07-21 01:36:54.543655798 +0000 UTC m=+38.130821762 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e6d9baf0-743b-48f5-9374-4443c65ea048-service-ca-bundle") pod "router-default-6f7df7d7cb-gcx5w" (UID: "e6d9baf0-743b-48f5-9374-4443c65ea048") : configmap references non-existent config key: service-ca.crt Jul 21 01:36:53.543899 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:53.543704 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 21 01:36:53.543899 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:53.543720 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 21 01:36:53.543899 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:53.543741 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b47d488dd-h2wcn: secret "image-registry-tls" not found Jul 21 01:36:53.543899 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:53.543760 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6d9baf0-743b-48f5-9374-4443c65ea048-metrics-certs podName:e6d9baf0-743b-48f5-9374-4443c65ea048 nodeName:}" failed. No retries permitted until 2026-07-21 01:36:54.54374366 +0000 UTC m=+38.130909634 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e6d9baf0-743b-48f5-9374-4443c65ea048-metrics-certs") pod "router-default-6f7df7d7cb-gcx5w" (UID: "e6d9baf0-743b-48f5-9374-4443c65ea048") : secret "router-metrics-certs-default" not found Jul 21 01:36:53.543899 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.543718 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1e930d67-d711-419c-af02-86b43027a95d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-9ctn2\" (UID: \"1e930d67-d711-419c-af02-86b43027a95d\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-9ctn2" Jul 21 01:36:53.543899 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:53.543781 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd13d7ca-7c61-4c40-a86e-a58c1a7d1390-registry-tls podName:fd13d7ca-7c61-4c40-a86e-a58c1a7d1390 nodeName:}" failed. No retries permitted until 2026-07-21 01:36:54.543768044 +0000 UTC m=+38.130934027 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fd13d7ca-7c61-4c40-a86e-a58c1a7d1390-registry-tls") pod "image-registry-b47d488dd-h2wcn" (UID: "fd13d7ca-7c61-4c40-a86e-a58c1a7d1390") : secret "image-registry-tls" not found Jul 21 01:36:53.543899 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:53.543784 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 21 01:36:53.543899 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:53.543724 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 21 01:36:53.543899 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:53.543822 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e930d67-d711-419c-af02-86b43027a95d-cluster-monitoring-operator-tls podName:1e930d67-d711-419c-af02-86b43027a95d nodeName:}" failed. No retries permitted until 2026-07-21 01:36:54.543810584 +0000 UTC m=+38.130976541 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1e930d67-d711-419c-af02-86b43027a95d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-9ctn2" (UID: "1e930d67-d711-419c-af02-86b43027a95d") : secret "cluster-monitoring-operator-tls" not found Jul 21 01:36:53.543899 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:53.543859 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7843343-f8dd-4ea6-a8f4-e3153a0b3ab2-samples-operator-tls podName:b7843343-f8dd-4ea6-a8f4-e3153a0b3ab2 nodeName:}" failed. No retries permitted until 2026-07-21 01:36:54.543832815 +0000 UTC m=+38.130998785 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b7843343-f8dd-4ea6-a8f4-e3153a0b3ab2-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-g86ww" (UID: "b7843343-f8dd-4ea6-a8f4-e3153a0b3ab2") : secret "samples-operator-tls" not found Jul 21 01:36:53.644842 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.644804 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0add4762-9fef-4072-a05a-cdca30f4cdcf-metrics-tls\") pod \"dns-default-v899s\" (UID: \"0add4762-9fef-4072-a05a-cdca30f4cdcf\") " pod="openshift-dns/dns-default-v899s" Jul 21 01:36:53.645017 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:53.644972 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 01:36:53.645059 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:53.645042 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0add4762-9fef-4072-a05a-cdca30f4cdcf-metrics-tls podName:0add4762-9fef-4072-a05a-cdca30f4cdcf nodeName:}" failed. No retries permitted until 2026-07-21 01:36:54.645024485 +0000 UTC m=+38.232190441 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0add4762-9fef-4072-a05a-cdca30f4cdcf-metrics-tls") pod "dns-default-v899s" (UID: "0add4762-9fef-4072-a05a-cdca30f4cdcf") : secret "dns-default-metrics-tls" not found Jul 21 01:36:53.870509 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.870405 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vdpxq" Jul 21 01:36:53.870509 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.870439 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pczgw" Jul 21 01:36:53.870509 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.870439 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-24nj2" Jul 21 01:36:53.873171 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.873148 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 21 01:36:53.873171 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.873163 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5qdkq\"" Jul 21 01:36:53.873171 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.873170 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 21 01:36:53.873803 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:53.873234 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-8s87n\"" Jul 21 01:36:54.553982 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:54.553943 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1e930d67-d711-419c-af02-86b43027a95d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-9ctn2\" (UID: \"1e930d67-d711-419c-af02-86b43027a95d\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-9ctn2" Jul 21 01:36:54.554158 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:54.554050 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6d9baf0-743b-48f5-9374-4443c65ea048-metrics-certs\") pod \"router-default-6f7df7d7cb-gcx5w\" (UID: \"e6d9baf0-743b-48f5-9374-4443c65ea048\") " pod="openshift-ingress/router-default-6f7df7d7cb-gcx5w" Jul 21 01:36:54.554158 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:54.554087 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6d9baf0-743b-48f5-9374-4443c65ea048-service-ca-bundle\") pod \"router-default-6f7df7d7cb-gcx5w\" (UID: \"e6d9baf0-743b-48f5-9374-4443c65ea048\") " pod="openshift-ingress/router-default-6f7df7d7cb-gcx5w" Jul 21 01:36:54.554158 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:54.554120 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 21 01:36:54.554313 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:54.554187 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e930d67-d711-419c-af02-86b43027a95d-cluster-monitoring-operator-tls podName:1e930d67-d711-419c-af02-86b43027a95d nodeName:}" failed. No retries permitted until 2026-07-21 01:36:56.554166556 +0000 UTC m=+40.141332515 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1e930d67-d711-419c-af02-86b43027a95d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-9ctn2" (UID: "1e930d67-d711-419c-af02-86b43027a95d") : secret "cluster-monitoring-operator-tls" not found Jul 21 01:36:54.554313 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:54.554125 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd13d7ca-7c61-4c40-a86e-a58c1a7d1390-registry-tls\") pod \"image-registry-b47d488dd-h2wcn\" (UID: \"fd13d7ca-7c61-4c40-a86e-a58c1a7d1390\") " pod="openshift-image-registry/image-registry-b47d488dd-h2wcn" Jul 21 01:36:54.554313 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:54.554210 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 21 01:36:54.554313 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:54.554236 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e6d9baf0-743b-48f5-9374-4443c65ea048-service-ca-bundle podName:e6d9baf0-743b-48f5-9374-4443c65ea048 nodeName:}" failed. No retries permitted until 2026-07-21 01:36:56.554218982 +0000 UTC m=+40.141384973 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e6d9baf0-743b-48f5-9374-4443c65ea048-service-ca-bundle") pod "router-default-6f7df7d7cb-gcx5w" (UID: "e6d9baf0-743b-48f5-9374-4443c65ea048") : configmap references non-existent config key: service-ca.crt Jul 21 01:36:54.554313 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:54.554212 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 21 01:36:54.554313 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:54.554260 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b47d488dd-h2wcn: secret "image-registry-tls" not found Jul 21 01:36:54.554313 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:54.554269 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b7843343-f8dd-4ea6-a8f4-e3153a0b3ab2-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-g86ww\" (UID: \"b7843343-f8dd-4ea6-a8f4-e3153a0b3ab2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-g86ww" Jul 21 01:36:54.554313 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:54.554295 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd13d7ca-7c61-4c40-a86e-a58c1a7d1390-registry-tls podName:fd13d7ca-7c61-4c40-a86e-a58c1a7d1390 nodeName:}" failed. No retries permitted until 2026-07-21 01:36:56.554281637 +0000 UTC m=+40.141447628 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fd13d7ca-7c61-4c40-a86e-a58c1a7d1390-registry-tls") pod "image-registry-b47d488dd-h2wcn" (UID: "fd13d7ca-7c61-4c40-a86e-a58c1a7d1390") : secret "image-registry-tls" not found Jul 21 01:36:54.554313 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:54.554313 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 21 01:36:54.554710 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:54.554323 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6d9baf0-743b-48f5-9374-4443c65ea048-metrics-certs podName:e6d9baf0-743b-48f5-9374-4443c65ea048 nodeName:}" failed. No retries permitted until 2026-07-21 01:36:56.554313331 +0000 UTC m=+40.141479288 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e6d9baf0-743b-48f5-9374-4443c65ea048-metrics-certs") pod "router-default-6f7df7d7cb-gcx5w" (UID: "e6d9baf0-743b-48f5-9374-4443c65ea048") : secret "router-metrics-certs-default" not found Jul 21 01:36:54.554710 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:54.554373 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7843343-f8dd-4ea6-a8f4-e3153a0b3ab2-samples-operator-tls podName:b7843343-f8dd-4ea6-a8f4-e3153a0b3ab2 nodeName:}" failed. No retries permitted until 2026-07-21 01:36:56.554357327 +0000 UTC m=+40.141523342 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b7843343-f8dd-4ea6-a8f4-e3153a0b3ab2-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-g86ww" (UID: "b7843343-f8dd-4ea6-a8f4-e3153a0b3ab2") : secret "samples-operator-tls" not found Jul 21 01:36:54.656866 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:54.655862 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0add4762-9fef-4072-a05a-cdca30f4cdcf-metrics-tls\") pod \"dns-default-v899s\" (UID: \"0add4762-9fef-4072-a05a-cdca30f4cdcf\") " pod="openshift-dns/dns-default-v899s" Jul 21 01:36:54.656866 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:54.656071 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 01:36:54.656866 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:54.656139 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0add4762-9fef-4072-a05a-cdca30f4cdcf-metrics-tls podName:0add4762-9fef-4072-a05a-cdca30f4cdcf nodeName:}" failed. No retries permitted until 2026-07-21 01:36:56.656121059 +0000 UTC m=+40.243287032 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0add4762-9fef-4072-a05a-cdca30f4cdcf-metrics-tls") pod "dns-default-v899s" (UID: "0add4762-9fef-4072-a05a-cdca30f4cdcf") : secret "dns-default-metrics-tls" not found Jul 21 01:36:54.905579 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:54.905483 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-9h5pm"] Jul 21 01:36:54.908337 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:54.908177 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bmgfc"] Jul 21 01:36:54.912648 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:54.912609 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e08e8c3_dd3e_4792_8096_cf150dbcb58b.slice/crio-503fc8efb3aa788d233464503597517e3324b6d0c2dfd13f2d9122563c04e695 WatchSource:0}: Error finding container 503fc8efb3aa788d233464503597517e3324b6d0c2dfd13f2d9122563c04e695: Status 404 returned error can't find the container with id 503fc8efb3aa788d233464503597517e3324b6d0c2dfd13f2d9122563c04e695 Jul 21 01:36:54.913574 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:54.913548 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d402931_b366_48df_acda_2e144e7c6c70.slice/crio-5018c8476f3e5de45eec7a82a9c9c9f7da29797c8b96894b98f6aa09f67dfc69 WatchSource:0}: Error finding container 5018c8476f3e5de45eec7a82a9c9c9f7da29797c8b96894b98f6aa09f67dfc69: Status 404 returned error can't find the container with id 5018c8476f3e5de45eec7a82a9c9c9f7da29797c8b96894b98f6aa09f67dfc69 Jul 21 01:36:54.918927 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:54.918894 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-dprcn"] Jul 21 01:36:54.922441 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:54.922316 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbbe27be_311c_40ea_9bdd_80b5fb532422.slice/crio-4417d8f264ff94d33a5ef98f3bf96b9221609576be6632f3f765f592159bbacc WatchSource:0}: Error finding container 4417d8f264ff94d33a5ef98f3bf96b9221609576be6632f3f765f592159bbacc: Status 404 returned error can't find the container with id 4417d8f264ff94d33a5ef98f3bf96b9221609576be6632f3f765f592159bbacc Jul 21 01:36:54.926999 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:54.926974 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-4mtp7"] Jul 21 01:36:54.930048 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:54.930026 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-2pscn"] Jul 21 01:36:54.930851 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:54.930813 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c380009_9bf5_4b73_803a_5a1ea9c1abf0.slice/crio-f34c50d3c61a492a746a7cbc0817c008eccdb85e87f4925b1a4e267fd9f3a2e9 WatchSource:0}: Error finding container f34c50d3c61a492a746a7cbc0817c008eccdb85e87f4925b1a4e267fd9f3a2e9: Status 404 returned error can't find the container with id f34c50d3c61a492a746a7cbc0817c008eccdb85e87f4925b1a4e267fd9f3a2e9 Jul 21 01:36:54.934362 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:54.934336 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-wnjpr"] Jul 21 01:36:54.937184 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:54.937133 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-vxvqz"] Jul 21 01:36:54.938166 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:54.938139 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd67dd362_e3a6_4ad1_9f9f_8f628ffce9a2.slice/crio-5bddab301628a2d9f768117c6bc736b57af186e309b1323142ad8ceec8ee1c61 WatchSource:0}: Error finding container 5bddab301628a2d9f768117c6bc736b57af186e309b1323142ad8ceec8ee1c61: Status 404 returned error can't find the container with id 5bddab301628a2d9f768117c6bc736b57af186e309b1323142ad8ceec8ee1c61 Jul 21 01:36:54.940170 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:36:54.940145 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea46ff20_2a04_4b13_92ed_371a4eb06987.slice/crio-1a9ba63babf8b721abfcb9e4fff7d457d8144854b95a1e800995ff592117be8f WatchSource:0}: Error finding container 1a9ba63babf8b721abfcb9e4fff7d457d8144854b95a1e800995ff592117be8f: Status 404 returned error can't find the container with id 1a9ba63babf8b721abfcb9e4fff7d457d8144854b95a1e800995ff592117be8f Jul 21 01:36:55.043681 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:55.043648 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-5gfm8"] Jul 21 01:36:55.046819 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:55.046798 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-5gfm8" Jul 21 01:36:55.049009 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:55.048981 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 21 01:36:55.049157 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:55.049138 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-6wkg4\"" Jul 21 01:36:55.049294 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:55.049275 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 21 01:36:55.057407 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:55.057287 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-5gfm8"] Jul 21 01:36:55.137733 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:55.137656 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-vxvqz" event={"ID":"ea46ff20-2a04-4b13-92ed-371a4eb06987","Type":"ContainerStarted","Data":"1a9ba63babf8b721abfcb9e4fff7d457d8144854b95a1e800995ff592117be8f"} Jul 21 01:36:55.138748 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:55.138728 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-2pscn" event={"ID":"de3854b8-6d90-46d9-9ab9-60a83f7a58be","Type":"ContainerStarted","Data":"a738866e5fb7b9a6ee667aadaa5e9560363e05370caf286cf67d418d48c5aa2d"} Jul 21 01:36:55.141443 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:55.141411 2580 generic.go:356] "Generic (PLEG): container finished" podID="64ab7299-1b1f-4d7e-8c26-998adf4499db" containerID="6eac5be5b0e7c37e12b71f49a323b79aa947e0ee691b21d78d784e77b209cbb8" exitCode=0 Jul 21 01:36:55.141546 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:55.141495 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sm5jf" event={"ID":"64ab7299-1b1f-4d7e-8c26-998adf4499db","Type":"ContainerDied","Data":"6eac5be5b0e7c37e12b71f49a323b79aa947e0ee691b21d78d784e77b209cbb8"} Jul 21 01:36:55.142623 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:55.142595 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-dprcn" event={"ID":"cbbe27be-311c-40ea-9bdd-80b5fb532422","Type":"ContainerStarted","Data":"4417d8f264ff94d33a5ef98f3bf96b9221609576be6632f3f765f592159bbacc"} Jul 21 01:36:55.145809 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:55.145771 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bmgfc" event={"ID":"1d402931-b366-48df-acda-2e144e7c6c70","Type":"ContainerStarted","Data":"5018c8476f3e5de45eec7a82a9c9c9f7da29797c8b96894b98f6aa09f67dfc69"} Jul 21 01:36:55.147080 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:55.147050 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-wnjpr" event={"ID":"d67dd362-e3a6-4ad1-9f9f-8f628ffce9a2","Type":"ContainerStarted","Data":"5bddab301628a2d9f768117c6bc736b57af186e309b1323142ad8ceec8ee1c61"} Jul 21 01:36:55.148149 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:55.148126 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-4mtp7" event={"ID":"9c380009-9bf5-4b73-803a-5a1ea9c1abf0","Type":"ContainerStarted","Data":"f34c50d3c61a492a746a7cbc0817c008eccdb85e87f4925b1a4e267fd9f3a2e9"} Jul 21 01:36:55.149199 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:55.149178 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-9h5pm" event={"ID":"6e08e8c3-dd3e-4792-8096-cf150dbcb58b","Type":"ContainerStarted","Data":"503fc8efb3aa788d233464503597517e3324b6d0c2dfd13f2d9122563c04e695"} Jul 21 01:36:55.160319 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:55.160280 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/15628691-46c2-418e-8eca-47f963849e13-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-5gfm8\" (UID: \"15628691-46c2-418e-8eca-47f963849e13\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-5gfm8" Jul 21 01:36:55.160423 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:55.160343 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/15628691-46c2-418e-8eca-47f963849e13-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-5gfm8\" (UID: \"15628691-46c2-418e-8eca-47f963849e13\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-5gfm8" Jul 21 01:36:55.260841 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:55.260807 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/15628691-46c2-418e-8eca-47f963849e13-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-5gfm8\" (UID: \"15628691-46c2-418e-8eca-47f963849e13\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-5gfm8" Jul 21 01:36:55.261108 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:55.261088 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/15628691-46c2-418e-8eca-47f963849e13-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-5gfm8\" (UID: \"15628691-46c2-418e-8eca-47f963849e13\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-5gfm8" Jul 21 01:36:55.261534 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:55.261225 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 21 01:36:55.261534 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:55.261290 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15628691-46c2-418e-8eca-47f963849e13-networking-console-plugin-cert podName:15628691-46c2-418e-8eca-47f963849e13 nodeName:}" failed. No retries permitted until 2026-07-21 01:36:55.761270624 +0000 UTC m=+39.348436580 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/15628691-46c2-418e-8eca-47f963849e13-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-5gfm8" (UID: "15628691-46c2-418e-8eca-47f963849e13") : secret "networking-console-plugin-cert" not found Jul 21 01:36:55.261882 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:55.261855 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/15628691-46c2-418e-8eca-47f963849e13-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-5gfm8\" (UID: \"15628691-46c2-418e-8eca-47f963849e13\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-5gfm8" Jul 21 01:36:55.765742 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:55.765063 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/15628691-46c2-418e-8eca-47f963849e13-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-5gfm8\" (UID: \"15628691-46c2-418e-8eca-47f963849e13\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-5gfm8" Jul 21 01:36:55.765742 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:55.765315 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 21 01:36:55.765742 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:55.765376 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15628691-46c2-418e-8eca-47f963849e13-networking-console-plugin-cert podName:15628691-46c2-418e-8eca-47f963849e13 nodeName:}" failed. No retries permitted until 2026-07-21 01:36:56.76535853 +0000 UTC m=+40.352524505 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/15628691-46c2-418e-8eca-47f963849e13-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-5gfm8" (UID: "15628691-46c2-418e-8eca-47f963849e13") : secret "networking-console-plugin-cert" not found Jul 21 01:36:56.170391 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:56.168915 2580 generic.go:356] "Generic (PLEG): container finished" podID="64ab7299-1b1f-4d7e-8c26-998adf4499db" containerID="c7b88bb4e5369e6c589b4487a84fa0d8c47f38384d069942d2771b9b769778b2" exitCode=0 Jul 21 01:36:56.170391 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:56.168984 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sm5jf" event={"ID":"64ab7299-1b1f-4d7e-8c26-998adf4499db","Type":"ContainerDied","Data":"c7b88bb4e5369e6c589b4487a84fa0d8c47f38384d069942d2771b9b769778b2"} Jul 21 01:36:56.579504 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:56.577886 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6d9baf0-743b-48f5-9374-4443c65ea048-service-ca-bundle\") pod \"router-default-6f7df7d7cb-gcx5w\" (UID: \"e6d9baf0-743b-48f5-9374-4443c65ea048\") " pod="openshift-ingress/router-default-6f7df7d7cb-gcx5w" Jul 21 01:36:56.579504 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:56.578089 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e6d9baf0-743b-48f5-9374-4443c65ea048-service-ca-bundle podName:e6d9baf0-743b-48f5-9374-4443c65ea048 nodeName:}" failed. No retries permitted until 2026-07-21 01:37:00.578063144 +0000 UTC m=+44.165229106 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e6d9baf0-743b-48f5-9374-4443c65ea048-service-ca-bundle") pod "router-default-6f7df7d7cb-gcx5w" (UID: "e6d9baf0-743b-48f5-9374-4443c65ea048") : configmap references non-existent config key: service-ca.crt Jul 21 01:36:56.579504 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:56.578171 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd13d7ca-7c61-4c40-a86e-a58c1a7d1390-registry-tls\") pod \"image-registry-b47d488dd-h2wcn\" (UID: \"fd13d7ca-7c61-4c40-a86e-a58c1a7d1390\") " pod="openshift-image-registry/image-registry-b47d488dd-h2wcn" Jul 21 01:36:56.579504 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:56.578214 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b7843343-f8dd-4ea6-a8f4-e3153a0b3ab2-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-g86ww\" (UID: \"b7843343-f8dd-4ea6-a8f4-e3153a0b3ab2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-g86ww" Jul 21 01:36:56.579504 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:56.578280 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1e930d67-d711-419c-af02-86b43027a95d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-9ctn2\" (UID: \"1e930d67-d711-419c-af02-86b43027a95d\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-9ctn2" Jul 21 01:36:56.579504 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:56.578375 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6d9baf0-743b-48f5-9374-4443c65ea048-metrics-certs\") pod \"router-default-6f7df7d7cb-gcx5w\" (UID: \"e6d9baf0-743b-48f5-9374-4443c65ea048\") " pod="openshift-ingress/router-default-6f7df7d7cb-gcx5w" Jul 21 01:36:56.579504 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:56.578622 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 21 01:36:56.579504 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:56.578677 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 21 01:36:56.579504 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:56.578747 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 21 01:36:56.579504 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:56.578760 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b47d488dd-h2wcn: secret "image-registry-tls" not found Jul 21 01:36:56.579504 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:56.578779 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 21 01:36:56.579504 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:56.578682 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7843343-f8dd-4ea6-a8f4-e3153a0b3ab2-samples-operator-tls podName:b7843343-f8dd-4ea6-a8f4-e3153a0b3ab2 nodeName:}" failed. No retries permitted until 2026-07-21 01:37:00.578664598 +0000 UTC m=+44.165830572 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b7843343-f8dd-4ea6-a8f4-e3153a0b3ab2-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-g86ww" (UID: "b7843343-f8dd-4ea6-a8f4-e3153a0b3ab2") : secret "samples-operator-tls" not found Jul 21 01:36:56.579504 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:56.578808 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6d9baf0-743b-48f5-9374-4443c65ea048-metrics-certs podName:e6d9baf0-743b-48f5-9374-4443c65ea048 nodeName:}" failed. No retries permitted until 2026-07-21 01:37:00.578796292 +0000 UTC m=+44.165962253 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e6d9baf0-743b-48f5-9374-4443c65ea048-metrics-certs") pod "router-default-6f7df7d7cb-gcx5w" (UID: "e6d9baf0-743b-48f5-9374-4443c65ea048") : secret "router-metrics-certs-default" not found Jul 21 01:36:56.579504 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:56.578824 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd13d7ca-7c61-4c40-a86e-a58c1a7d1390-registry-tls podName:fd13d7ca-7c61-4c40-a86e-a58c1a7d1390 nodeName:}" failed. No retries permitted until 2026-07-21 01:37:00.578814299 +0000 UTC m=+44.165980259 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fd13d7ca-7c61-4c40-a86e-a58c1a7d1390-registry-tls") pod "image-registry-b47d488dd-h2wcn" (UID: "fd13d7ca-7c61-4c40-a86e-a58c1a7d1390") : secret "image-registry-tls" not found Jul 21 01:36:56.579504 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:56.578839 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e930d67-d711-419c-af02-86b43027a95d-cluster-monitoring-operator-tls podName:1e930d67-d711-419c-af02-86b43027a95d nodeName:}" failed. No retries permitted until 2026-07-21 01:37:00.578830585 +0000 UTC m=+44.165996545 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1e930d67-d711-419c-af02-86b43027a95d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-9ctn2" (UID: "1e930d67-d711-419c-af02-86b43027a95d") : secret "cluster-monitoring-operator-tls" not found Jul 21 01:36:56.680114 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:56.679481 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0add4762-9fef-4072-a05a-cdca30f4cdcf-metrics-tls\") pod \"dns-default-v899s\" (UID: \"0add4762-9fef-4072-a05a-cdca30f4cdcf\") " pod="openshift-dns/dns-default-v899s" Jul 21 01:36:56.680114 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:56.679711 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 01:36:56.680114 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:56.679771 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0add4762-9fef-4072-a05a-cdca30f4cdcf-metrics-tls podName:0add4762-9fef-4072-a05a-cdca30f4cdcf nodeName:}" failed. No retries permitted until 2026-07-21 01:37:00.679753337 +0000 UTC m=+44.266919310 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0add4762-9fef-4072-a05a-cdca30f4cdcf-metrics-tls") pod "dns-default-v899s" (UID: "0add4762-9fef-4072-a05a-cdca30f4cdcf") : secret "dns-default-metrics-tls" not found Jul 21 01:36:56.781250 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:56.780549 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/15628691-46c2-418e-8eca-47f963849e13-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-5gfm8\" (UID: \"15628691-46c2-418e-8eca-47f963849e13\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-5gfm8" Jul 21 01:36:56.781250 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:56.780807 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 21 01:36:56.781250 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:56.780873 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15628691-46c2-418e-8eca-47f963849e13-networking-console-plugin-cert podName:15628691-46c2-418e-8eca-47f963849e13 nodeName:}" failed. No retries permitted until 2026-07-21 01:36:58.780855248 +0000 UTC m=+42.368021208 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/15628691-46c2-418e-8eca-47f963849e13-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-5gfm8" (UID: "15628691-46c2-418e-8eca-47f963849e13") : secret "networking-console-plugin-cert" not found Jul 21 01:36:57.181851 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:57.181166 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sm5jf" event={"ID":"64ab7299-1b1f-4d7e-8c26-998adf4499db","Type":"ContainerStarted","Data":"568e8350534de1da1c7a546ee7e5d2be8cf17525273865d954b57892f30acb5a"} Jul 21 01:36:57.203515 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:57.203187 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-sm5jf" podStartSLOduration=7.60799605 podStartE2EDuration="39.203167008s" totalImagesPullingTime="31.595170958s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 01:36:18 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:36:57.201743982 +0000 UTC m=+40.788909997" watchObservedRunningTime="2026-07-21 01:36:57.203167008 +0000 UTC m=+40.790332984" Jul 21 01:36:58.805118 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:36:58.805073 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/15628691-46c2-418e-8eca-47f963849e13-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-5gfm8\" (UID: \"15628691-46c2-418e-8eca-47f963849e13\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-5gfm8" Jul 21 01:36:58.805555 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:58.805226 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 21 01:36:58.805555 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:36:58.805292 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15628691-46c2-418e-8eca-47f963849e13-networking-console-plugin-cert podName:15628691-46c2-418e-8eca-47f963849e13 nodeName:}" failed. No retries permitted until 2026-07-21 01:37:02.805277018 +0000 UTC m=+46.392442974 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/15628691-46c2-418e-8eca-47f963849e13-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-5gfm8" (UID: "15628691-46c2-418e-8eca-47f963849e13") : secret "networking-console-plugin-cert" not found Jul 21 01:37:00.622197 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:00.622103 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6d9baf0-743b-48f5-9374-4443c65ea048-service-ca-bundle\") pod \"router-default-6f7df7d7cb-gcx5w\" (UID: \"e6d9baf0-743b-48f5-9374-4443c65ea048\") " pod="openshift-ingress/router-default-6f7df7d7cb-gcx5w" Jul 21 01:37:00.622197 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:00.622163 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd13d7ca-7c61-4c40-a86e-a58c1a7d1390-registry-tls\") pod \"image-registry-b47d488dd-h2wcn\" (UID: \"fd13d7ca-7c61-4c40-a86e-a58c1a7d1390\") " pod="openshift-image-registry/image-registry-b47d488dd-h2wcn" Jul 21 01:37:00.622731 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:00.622202 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b7843343-f8dd-4ea6-a8f4-e3153a0b3ab2-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-g86ww\" (UID: \"b7843343-f8dd-4ea6-a8f4-e3153a0b3ab2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-g86ww" Jul 21 01:37:00.622731 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:00.622262 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1e930d67-d711-419c-af02-86b43027a95d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-9ctn2\" (UID: \"1e930d67-d711-419c-af02-86b43027a95d\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-9ctn2" Jul 21 01:37:00.622731 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:37:00.622271 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 21 01:37:00.622731 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:37:00.622289 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e6d9baf0-743b-48f5-9374-4443c65ea048-service-ca-bundle podName:e6d9baf0-743b-48f5-9374-4443c65ea048 nodeName:}" failed. No retries permitted until 2026-07-21 01:37:08.622266958 +0000 UTC m=+52.209432932 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e6d9baf0-743b-48f5-9374-4443c65ea048-service-ca-bundle") pod "router-default-6f7df7d7cb-gcx5w" (UID: "e6d9baf0-743b-48f5-9374-4443c65ea048") : configmap references non-existent config key: service-ca.crt Jul 21 01:37:00.622731 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:37:00.622297 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b47d488dd-h2wcn: secret "image-registry-tls" not found Jul 21 01:37:00.622731 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:37:00.622339 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 21 01:37:00.622731 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:37:00.622340 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 21 01:37:00.622731 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:37:00.622387 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e930d67-d711-419c-af02-86b43027a95d-cluster-monitoring-operator-tls podName:1e930d67-d711-419c-af02-86b43027a95d nodeName:}" failed. No retries permitted until 2026-07-21 01:37:08.622375009 +0000 UTC m=+52.209540987 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1e930d67-d711-419c-af02-86b43027a95d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-9ctn2" (UID: "1e930d67-d711-419c-af02-86b43027a95d") : secret "cluster-monitoring-operator-tls" not found Jul 21 01:37:00.622731 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:37:00.622409 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7843343-f8dd-4ea6-a8f4-e3153a0b3ab2-samples-operator-tls podName:b7843343-f8dd-4ea6-a8f4-e3153a0b3ab2 nodeName:}" failed. No retries permitted until 2026-07-21 01:37:08.622394143 +0000 UTC m=+52.209560104 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b7843343-f8dd-4ea6-a8f4-e3153a0b3ab2-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-g86ww" (UID: "b7843343-f8dd-4ea6-a8f4-e3153a0b3ab2") : secret "samples-operator-tls" not found Jul 21 01:37:00.622731 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:00.622467 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6d9baf0-743b-48f5-9374-4443c65ea048-metrics-certs\") pod \"router-default-6f7df7d7cb-gcx5w\" (UID: \"e6d9baf0-743b-48f5-9374-4443c65ea048\") " pod="openshift-ingress/router-default-6f7df7d7cb-gcx5w" Jul 21 01:37:00.622731 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:37:00.622478 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd13d7ca-7c61-4c40-a86e-a58c1a7d1390-registry-tls podName:fd13d7ca-7c61-4c40-a86e-a58c1a7d1390 nodeName:}" failed. No retries permitted until 2026-07-21 01:37:08.622445116 +0000 UTC m=+52.209611086 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fd13d7ca-7c61-4c40-a86e-a58c1a7d1390-registry-tls") pod "image-registry-b47d488dd-h2wcn" (UID: "fd13d7ca-7c61-4c40-a86e-a58c1a7d1390") : secret "image-registry-tls" not found Jul 21 01:37:00.622731 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:37:00.622535 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 21 01:37:00.622731 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:37:00.622579 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6d9baf0-743b-48f5-9374-4443c65ea048-metrics-certs podName:e6d9baf0-743b-48f5-9374-4443c65ea048 nodeName:}" failed. No retries permitted until 2026-07-21 01:37:08.622566611 +0000 UTC m=+52.209732569 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e6d9baf0-743b-48f5-9374-4443c65ea048-metrics-certs") pod "router-default-6f7df7d7cb-gcx5w" (UID: "e6d9baf0-743b-48f5-9374-4443c65ea048") : secret "router-metrics-certs-default" not found Jul 21 01:37:00.722959 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:00.722915 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0add4762-9fef-4072-a05a-cdca30f4cdcf-metrics-tls\") pod \"dns-default-v899s\" (UID: \"0add4762-9fef-4072-a05a-cdca30f4cdcf\") " pod="openshift-dns/dns-default-v899s" Jul 21 01:37:00.723135 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:37:00.723083 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 01:37:00.723215 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:37:00.723160 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0add4762-9fef-4072-a05a-cdca30f4cdcf-metrics-tls podName:0add4762-9fef-4072-a05a-cdca30f4cdcf nodeName:}" failed. No retries permitted until 2026-07-21 01:37:08.723139617 +0000 UTC m=+52.310305595 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0add4762-9fef-4072-a05a-cdca30f4cdcf-metrics-tls") pod "dns-default-v899s" (UID: "0add4762-9fef-4072-a05a-cdca30f4cdcf") : secret "dns-default-metrics-tls" not found Jul 21 01:37:02.842471 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:02.842384 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/15628691-46c2-418e-8eca-47f963849e13-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-5gfm8\" (UID: \"15628691-46c2-418e-8eca-47f963849e13\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-5gfm8" Jul 21 01:37:02.842920 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:37:02.842569 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 21 01:37:02.842920 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:37:02.842654 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15628691-46c2-418e-8eca-47f963849e13-networking-console-plugin-cert podName:15628691-46c2-418e-8eca-47f963849e13 nodeName:}" failed. No retries permitted until 2026-07-21 01:37:10.842631612 +0000 UTC m=+54.429797571 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/15628691-46c2-418e-8eca-47f963849e13-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-5gfm8" (UID: "15628691-46c2-418e-8eca-47f963849e13") : secret "networking-console-plugin-cert" not found Jul 21 01:37:03.203367 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:03.203317 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-dprcn" event={"ID":"cbbe27be-311c-40ea-9bdd-80b5fb532422","Type":"ContainerStarted","Data":"11c805c9ced36c85e086433f14f5f32299cac73a1c04886a48d241a7aa573e64"} Jul 21 01:37:03.205201 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:03.205170 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bmgfc" event={"ID":"1d402931-b366-48df-acda-2e144e7c6c70","Type":"ContainerStarted","Data":"c0e0e13acf7e0655fa4f0a94d15fc40c26fbfc8f224ca936181f74c689f80ca2"} Jul 21 01:37:03.206950 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:03.206923 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-wnjpr" event={"ID":"d67dd362-e3a6-4ad1-9f9f-8f628ffce9a2","Type":"ContainerStarted","Data":"d0f1b677a4dd6cb2c846088f8f818252d3b5d122139ed6f40e3a3cef0e70bf8e"} Jul 21 01:37:03.209102 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:03.209081 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-4mtp7_9c380009-9bf5-4b73-803a-5a1ea9c1abf0/console-operator/0.log" Jul 21 01:37:03.209212 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:03.209122 2580 generic.go:356] "Generic (PLEG): container finished" podID="9c380009-9bf5-4b73-803a-5a1ea9c1abf0" containerID="f30af647337a6e63b87729062d9ac1c097e9bb32117b06e89f2b018863dfb15a" exitCode=255 Jul 21 01:37:03.209333 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:03.209309 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-4mtp7" event={"ID":"9c380009-9bf5-4b73-803a-5a1ea9c1abf0","Type":"ContainerDied","Data":"f30af647337a6e63b87729062d9ac1c097e9bb32117b06e89f2b018863dfb15a"} Jul 21 01:37:03.209473 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:03.209438 2580 scope.go:117] "RemoveContainer" containerID="f30af647337a6e63b87729062d9ac1c097e9bb32117b06e89f2b018863dfb15a" Jul 21 01:37:03.211538 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:03.211515 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-9h5pm" event={"ID":"6e08e8c3-dd3e-4792-8096-cf150dbcb58b","Type":"ContainerStarted","Data":"ec01f40376f16ad3f84e510f5eabbc39394c2c589eae3feb4f2a337eefe47385"} Jul 21 01:37:03.213257 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:03.213218 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-vxvqz" event={"ID":"ea46ff20-2a04-4b13-92ed-371a4eb06987","Type":"ContainerStarted","Data":"afd119049fe643020c5538b2bd8ab5758f5d43dd226c554b7355540c559dfa0c"} Jul 21 01:37:03.215037 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:03.215016 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-2pscn" event={"ID":"de3854b8-6d90-46d9-9ab9-60a83f7a58be","Type":"ContainerStarted","Data":"e31e973327c99c790bb72616aa8255ab8582cf8f4448dd66cbe8b7e495d3a4e5"} Jul 21 01:37:03.218586 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:03.218539 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-dprcn" podStartSLOduration=26.571699831 podStartE2EDuration="34.218520937s" totalImagesPullingTime="7.646821106s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 01:36:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:37:03.217111702 +0000 UTC m=+46.804277683" watchObservedRunningTime="2026-07-21 01:37:03.218520937 +0000 UTC m=+46.805686895" Jul 21 01:37:03.252629 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:03.252577 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-2pscn" podStartSLOduration=25.620397195 podStartE2EDuration="33.252557911s" totalImagesPullingTime="7.632160716s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 01:36:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:37:03.252203164 +0000 UTC m=+46.839369142" watchObservedRunningTime="2026-07-21 01:37:03.252557911 +0000 UTC m=+46.839723891" Jul 21 01:37:03.253200 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:03.253161 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-vxvqz" podStartSLOduration=33.675493826 podStartE2EDuration="41.253147577s" totalImagesPullingTime="7.577653751s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 01:36:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:37:03.235552206 +0000 UTC m=+46.822718201" watchObservedRunningTime="2026-07-21 01:37:03.253147577 +0000 UTC m=+46.840313550" Jul 21 01:37:03.269791 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:03.269702 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-555d9d7885-wnjpr" podStartSLOduration=35.636353009 podStartE2EDuration="43.269680573s" totalImagesPullingTime="7.633327564s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 01:36:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:37:03.268924168 +0000 UTC m=+46.856090174" watchObservedRunningTime="2026-07-21 01:37:03.269680573 +0000 UTC m=+46.856846552" Jul 21 01:37:03.287634 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:03.287316 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-9h5pm" podStartSLOduration=28.630004797 podStartE2EDuration="36.287292093s" totalImagesPullingTime="7.657287296s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 01:36:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:37:03.285324714 +0000 UTC m=+46.872490694" watchObservedRunningTime="2026-07-21 01:37:03.287292093 +0000 UTC m=+46.874458072" Jul 21 01:37:03.309065 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:03.308730 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-bmgfc" podStartSLOduration=3.6472674449999998 podStartE2EDuration="11.308707428s" totalImagesPullingTime="7.661439983s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 01:36:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:37:03.306329208 +0000 UTC m=+46.893495187" watchObservedRunningTime="2026-07-21 01:37:03.308707428 +0000 UTC m=+46.895873407" Jul 21 01:37:04.219496 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:04.219469 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-4mtp7_9c380009-9bf5-4b73-803a-5a1ea9c1abf0/console-operator/1.log" Jul 21 01:37:04.219936 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:04.219911 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-4mtp7_9c380009-9bf5-4b73-803a-5a1ea9c1abf0/console-operator/0.log" Jul 21 01:37:04.220006 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:04.219954 2580 generic.go:356] "Generic (PLEG): container finished" podID="9c380009-9bf5-4b73-803a-5a1ea9c1abf0" containerID="e0c0230bf56e92986c211934ee9ce14887e5e45db8e4bb96ef45ca8d8d7930e4" exitCode=255 Jul 21 01:37:04.220114 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:04.220063 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-4mtp7" event={"ID":"9c380009-9bf5-4b73-803a-5a1ea9c1abf0","Type":"ContainerDied","Data":"e0c0230bf56e92986c211934ee9ce14887e5e45db8e4bb96ef45ca8d8d7930e4"} Jul 21 01:37:04.220198 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:04.220159 2580 scope.go:117] "RemoveContainer" containerID="f30af647337a6e63b87729062d9ac1c097e9bb32117b06e89f2b018863dfb15a" Jul 21 01:37:04.220282 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:04.220266 2580 scope.go:117] "RemoveContainer" containerID="e0c0230bf56e92986c211934ee9ce14887e5e45db8e4bb96ef45ca8d8d7930e4" Jul 21 01:37:04.220515 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:37:04.220485 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-4mtp7_openshift-console-operator(9c380009-9bf5-4b73-803a-5a1ea9c1abf0)\"" pod="openshift-console-operator/console-operator-565b7b569d-4mtp7" podUID="9c380009-9bf5-4b73-803a-5a1ea9c1abf0" Jul 21 01:37:04.748649 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:04.748616 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-mqxgn"] Jul 21 01:37:04.777112 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:04.777080 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mqxgn"] Jul 21 01:37:04.777257 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:04.777144 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mqxgn" Jul 21 01:37:04.779369 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:04.779343 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 21 01:37:04.779517 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:04.779393 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 21 01:37:04.780012 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:04.779997 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5pn52\"" Jul 21 01:37:04.866828 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:04.866788 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf95d\" (UniqueName: \"kubernetes.io/projected/e0343507-90fb-4245-bd91-73476ed1a05b-kube-api-access-kf95d\") pod \"insights-runtime-extractor-mqxgn\" (UID: \"e0343507-90fb-4245-bd91-73476ed1a05b\") " pod="openshift-insights/insights-runtime-extractor-mqxgn" Jul 21 01:37:04.866828 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:04.866829 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e0343507-90fb-4245-bd91-73476ed1a05b-crio-socket\") pod \"insights-runtime-extractor-mqxgn\" (UID: \"e0343507-90fb-4245-bd91-73476ed1a05b\") " pod="openshift-insights/insights-runtime-extractor-mqxgn" Jul 21 01:37:04.867038 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:04.866937 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e0343507-90fb-4245-bd91-73476ed1a05b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mqxgn\" (UID: \"e0343507-90fb-4245-bd91-73476ed1a05b\") " pod="openshift-insights/insights-runtime-extractor-mqxgn" Jul 21 01:37:04.867081 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:04.867061 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e0343507-90fb-4245-bd91-73476ed1a05b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mqxgn\" (UID: \"e0343507-90fb-4245-bd91-73476ed1a05b\") " pod="openshift-insights/insights-runtime-extractor-mqxgn" Jul 21 01:37:04.867119 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:04.867101 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e0343507-90fb-4245-bd91-73476ed1a05b-data-volume\") pod \"insights-runtime-extractor-mqxgn\" (UID: \"e0343507-90fb-4245-bd91-73476ed1a05b\") " pod="openshift-insights/insights-runtime-extractor-mqxgn" Jul 21 01:37:04.967949 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:04.967912 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e0343507-90fb-4245-bd91-73476ed1a05b-data-volume\") pod \"insights-runtime-extractor-mqxgn\" (UID: \"e0343507-90fb-4245-bd91-73476ed1a05b\") " pod="openshift-insights/insights-runtime-extractor-mqxgn" Jul 21 01:37:04.968130 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:04.967960 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf95d\" (UniqueName: \"kubernetes.io/projected/e0343507-90fb-4245-bd91-73476ed1a05b-kube-api-access-kf95d\") pod \"insights-runtime-extractor-mqxgn\" (UID: \"e0343507-90fb-4245-bd91-73476ed1a05b\") " pod="openshift-insights/insights-runtime-extractor-mqxgn" Jul 21 01:37:04.968130 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:04.967979 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e0343507-90fb-4245-bd91-73476ed1a05b-crio-socket\") pod \"insights-runtime-extractor-mqxgn\" (UID: \"e0343507-90fb-4245-bd91-73476ed1a05b\") " pod="openshift-insights/insights-runtime-extractor-mqxgn" Jul 21 01:37:04.968227 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:04.968139 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e0343507-90fb-4245-bd91-73476ed1a05b-crio-socket\") pod \"insights-runtime-extractor-mqxgn\" (UID: \"e0343507-90fb-4245-bd91-73476ed1a05b\") " pod="openshift-insights/insights-runtime-extractor-mqxgn" Jul 21 01:37:04.968227 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:04.968180 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e0343507-90fb-4245-bd91-73476ed1a05b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mqxgn\" (UID: \"e0343507-90fb-4245-bd91-73476ed1a05b\") " pod="openshift-insights/insights-runtime-extractor-mqxgn" Jul 21 01:37:04.968306 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:37:04.968255 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 21 01:37:04.968306 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:04.968285 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e0343507-90fb-4245-bd91-73476ed1a05b-data-volume\") pod \"insights-runtime-extractor-mqxgn\" (UID: \"e0343507-90fb-4245-bd91-73476ed1a05b\") " pod="openshift-insights/insights-runtime-extractor-mqxgn" Jul 21 01:37:04.968405 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:37:04.968310 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e0343507-90fb-4245-bd91-73476ed1a05b-kube-rbac-proxy-cm podName:e0343507-90fb-4245-bd91-73476ed1a05b nodeName:}" failed. No retries permitted until 2026-07-21 01:37:05.468292655 +0000 UTC m=+49.055458627 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e0343507-90fb-4245-bd91-73476ed1a05b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mqxgn" (UID: "e0343507-90fb-4245-bd91-73476ed1a05b") : configmap "kube-rbac-proxy" not found Jul 21 01:37:04.968493 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:04.968418 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e0343507-90fb-4245-bd91-73476ed1a05b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mqxgn\" (UID: \"e0343507-90fb-4245-bd91-73476ed1a05b\") " pod="openshift-insights/insights-runtime-extractor-mqxgn" Jul 21 01:37:04.968545 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:37:04.968533 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 21 01:37:04.968585 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:37:04.968577 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0343507-90fb-4245-bd91-73476ed1a05b-insights-runtime-extractor-tls podName:e0343507-90fb-4245-bd91-73476ed1a05b nodeName:}" failed. No retries permitted until 2026-07-21 01:37:05.468566073 +0000 UTC m=+49.055732030 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e0343507-90fb-4245-bd91-73476ed1a05b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mqxgn" (UID: "e0343507-90fb-4245-bd91-73476ed1a05b") : secret "insights-runtime-extractor-tls" not found Jul 21 01:37:04.978758 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:04.978735 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf95d\" (UniqueName: \"kubernetes.io/projected/e0343507-90fb-4245-bd91-73476ed1a05b-kube-api-access-kf95d\") pod \"insights-runtime-extractor-mqxgn\" (UID: \"e0343507-90fb-4245-bd91-73476ed1a05b\") " pod="openshift-insights/insights-runtime-extractor-mqxgn" Jul 21 01:37:05.224572 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:05.224496 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-4mtp7_9c380009-9bf5-4b73-803a-5a1ea9c1abf0/console-operator/1.log" Jul 21 01:37:05.224920 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:05.224803 2580 scope.go:117] "RemoveContainer" containerID="e0c0230bf56e92986c211934ee9ce14887e5e45db8e4bb96ef45ca8d8d7930e4" Jul 21 01:37:05.224978 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:37:05.224960 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-4mtp7_openshift-console-operator(9c380009-9bf5-4b73-803a-5a1ea9c1abf0)\"" pod="openshift-console-operator/console-operator-565b7b569d-4mtp7" podUID="9c380009-9bf5-4b73-803a-5a1ea9c1abf0" Jul 21 01:37:05.473416 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:05.473373 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e0343507-90fb-4245-bd91-73476ed1a05b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mqxgn\" (UID: \"e0343507-90fb-4245-bd91-73476ed1a05b\") " pod="openshift-insights/insights-runtime-extractor-mqxgn" Jul 21 01:37:05.473599 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:37:05.473554 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 21 01:37:05.473642 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:37:05.473606 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 21 01:37:05.473675 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:05.473528 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e0343507-90fb-4245-bd91-73476ed1a05b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mqxgn\" (UID: \"e0343507-90fb-4245-bd91-73476ed1a05b\") " pod="openshift-insights/insights-runtime-extractor-mqxgn" Jul 21 01:37:05.473675 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:37:05.473657 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0343507-90fb-4245-bd91-73476ed1a05b-insights-runtime-extractor-tls podName:e0343507-90fb-4245-bd91-73476ed1a05b nodeName:}" failed. No retries permitted until 2026-07-21 01:37:06.473637831 +0000 UTC m=+50.060803803 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e0343507-90fb-4245-bd91-73476ed1a05b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mqxgn" (UID: "e0343507-90fb-4245-bd91-73476ed1a05b") : secret "insights-runtime-extractor-tls" not found Jul 21 01:37:05.473763 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:37:05.473742 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e0343507-90fb-4245-bd91-73476ed1a05b-kube-rbac-proxy-cm podName:e0343507-90fb-4245-bd91-73476ed1a05b nodeName:}" failed. No retries permitted until 2026-07-21 01:37:06.473717332 +0000 UTC m=+50.060883292 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e0343507-90fb-4245-bd91-73476ed1a05b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mqxgn" (UID: "e0343507-90fb-4245-bd91-73476ed1a05b") : configmap "kube-rbac-proxy" not found Jul 21 01:37:06.482249 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:06.482209 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e0343507-90fb-4245-bd91-73476ed1a05b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mqxgn\" (UID: \"e0343507-90fb-4245-bd91-73476ed1a05b\") " pod="openshift-insights/insights-runtime-extractor-mqxgn" Jul 21 01:37:06.482674 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:06.482328 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e0343507-90fb-4245-bd91-73476ed1a05b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mqxgn\" (UID: \"e0343507-90fb-4245-bd91-73476ed1a05b\") " pod="openshift-insights/insights-runtime-extractor-mqxgn" Jul 21 01:37:06.482674 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:37:06.482368 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 21 01:37:06.482674 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:37:06.482430 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0343507-90fb-4245-bd91-73476ed1a05b-insights-runtime-extractor-tls podName:e0343507-90fb-4245-bd91-73476ed1a05b nodeName:}" failed. No retries permitted until 2026-07-21 01:37:08.482411015 +0000 UTC m=+52.069577014 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e0343507-90fb-4245-bd91-73476ed1a05b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mqxgn" (UID: "e0343507-90fb-4245-bd91-73476ed1a05b") : secret "insights-runtime-extractor-tls" not found Jul 21 01:37:06.482674 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:37:06.482483 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 21 01:37:06.482674 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:37:06.482562 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e0343507-90fb-4245-bd91-73476ed1a05b-kube-rbac-proxy-cm podName:e0343507-90fb-4245-bd91-73476ed1a05b nodeName:}" failed. No retries permitted until 2026-07-21 01:37:08.482550073 +0000 UTC m=+52.069716032 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e0343507-90fb-4245-bd91-73476ed1a05b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mqxgn" (UID: "e0343507-90fb-4245-bd91-73476ed1a05b") : configmap "kube-rbac-proxy" not found Jul 21 01:37:06.547327 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:06.547298 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5n4nl_a1ca57f9-11f8-48c3-9ff6-a7b2561fede0/dns-node-resolver/0.log" Jul 21 01:37:07.945102 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:07.945075 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zxcjm_1489d404-17a6-4bf1-9a85-9d3787aece54/node-ca/0.log" Jul 21 01:37:08.503166 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:08.503134 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e0343507-90fb-4245-bd91-73476ed1a05b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mqxgn\" (UID: \"e0343507-90fb-4245-bd91-73476ed1a05b\") " pod="openshift-insights/insights-runtime-extractor-mqxgn" Jul 21 01:37:08.503344 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:08.503226 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e0343507-90fb-4245-bd91-73476ed1a05b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mqxgn\" (UID: \"e0343507-90fb-4245-bd91-73476ed1a05b\") " pod="openshift-insights/insights-runtime-extractor-mqxgn" Jul 21 01:37:08.503344 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:37:08.503257 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 21 01:37:08.503344 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:37:08.503321 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e0343507-90fb-4245-bd91-73476ed1a05b-kube-rbac-proxy-cm podName:e0343507-90fb-4245-bd91-73476ed1a05b nodeName:}" failed. No retries permitted until 2026-07-21 01:37:12.503304228 +0000 UTC m=+56.090470184 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e0343507-90fb-4245-bd91-73476ed1a05b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mqxgn" (UID: "e0343507-90fb-4245-bd91-73476ed1a05b") : configmap "kube-rbac-proxy" not found Jul 21 01:37:08.503501 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:37:08.503369 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 21 01:37:08.503501 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:37:08.503422 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0343507-90fb-4245-bd91-73476ed1a05b-insights-runtime-extractor-tls podName:e0343507-90fb-4245-bd91-73476ed1a05b nodeName:}" failed. No retries permitted until 2026-07-21 01:37:12.5034097 +0000 UTC m=+56.090575662 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e0343507-90fb-4245-bd91-73476ed1a05b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mqxgn" (UID: "e0343507-90fb-4245-bd91-73476ed1a05b") : secret "insights-runtime-extractor-tls" not found Jul 21 01:37:08.545764 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:08.545735 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bmgfc_1d402931-b366-48df-acda-2e144e7c6c70/serve-healthcheck-canary/0.log" Jul 21 01:37:08.704978 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:08.704942 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6d9baf0-743b-48f5-9374-4443c65ea048-service-ca-bundle\") pod \"router-default-6f7df7d7cb-gcx5w\" (UID: \"e6d9baf0-743b-48f5-9374-4443c65ea048\") " pod="openshift-ingress/router-default-6f7df7d7cb-gcx5w" Jul 21 01:37:08.705177 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:08.704998 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd13d7ca-7c61-4c40-a86e-a58c1a7d1390-registry-tls\") pod \"image-registry-b47d488dd-h2wcn\" (UID: \"fd13d7ca-7c61-4c40-a86e-a58c1a7d1390\") " pod="openshift-image-registry/image-registry-b47d488dd-h2wcn" Jul 21 01:37:08.705177 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:08.705030 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b7843343-f8dd-4ea6-a8f4-e3153a0b3ab2-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-g86ww\" (UID: \"b7843343-f8dd-4ea6-a8f4-e3153a0b3ab2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-g86ww" Jul 21 01:37:08.705177 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:08.705067 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1e930d67-d711-419c-af02-86b43027a95d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-9ctn2\" (UID: \"1e930d67-d711-419c-af02-86b43027a95d\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-9ctn2" Jul 21 01:37:08.705177 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:37:08.705130 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e6d9baf0-743b-48f5-9374-4443c65ea048-service-ca-bundle podName:e6d9baf0-743b-48f5-9374-4443c65ea048 nodeName:}" failed. No retries permitted until 2026-07-21 01:37:24.705112003 +0000 UTC m=+68.292277965 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e6d9baf0-743b-48f5-9374-4443c65ea048-service-ca-bundle") pod "router-default-6f7df7d7cb-gcx5w" (UID: "e6d9baf0-743b-48f5-9374-4443c65ea048") : configmap references non-existent config key: service-ca.crt Jul 21 01:37:08.705177 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:37:08.705166 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 21 01:37:08.705424 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:37:08.705186 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 21 01:37:08.705424 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:08.705209 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6d9baf0-743b-48f5-9374-4443c65ea048-metrics-certs\") pod \"router-default-6f7df7d7cb-gcx5w\" (UID: \"e6d9baf0-743b-48f5-9374-4443c65ea048\") " pod="openshift-ingress/router-default-6f7df7d7cb-gcx5w" Jul 21 01:37:08.705424 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:37:08.705242 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7843343-f8dd-4ea6-a8f4-e3153a0b3ab2-samples-operator-tls podName:b7843343-f8dd-4ea6-a8f4-e3153a0b3ab2 nodeName:}" failed. No retries permitted until 2026-07-21 01:37:24.705230952 +0000 UTC m=+68.292396927 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b7843343-f8dd-4ea6-a8f4-e3153a0b3ab2-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-g86ww" (UID: "b7843343-f8dd-4ea6-a8f4-e3153a0b3ab2") : secret "samples-operator-tls" not found Jul 21 01:37:08.705424 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:37:08.705246 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 21 01:37:08.705424 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:37:08.705191 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b47d488dd-h2wcn: secret "image-registry-tls" not found Jul 21 01:37:08.705424 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:37:08.705282 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 21 01:37:08.705424 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:37:08.705307 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e930d67-d711-419c-af02-86b43027a95d-cluster-monitoring-operator-tls podName:1e930d67-d711-419c-af02-86b43027a95d nodeName:}" failed. No retries permitted until 2026-07-21 01:37:24.705287329 +0000 UTC m=+68.292453285 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1e930d67-d711-419c-af02-86b43027a95d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-9ctn2" (UID: "1e930d67-d711-419c-af02-86b43027a95d") : secret "cluster-monitoring-operator-tls" not found Jul 21 01:37:08.705424 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:37:08.705329 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6d9baf0-743b-48f5-9374-4443c65ea048-metrics-certs podName:e6d9baf0-743b-48f5-9374-4443c65ea048 nodeName:}" failed. No retries permitted until 2026-07-21 01:37:24.705313824 +0000 UTC m=+68.292479780 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e6d9baf0-743b-48f5-9374-4443c65ea048-metrics-certs") pod "router-default-6f7df7d7cb-gcx5w" (UID: "e6d9baf0-743b-48f5-9374-4443c65ea048") : secret "router-metrics-certs-default" not found Jul 21 01:37:08.705424 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:37:08.705347 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd13d7ca-7c61-4c40-a86e-a58c1a7d1390-registry-tls podName:fd13d7ca-7c61-4c40-a86e-a58c1a7d1390 nodeName:}" failed. No retries permitted until 2026-07-21 01:37:24.705336525 +0000 UTC m=+68.292502491 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fd13d7ca-7c61-4c40-a86e-a58c1a7d1390-registry-tls") pod "image-registry-b47d488dd-h2wcn" (UID: "fd13d7ca-7c61-4c40-a86e-a58c1a7d1390") : secret "image-registry-tls" not found Jul 21 01:37:08.806124 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:08.806027 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0add4762-9fef-4072-a05a-cdca30f4cdcf-metrics-tls\") pod \"dns-default-v899s\" (UID: \"0add4762-9fef-4072-a05a-cdca30f4cdcf\") " pod="openshift-dns/dns-default-v899s" Jul 21 01:37:08.806272 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:37:08.806204 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 01:37:08.806272 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:37:08.806263 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0add4762-9fef-4072-a05a-cdca30f4cdcf-metrics-tls podName:0add4762-9fef-4072-a05a-cdca30f4cdcf nodeName:}" failed. No retries permitted until 2026-07-21 01:37:24.806249525 +0000 UTC m=+68.393415484 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0add4762-9fef-4072-a05a-cdca30f4cdcf-metrics-tls") pod "dns-default-v899s" (UID: "0add4762-9fef-4072-a05a-cdca30f4cdcf") : secret "dns-default-metrics-tls" not found Jul 21 01:37:10.926823 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:10.926781 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/15628691-46c2-418e-8eca-47f963849e13-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-5gfm8\" (UID: \"15628691-46c2-418e-8eca-47f963849e13\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-5gfm8" Jul 21 01:37:10.927252 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:37:10.926922 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 21 01:37:10.927252 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:37:10.926981 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15628691-46c2-418e-8eca-47f963849e13-networking-console-plugin-cert podName:15628691-46c2-418e-8eca-47f963849e13 nodeName:}" failed. No retries permitted until 2026-07-21 01:37:26.926966343 +0000 UTC m=+70.514132304 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/15628691-46c2-418e-8eca-47f963849e13-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-5gfm8" (UID: "15628691-46c2-418e-8eca-47f963849e13") : secret "networking-console-plugin-cert" not found Jul 21 01:37:12.542120 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:12.542077 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e0343507-90fb-4245-bd91-73476ed1a05b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mqxgn\" (UID: \"e0343507-90fb-4245-bd91-73476ed1a05b\") " pod="openshift-insights/insights-runtime-extractor-mqxgn" Jul 21 01:37:12.542628 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:12.542176 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e0343507-90fb-4245-bd91-73476ed1a05b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mqxgn\" (UID: \"e0343507-90fb-4245-bd91-73476ed1a05b\") " pod="openshift-insights/insights-runtime-extractor-mqxgn" Jul 21 01:37:12.542628 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:37:12.542241 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 21 01:37:12.542628 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:37:12.542323 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0343507-90fb-4245-bd91-73476ed1a05b-insights-runtime-extractor-tls podName:e0343507-90fb-4245-bd91-73476ed1a05b nodeName:}" failed. No retries permitted until 2026-07-21 01:37:20.542302257 +0000 UTC m=+64.129468237 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e0343507-90fb-4245-bd91-73476ed1a05b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mqxgn" (UID: "e0343507-90fb-4245-bd91-73476ed1a05b") : secret "insights-runtime-extractor-tls" not found Jul 21 01:37:12.542796 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:12.542656 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e0343507-90fb-4245-bd91-73476ed1a05b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mqxgn\" (UID: \"e0343507-90fb-4245-bd91-73476ed1a05b\") " pod="openshift-insights/insights-runtime-extractor-mqxgn" Jul 21 01:37:13.182927 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:13.182892 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-4mtp7" Jul 21 01:37:13.183106 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:13.182952 2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-565b7b569d-4mtp7" Jul 21 01:37:13.183296 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:13.183283 2580 scope.go:117] "RemoveContainer" containerID="e0c0230bf56e92986c211934ee9ce14887e5e45db8e4bb96ef45ca8d8d7930e4" Jul 21 01:37:13.183475 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:37:13.183429 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-4mtp7_openshift-console-operator(9c380009-9bf5-4b73-803a-5a1ea9c1abf0)\"" pod="openshift-console-operator/console-operator-565b7b569d-4mtp7" podUID="9c380009-9bf5-4b73-803a-5a1ea9c1abf0" Jul 21 01:37:19.130908 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:19.130879 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5m9tn" Jul 21 01:37:20.611265 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:20.611220 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e0343507-90fb-4245-bd91-73476ed1a05b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mqxgn\" (UID: \"e0343507-90fb-4245-bd91-73476ed1a05b\") " pod="openshift-insights/insights-runtime-extractor-mqxgn" Jul 21 01:37:20.613493 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:20.613470 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e0343507-90fb-4245-bd91-73476ed1a05b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mqxgn\" (UID: \"e0343507-90fb-4245-bd91-73476ed1a05b\") " pod="openshift-insights/insights-runtime-extractor-mqxgn" Jul 21 01:37:20.690953 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:20.690922 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5pn52\"" Jul 21 01:37:20.697677 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:20.697658 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mqxgn" Jul 21 01:37:20.827863 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:20.827828 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mqxgn"] Jul 21 01:37:20.830908 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:37:20.830883 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0343507_90fb_4245_bd91_73476ed1a05b.slice/crio-aed45133b67c48d513daae27b56e814220a1e43527a5661b516745812271bef7 WatchSource:0}: Error finding container aed45133b67c48d513daae27b56e814220a1e43527a5661b516745812271bef7: Status 404 returned error can't find the container with id aed45133b67c48d513daae27b56e814220a1e43527a5661b516745812271bef7 Jul 21 01:37:21.278005 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:21.277966 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mqxgn" event={"ID":"e0343507-90fb-4245-bd91-73476ed1a05b","Type":"ContainerStarted","Data":"aed45133b67c48d513daae27b56e814220a1e43527a5661b516745812271bef7"} Jul 21 01:37:22.632842 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:22.632805 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7346c638-55dd-4d75-9d69-cead4209e04c-metrics-certs\") pod \"network-metrics-daemon-vdpxq\" (UID: \"7346c638-55dd-4d75-9d69-cead4209e04c\") " pod="openshift-multus/network-metrics-daemon-vdpxq" Jul 21 01:37:22.633310 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:22.632849 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17bdceb8-8016-4d0b-8b3c-f36507279428-original-pull-secret\") pod \"global-pull-secret-syncer-24nj2\" (UID: \"17bdceb8-8016-4d0b-8b3c-f36507279428\") " pod="kube-system/global-pull-secret-syncer-24nj2" Jul 21 01:37:22.633310 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:22.632981 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlprc\" (UniqueName: \"kubernetes.io/projected/1527759a-5529-4c49-a9e5-cdc780b0b21a-kube-api-access-mlprc\") pod \"network-check-target-pczgw\" (UID: \"1527759a-5529-4c49-a9e5-cdc780b0b21a\") " pod="openshift-network-diagnostics/network-check-target-pczgw" Jul 21 01:37:22.635413 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:22.635382 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 21 01:37:22.635569 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:22.635484 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 21 01:37:22.635760 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:22.635731 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlprc\" (UniqueName: \"kubernetes.io/projected/1527759a-5529-4c49-a9e5-cdc780b0b21a-kube-api-access-mlprc\") pod \"network-check-target-pczgw\" (UID: \"1527759a-5529-4c49-a9e5-cdc780b0b21a\") " pod="openshift-network-diagnostics/network-check-target-pczgw" Jul 21 01:37:22.645836 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:22.645806 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17bdceb8-8016-4d0b-8b3c-f36507279428-original-pull-secret\") pod \"global-pull-secret-syncer-24nj2\" (UID: \"17bdceb8-8016-4d0b-8b3c-f36507279428\") " pod="kube-system/global-pull-secret-syncer-24nj2" Jul 21 01:37:22.645970 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:22.645907 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7346c638-55dd-4d75-9d69-cead4209e04c-metrics-certs\") pod \"network-metrics-daemon-vdpxq\" (UID: \"7346c638-55dd-4d75-9d69-cead4209e04c\") " pod="openshift-multus/network-metrics-daemon-vdpxq" Jul 21 01:37:22.684939 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:22.684908 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5qdkq\"" Jul 21 01:37:22.691722 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:22.691698 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-8s87n\"" Jul 21 01:37:22.692648 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:22.692630 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vdpxq" Jul 21 01:37:22.694109 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:22.694091 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-24nj2" Jul 21 01:37:22.699881 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:22.699863 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pczgw" Jul 21 01:37:22.855843 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:22.855766 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vdpxq"] Jul 21 01:37:22.859609 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:37:22.859575 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7346c638_55dd_4d75_9d69_cead4209e04c.slice/crio-5d525c1abcc580beb5c60c95f4775eff1589dc6e33d770d9bbcd1460be9eb433 WatchSource:0}: Error finding container 5d525c1abcc580beb5c60c95f4775eff1589dc6e33d770d9bbcd1460be9eb433: Status 404 returned error can't find the container with id 5d525c1abcc580beb5c60c95f4775eff1589dc6e33d770d9bbcd1460be9eb433 Jul 21 01:37:23.085070 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:23.085035 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pczgw"] Jul 21 01:37:23.087420 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:23.087389 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-24nj2"] Jul 21 01:37:23.286194 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:23.286158 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vdpxq" event={"ID":"7346c638-55dd-4d75-9d69-cead4209e04c","Type":"ContainerStarted","Data":"5d525c1abcc580beb5c60c95f4775eff1589dc6e33d770d9bbcd1460be9eb433"} Jul 21 01:37:24.754706 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:24.754663 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd13d7ca-7c61-4c40-a86e-a58c1a7d1390-registry-tls\") pod \"image-registry-b47d488dd-h2wcn\" (UID: \"fd13d7ca-7c61-4c40-a86e-a58c1a7d1390\") " pod="openshift-image-registry/image-registry-b47d488dd-h2wcn" Jul 21 01:37:24.755181 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:24.754726 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b7843343-f8dd-4ea6-a8f4-e3153a0b3ab2-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-g86ww\" (UID: \"b7843343-f8dd-4ea6-a8f4-e3153a0b3ab2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-g86ww" Jul 21 01:37:24.755181 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:24.754767 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1e930d67-d711-419c-af02-86b43027a95d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-9ctn2\" (UID: \"1e930d67-d711-419c-af02-86b43027a95d\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-9ctn2" Jul 21 01:37:24.755181 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:24.754824 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6d9baf0-743b-48f5-9374-4443c65ea048-metrics-certs\") pod \"router-default-6f7df7d7cb-gcx5w\" (UID: \"e6d9baf0-743b-48f5-9374-4443c65ea048\") " pod="openshift-ingress/router-default-6f7df7d7cb-gcx5w" Jul 21 01:37:24.755181 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:24.754858 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6d9baf0-743b-48f5-9374-4443c65ea048-service-ca-bundle\") pod \"router-default-6f7df7d7cb-gcx5w\" (UID: \"e6d9baf0-743b-48f5-9374-4443c65ea048\") " pod="openshift-ingress/router-default-6f7df7d7cb-gcx5w" Jul 21 01:37:24.755620 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:24.755592 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6d9baf0-743b-48f5-9374-4443c65ea048-service-ca-bundle\") pod \"router-default-6f7df7d7cb-gcx5w\" (UID: \"e6d9baf0-743b-48f5-9374-4443c65ea048\") " pod="openshift-ingress/router-default-6f7df7d7cb-gcx5w" Jul 21 01:37:24.757737 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:24.757691 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd13d7ca-7c61-4c40-a86e-a58c1a7d1390-registry-tls\") pod \"image-registry-b47d488dd-h2wcn\" (UID: \"fd13d7ca-7c61-4c40-a86e-a58c1a7d1390\") " pod="openshift-image-registry/image-registry-b47d488dd-h2wcn" Jul 21 01:37:24.757737 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:24.757707 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b7843343-f8dd-4ea6-a8f4-e3153a0b3ab2-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-g86ww\" (UID: \"b7843343-f8dd-4ea6-a8f4-e3153a0b3ab2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-g86ww" Jul 21 01:37:24.757878 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:24.757845 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6d9baf0-743b-48f5-9374-4443c65ea048-metrics-certs\") pod \"router-default-6f7df7d7cb-gcx5w\" (UID: \"e6d9baf0-743b-48f5-9374-4443c65ea048\") " pod="openshift-ingress/router-default-6f7df7d7cb-gcx5w" Jul 21 01:37:24.757920 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:24.757845 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1e930d67-d711-419c-af02-86b43027a95d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-9ctn2\" (UID: \"1e930d67-d711-419c-af02-86b43027a95d\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-9ctn2" Jul 21 01:37:24.856262 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:24.856222 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0add4762-9fef-4072-a05a-cdca30f4cdcf-metrics-tls\") pod \"dns-default-v899s\" (UID: \"0add4762-9fef-4072-a05a-cdca30f4cdcf\") " pod="openshift-dns/dns-default-v899s" Jul 21 01:37:24.858402 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:24.858375 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0add4762-9fef-4072-a05a-cdca30f4cdcf-metrics-tls\") pod \"dns-default-v899s\" (UID: \"0add4762-9fef-4072-a05a-cdca30f4cdcf\") " pod="openshift-dns/dns-default-v899s" Jul 21 01:37:24.870807 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:24.870782 2580 scope.go:117] "RemoveContainer" containerID="e0c0230bf56e92986c211934ee9ce14887e5e45db8e4bb96ef45ca8d8d7930e4" Jul 21 01:37:24.937243 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:24.937211 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-r74bp\"" Jul 21 01:37:24.944284 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:24.944263 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b47d488dd-h2wcn" Jul 21 01:37:24.968038 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:24.968008 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-qdcfq\"" Jul 21 01:37:24.976031 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:24.976014 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6f7df7d7cb-gcx5w" Jul 21 01:37:24.991531 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:24.991506 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-727qm\"" Jul 21 01:37:25.000331 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:25.000308 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-9ctn2" Jul 21 01:37:25.013086 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:25.013028 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-tsgng\"" Jul 21 01:37:25.021123 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:25.021100 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-g86ww" Jul 21 01:37:25.022762 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:25.022748 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wcxpb\"" Jul 21 01:37:25.030820 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:25.030803 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-v899s" Jul 21 01:37:25.254202 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:37:25.254176 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17bdceb8_8016_4d0b_8b3c_f36507279428.slice/crio-b66ca47f5ade66ccb8442424a17bb051e28b21e97077309ff79632513c5c5280 WatchSource:0}: Error finding container b66ca47f5ade66ccb8442424a17bb051e28b21e97077309ff79632513c5c5280: Status 404 returned error can't find the container with id b66ca47f5ade66ccb8442424a17bb051e28b21e97077309ff79632513c5c5280 Jul 21 01:37:25.294287 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:25.294248 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-24nj2" event={"ID":"17bdceb8-8016-4d0b-8b3c-f36507279428","Type":"ContainerStarted","Data":"b66ca47f5ade66ccb8442424a17bb051e28b21e97077309ff79632513c5c5280"} Jul 21 01:37:25.366852 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:37:25.366710 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1527759a_5529_4c49_a9e5_cdc780b0b21a.slice/crio-5d4e376c0cba1a7c6c9cc57ee374bb54fe5e0567e8ef7efcd21ee9e6e159ace9 WatchSource:0}: Error finding container 5d4e376c0cba1a7c6c9cc57ee374bb54fe5e0567e8ef7efcd21ee9e6e159ace9: Status 404 returned error can't find the container with id 5d4e376c0cba1a7c6c9cc57ee374bb54fe5e0567e8ef7efcd21ee9e6e159ace9 Jul 21 01:37:26.319499 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:26.319377 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pczgw" event={"ID":"1527759a-5529-4c49-a9e5-cdc780b0b21a","Type":"ContainerStarted","Data":"5d4e376c0cba1a7c6c9cc57ee374bb54fe5e0567e8ef7efcd21ee9e6e159ace9"} Jul 21 01:37:26.479304 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:26.479225 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-g86ww"] Jul 21 01:37:26.497416 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:26.497306 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-v899s"] Jul 21 01:37:26.524332 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:26.524244 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-b47d488dd-h2wcn"] Jul 21 01:37:26.543768 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:26.543692 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6f7df7d7cb-gcx5w"] Jul 21 01:37:26.696192 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:37:26.693724 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6d9baf0_743b_48f5_9374_4443c65ea048.slice/crio-b653636dc891ef1d842daadab42bcef438d6b0731078f3ac89f51b761037d643 WatchSource:0}: Error finding container b653636dc891ef1d842daadab42bcef438d6b0731078f3ac89f51b761037d643: Status 404 returned error can't find the container with id b653636dc891ef1d842daadab42bcef438d6b0731078f3ac89f51b761037d643 Jul 21 01:37:26.745668 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:26.745569 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-9ctn2"] Jul 21 01:37:26.979916 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:26.979865 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/15628691-46c2-418e-8eca-47f963849e13-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-5gfm8\" (UID: \"15628691-46c2-418e-8eca-47f963849e13\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-5gfm8" Jul 21 01:37:26.991191 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:26.990586 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/15628691-46c2-418e-8eca-47f963849e13-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-5gfm8\" (UID: \"15628691-46c2-418e-8eca-47f963849e13\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-5gfm8" Jul 21 01:37:27.164147 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:27.164115 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-6wkg4\"" Jul 21 01:37:27.171930 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:27.171896 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-5gfm8" Jul 21 01:37:27.330473 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:27.330280 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-v899s" event={"ID":"0add4762-9fef-4072-a05a-cdca30f4cdcf","Type":"ContainerStarted","Data":"4a3a4a83bf1ff9418e72ccc63b96e2154807d2cdb231337f64c1bc49ad0b6532"} Jul 21 01:37:27.334359 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:27.334281 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-9ctn2" event={"ID":"1e930d67-d711-419c-af02-86b43027a95d","Type":"ContainerStarted","Data":"b620d17e4750bad7de501cf186e5bf7ca59dcabdc3bc7d6a2112bfd2e046266f"} Jul 21 01:37:27.337510 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:27.337275 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-4mtp7_9c380009-9bf5-4b73-803a-5a1ea9c1abf0/console-operator/1.log" Jul 21 01:37:27.337510 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:27.337372 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-4mtp7" event={"ID":"9c380009-9bf5-4b73-803a-5a1ea9c1abf0","Type":"ContainerStarted","Data":"5b4386c0723bfc1637f108e0b446ceadc83afff3e6337e2db206728b0eecf6dc"} Jul 21 01:37:27.338837 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:27.338775 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-4mtp7" Jul 21 01:37:27.342622 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:27.342376 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vdpxq" event={"ID":"7346c638-55dd-4d75-9d69-cead4209e04c","Type":"ContainerStarted","Data":"a1af5effd741922665295331ac023ba33d6b1ee265927671dd98b2c981a2e984"} Jul 21 01:37:27.342622 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:27.342427 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vdpxq" event={"ID":"7346c638-55dd-4d75-9d69-cead4209e04c","Type":"ContainerStarted","Data":"beeadfaea7d773a08313a29ba1f80f37c4ccd47e283fcd56bca275e6a7e4c9ed"} Jul 21 01:37:27.345302 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:27.345178 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pczgw" event={"ID":"1527759a-5529-4c49-a9e5-cdc780b0b21a","Type":"ContainerStarted","Data":"b6251fed2e42b27109c22aa0a3c2e7029e515568a6d012739e95311ee73e4a53"} Jul 21 01:37:27.345795 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:27.345690 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-pczgw" Jul 21 01:37:27.349339 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:27.349283 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6f7df7d7cb-gcx5w" event={"ID":"e6d9baf0-743b-48f5-9374-4443c65ea048","Type":"ContainerStarted","Data":"8dcaf0d8a349654bb63c45c021525ce3e4b3ddc9f3790028185554de22bf0b8e"} Jul 21 01:37:27.349339 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:27.349325 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6f7df7d7cb-gcx5w" event={"ID":"e6d9baf0-743b-48f5-9374-4443c65ea048","Type":"ContainerStarted","Data":"b653636dc891ef1d842daadab42bcef438d6b0731078f3ac89f51b761037d643"} Jul 21 01:37:27.352079 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:27.351415 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b47d488dd-h2wcn" event={"ID":"fd13d7ca-7c61-4c40-a86e-a58c1a7d1390","Type":"ContainerStarted","Data":"298e0cb45bdb396913774f6b76b2f4bd244c1e2ff2049dfd86e0212576ef9885"} Jul 21 01:37:27.352079 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:27.351490 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b47d488dd-h2wcn" event={"ID":"fd13d7ca-7c61-4c40-a86e-a58c1a7d1390","Type":"ContainerStarted","Data":"4456b9b619688cbcc405888e7cfca728acc3794ac957252bdfad5b29578c84b9"} Jul 21 01:37:27.352079 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:27.351974 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-b47d488dd-h2wcn" Jul 21 01:37:27.355799 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:27.355751 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-565b7b569d-4mtp7" podStartSLOduration=56.717107216 podStartE2EDuration="1m4.355734058s" totalImagesPullingTime="7.638626842s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 01:36:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:37:27.354742007 +0000 UTC m=+70.941907985" watchObservedRunningTime="2026-07-21 01:37:27.355734058 +0000 UTC m=+70.942900039" Jul 21 01:37:27.360899 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:27.360872 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-5gfm8"] Jul 21 01:37:27.361821 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:27.361763 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mqxgn" event={"ID":"e0343507-90fb-4245-bd91-73476ed1a05b","Type":"ContainerStarted","Data":"09fd0a5bc240404d2928ecfb87dcfac4125b5f76f06bd016d2635b9a38f6a7de"} Jul 21 01:37:27.365629 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:27.365603 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-g86ww" event={"ID":"b7843343-f8dd-4ea6-a8f4-e3153a0b3ab2","Type":"ContainerStarted","Data":"d7c2025bdfe07a8807540d9ce89f3f5b62cd6c8c830a899c2b5707044b272be8"} Jul 21 01:37:27.365911 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:37:27.365886 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15628691_46c2_418e_8eca_47f963849e13.slice/crio-5e211d4b8a6ba685739ea88a7692b45e5a1f278ee1ffddab7df4cce6fe04218f WatchSource:0}: Error finding container 5e211d4b8a6ba685739ea88a7692b45e5a1f278ee1ffddab7df4cce6fe04218f: Status 404 returned error can't find the container with id 5e211d4b8a6ba685739ea88a7692b45e5a1f278ee1ffddab7df4cce6fe04218f Jul 21 01:37:27.372607 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:27.372557 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-vdpxq" podStartSLOduration=65.495601002 podStartE2EDuration="1m9.372538181s" totalImagesPullingTime="3.876937179s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 01:36:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:37:27.371870242 +0000 UTC m=+70.959036242" watchObservedRunningTime="2026-07-21 01:37:27.372538181 +0000 UTC m=+70.959704163" Jul 21 01:37:27.395768 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:27.395707 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-6f7df7d7cb-gcx5w" podStartSLOduration=67.39568576 podStartE2EDuration="1m7.39568576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 01:36:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:37:27.394661248 +0000 UTC m=+70.981827245" watchObservedRunningTime="2026-07-21 01:37:27.39568576 +0000 UTC m=+70.982851739" Jul 21 01:37:27.418740 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:27.416511 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-b47d488dd-h2wcn" podStartSLOduration=69.416488042 podStartE2EDuration="1m9.416488042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 01:36:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:37:27.41605401 +0000 UTC m=+71.003219990" watchObservedRunningTime="2026-07-21 01:37:27.416488042 +0000 UTC m=+71.003654025" Jul 21 01:37:27.433536 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:27.432515 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-pczgw" podStartSLOduration=70.432490194 podStartE2EDuration="1m10.432490194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 01:36:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:37:27.431234176 +0000 UTC m=+71.018400180" watchObservedRunningTime="2026-07-21 01:37:27.432490194 +0000 UTC m=+71.019656172" Jul 21 01:37:27.721281 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:27.721098 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-565b7b569d-4mtp7" Jul 21 01:37:27.976798 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:27.976697 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-6f7df7d7cb-gcx5w" Jul 21 01:37:27.979650 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:27.979618 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-6f7df7d7cb-gcx5w" Jul 21 01:37:28.380216 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:28.378876 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-5gfm8" event={"ID":"15628691-46c2-418e-8eca-47f963849e13","Type":"ContainerStarted","Data":"5e211d4b8a6ba685739ea88a7692b45e5a1f278ee1ffddab7df4cce6fe04218f"} Jul 21 01:37:28.380216 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:28.379599 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-6f7df7d7cb-gcx5w" Jul 21 01:37:28.380999 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:28.380980 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-6f7df7d7cb-gcx5w" Jul 21 01:37:35.406931 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:35.406896 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-v899s" event={"ID":"0add4762-9fef-4072-a05a-cdca30f4cdcf","Type":"ContainerStarted","Data":"7da9c638667ede46bae748532825971ef3dc47c3d43345cb3b6567bd7681a2ca"} Jul 21 01:37:35.407410 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:35.406951 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-v899s" event={"ID":"0add4762-9fef-4072-a05a-cdca30f4cdcf","Type":"ContainerStarted","Data":"53ccddce4848d4eda25bed6465990d9f956b5778a9d66d2fcc77f014d911dbb1"} Jul 21 01:37:35.407410 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:35.407094 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-v899s" Jul 21 01:37:35.408478 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:35.408436 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-5gfm8" event={"ID":"15628691-46c2-418e-8eca-47f963849e13","Type":"ContainerStarted","Data":"07574ab0267b5b366428789387cb34f407021198aeb57cc7bffd875b2bab6fa2"} Jul 21 01:37:35.409995 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:35.409974 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-9ctn2" event={"ID":"1e930d67-d711-419c-af02-86b43027a95d","Type":"ContainerStarted","Data":"416f862ba486338a6970a06cbdf5f9ede8c9484bccdb9767271ad7be2521f110"} Jul 21 01:37:35.411943 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:35.411921 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mqxgn" event={"ID":"e0343507-90fb-4245-bd91-73476ed1a05b","Type":"ContainerStarted","Data":"5c881ceacf8707e165e1615db19144d3ab28360c25d90ef78f674c822d637fdc"} Jul 21 01:37:35.413717 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:35.413693 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-g86ww" event={"ID":"b7843343-f8dd-4ea6-a8f4-e3153a0b3ab2","Type":"ContainerStarted","Data":"8835e44e92c5a1c00306b2275180484424d342cb4940e04b53ce785d58aae9c6"} Jul 21 01:37:35.413811 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:35.413735 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-g86ww" event={"ID":"b7843343-f8dd-4ea6-a8f4-e3153a0b3ab2","Type":"ContainerStarted","Data":"5cebecbffda31701c04938287bc3f817a393c0500772d6d454ae954ea1c775d4"} Jul 21 01:37:35.415192 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:35.415172 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-24nj2" event={"ID":"17bdceb8-8016-4d0b-8b3c-f36507279428","Type":"ContainerStarted","Data":"5eb4a3fd23cc41da77100a99ec5d25944a6b80109b87d694a863e53201dd955c"} Jul 21 01:37:35.427331 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:35.427276 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-v899s" podStartSLOduration=35.438032796 podStartE2EDuration="43.427227881s" totalImagesPullingTime="7.989195085s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 01:36:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:37:35.425995876 +0000 UTC m=+79.013161866" watchObservedRunningTime="2026-07-21 01:37:35.427227881 +0000 UTC m=+79.014393861" Jul 21 01:37:35.441565 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:35.441503 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-9ctn2" podStartSLOduration=67.508263079 podStartE2EDuration="1m15.441483531s" totalImagesPullingTime="7.933220452s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 01:36:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:37:35.440383276 +0000 UTC m=+79.027549257" watchObservedRunningTime="2026-07-21 01:37:35.441483531 +0000 UTC m=+79.028649509" Jul 21 01:37:35.469021 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:35.468910 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-24nj2" podStartSLOduration=68.038813214 podStartE2EDuration="1m17.468890342s" totalImagesPullingTime="9.430077128s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 01:36:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:37:35.468492908 +0000 UTC m=+79.055658914" watchObservedRunningTime="2026-07-21 01:37:35.468890342 +0000 UTC m=+79.056056321" Jul 21 01:37:35.470022 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:35.469983 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-5gfm8" podStartSLOduration=33.150055819 podStartE2EDuration="40.469969578s" totalImagesPullingTime="7.319913759s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 01:36:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:37:35.455214623 +0000 UTC m=+79.042380602" watchObservedRunningTime="2026-07-21 01:37:35.469969578 +0000 UTC m=+79.057135602" Jul 21 01:37:35.500949 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:35.500889 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-g86ww" podStartSLOduration=65.5608987 podStartE2EDuration="1m13.500869103s" totalImagesPullingTime="7.939970403s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 01:36:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:37:35.500290183 +0000 UTC m=+79.087456162" watchObservedRunningTime="2026-07-21 01:37:35.500869103 +0000 UTC m=+79.088035083" Jul 21 01:37:39.432619 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:39.432569 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mqxgn" event={"ID":"e0343507-90fb-4245-bd91-73476ed1a05b","Type":"ContainerStarted","Data":"4fc352bcd79a7efa4eb7386683f9f02b2cd21bd6274b88c337c5cf6a4268f3d7"} Jul 21 01:37:39.452931 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:39.452877 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-mqxgn" podStartSLOduration=17.847563921 podStartE2EDuration="35.452860724s" totalImagesPullingTime="17.605296803s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 01:37:04 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:37:39.451188572 +0000 UTC m=+83.038354550" watchObservedRunningTime="2026-07-21 01:37:39.452860724 +0000 UTC m=+83.040026703" Jul 21 01:37:44.948993 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:44.948953 2580 patch_prober.go:28] interesting pod/image-registry-b47d488dd-h2wcn container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 21 01:37:44.949380 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:44.949010 2580 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-b47d488dd-h2wcn" podUID="fd13d7ca-7c61-4c40-a86e-a58c1a7d1390" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 21 01:37:45.420879 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:45.420849 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-v899s" Jul 21 01:37:49.388231 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:49.388200 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-b47d488dd-h2wcn" Jul 21 01:37:50.408016 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:50.407981 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6b6c89d7f8-7tjjt"] Jul 21 01:37:50.410846 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:50.410827 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b6c89d7f8-7tjjt" Jul 21 01:37:50.413899 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:50.413879 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-r5vll\"" Jul 21 01:37:50.413899 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:50.413890 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 21 01:37:50.414094 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:50.413918 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 21 01:37:50.414094 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:50.413931 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 21 01:37:50.414094 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:50.414017 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 21 01:37:50.414256 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:50.414239 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 21 01:37:50.414316 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:50.414278 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 21 01:37:50.414316 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:50.414289 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 21 01:37:50.418904 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:50.418885 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 21 01:37:50.420139 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:50.420119 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b6c89d7f8-7tjjt"] Jul 21 01:37:50.472303 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:50.472271 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7fe2a9ec-987b-46a3-9773-3cc1e34d0259-service-ca\") pod \"console-6b6c89d7f8-7tjjt\" (UID: \"7fe2a9ec-987b-46a3-9773-3cc1e34d0259\") " pod="openshift-console/console-6b6c89d7f8-7tjjt" Jul 21 01:37:50.472303 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:50.472306 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7fe2a9ec-987b-46a3-9773-3cc1e34d0259-console-serving-cert\") pod \"console-6b6c89d7f8-7tjjt\" (UID: \"7fe2a9ec-987b-46a3-9773-3cc1e34d0259\") " pod="openshift-console/console-6b6c89d7f8-7tjjt" Jul 21 01:37:50.472568 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:50.472323 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7fe2a9ec-987b-46a3-9773-3cc1e34d0259-console-config\") pod \"console-6b6c89d7f8-7tjjt\" (UID: \"7fe2a9ec-987b-46a3-9773-3cc1e34d0259\") " pod="openshift-console/console-6b6c89d7f8-7tjjt" Jul 21 01:37:50.472568 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:50.472339 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7fe2a9ec-987b-46a3-9773-3cc1e34d0259-trusted-ca-bundle\") pod \"console-6b6c89d7f8-7tjjt\" (UID: \"7fe2a9ec-987b-46a3-9773-3cc1e34d0259\") " pod="openshift-console/console-6b6c89d7f8-7tjjt" Jul 21 01:37:50.472568 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:50.472420 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk26c\" (UniqueName: \"kubernetes.io/projected/7fe2a9ec-987b-46a3-9773-3cc1e34d0259-kube-api-access-nk26c\") pod \"console-6b6c89d7f8-7tjjt\" (UID: \"7fe2a9ec-987b-46a3-9773-3cc1e34d0259\") " pod="openshift-console/console-6b6c89d7f8-7tjjt" Jul 21 01:37:50.472568 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:50.472502 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7fe2a9ec-987b-46a3-9773-3cc1e34d0259-console-oauth-config\") pod \"console-6b6c89d7f8-7tjjt\" (UID: \"7fe2a9ec-987b-46a3-9773-3cc1e34d0259\") " pod="openshift-console/console-6b6c89d7f8-7tjjt" Jul 21 01:37:50.472568 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:50.472535 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7fe2a9ec-987b-46a3-9773-3cc1e34d0259-oauth-serving-cert\") pod \"console-6b6c89d7f8-7tjjt\" (UID: \"7fe2a9ec-987b-46a3-9773-3cc1e34d0259\") " pod="openshift-console/console-6b6c89d7f8-7tjjt" Jul 21 01:37:50.573658 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:50.573621 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7fe2a9ec-987b-46a3-9773-3cc1e34d0259-service-ca\") pod \"console-6b6c89d7f8-7tjjt\" (UID: \"7fe2a9ec-987b-46a3-9773-3cc1e34d0259\") " pod="openshift-console/console-6b6c89d7f8-7tjjt" Jul 21 01:37:50.573658 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:50.573664 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7fe2a9ec-987b-46a3-9773-3cc1e34d0259-console-serving-cert\") pod \"console-6b6c89d7f8-7tjjt\" (UID: \"7fe2a9ec-987b-46a3-9773-3cc1e34d0259\") " pod="openshift-console/console-6b6c89d7f8-7tjjt" Jul 21 01:37:50.573860 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:50.573680 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7fe2a9ec-987b-46a3-9773-3cc1e34d0259-console-config\") pod \"console-6b6c89d7f8-7tjjt\" (UID: \"7fe2a9ec-987b-46a3-9773-3cc1e34d0259\") " pod="openshift-console/console-6b6c89d7f8-7tjjt" Jul 21 01:37:50.573860 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:50.573704 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7fe2a9ec-987b-46a3-9773-3cc1e34d0259-trusted-ca-bundle\") pod \"console-6b6c89d7f8-7tjjt\" (UID: \"7fe2a9ec-987b-46a3-9773-3cc1e34d0259\") " pod="openshift-console/console-6b6c89d7f8-7tjjt" Jul 21 01:37:50.573860 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:50.573742 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk26c\" (UniqueName: \"kubernetes.io/projected/7fe2a9ec-987b-46a3-9773-3cc1e34d0259-kube-api-access-nk26c\") pod \"console-6b6c89d7f8-7tjjt\" (UID: \"7fe2a9ec-987b-46a3-9773-3cc1e34d0259\") " pod="openshift-console/console-6b6c89d7f8-7tjjt" Jul 21 01:37:50.573860 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:50.573774 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7fe2a9ec-987b-46a3-9773-3cc1e34d0259-console-oauth-config\") pod \"console-6b6c89d7f8-7tjjt\" (UID: \"7fe2a9ec-987b-46a3-9773-3cc1e34d0259\") " pod="openshift-console/console-6b6c89d7f8-7tjjt" Jul 21 01:37:50.573860 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:50.573815 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7fe2a9ec-987b-46a3-9773-3cc1e34d0259-oauth-serving-cert\") pod \"console-6b6c89d7f8-7tjjt\" (UID: \"7fe2a9ec-987b-46a3-9773-3cc1e34d0259\") " pod="openshift-console/console-6b6c89d7f8-7tjjt" Jul 21 01:37:50.574440 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:50.574413 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7fe2a9ec-987b-46a3-9773-3cc1e34d0259-service-ca\") pod \"console-6b6c89d7f8-7tjjt\" (UID: \"7fe2a9ec-987b-46a3-9773-3cc1e34d0259\") " pod="openshift-console/console-6b6c89d7f8-7tjjt" Jul 21 01:37:50.574576 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:50.574527 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7fe2a9ec-987b-46a3-9773-3cc1e34d0259-console-config\") pod \"console-6b6c89d7f8-7tjjt\" (UID: \"7fe2a9ec-987b-46a3-9773-3cc1e34d0259\") " pod="openshift-console/console-6b6c89d7f8-7tjjt" Jul 21 01:37:50.574576 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:50.574541 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7fe2a9ec-987b-46a3-9773-3cc1e34d0259-oauth-serving-cert\") pod \"console-6b6c89d7f8-7tjjt\" (UID: \"7fe2a9ec-987b-46a3-9773-3cc1e34d0259\") " pod="openshift-console/console-6b6c89d7f8-7tjjt" Jul 21 01:37:50.574681 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:50.574634 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7fe2a9ec-987b-46a3-9773-3cc1e34d0259-trusted-ca-bundle\") pod \"console-6b6c89d7f8-7tjjt\" (UID: \"7fe2a9ec-987b-46a3-9773-3cc1e34d0259\") " pod="openshift-console/console-6b6c89d7f8-7tjjt" Jul 21 01:37:50.576336 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:50.576309 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7fe2a9ec-987b-46a3-9773-3cc1e34d0259-console-oauth-config\") pod \"console-6b6c89d7f8-7tjjt\" (UID: \"7fe2a9ec-987b-46a3-9773-3cc1e34d0259\") " pod="openshift-console/console-6b6c89d7f8-7tjjt" Jul 21 01:37:50.576387 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:50.576324 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7fe2a9ec-987b-46a3-9773-3cc1e34d0259-console-serving-cert\") pod \"console-6b6c89d7f8-7tjjt\" (UID: \"7fe2a9ec-987b-46a3-9773-3cc1e34d0259\") " pod="openshift-console/console-6b6c89d7f8-7tjjt" Jul 21 01:37:50.581056 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:50.581033 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk26c\" (UniqueName: \"kubernetes.io/projected/7fe2a9ec-987b-46a3-9773-3cc1e34d0259-kube-api-access-nk26c\") pod \"console-6b6c89d7f8-7tjjt\" (UID: \"7fe2a9ec-987b-46a3-9773-3cc1e34d0259\") " pod="openshift-console/console-6b6c89d7f8-7tjjt" Jul 21 01:37:50.721149 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:50.721060 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b6c89d7f8-7tjjt" Jul 21 01:37:50.851811 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:50.851778 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b6c89d7f8-7tjjt"] Jul 21 01:37:50.855000 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:37:50.854966 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fe2a9ec_987b_46a3_9773_3cc1e34d0259.slice/crio-00bf32de0bd5db0f00a0a06432a58a5933c3ee9a1a1b185825b28698a63453c7 WatchSource:0}: Error finding container 00bf32de0bd5db0f00a0a06432a58a5933c3ee9a1a1b185825b28698a63453c7: Status 404 returned error can't find the container with id 00bf32de0bd5db0f00a0a06432a58a5933c3ee9a1a1b185825b28698a63453c7 Jul 21 01:37:51.475705 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:51.475666 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b6c89d7f8-7tjjt" event={"ID":"7fe2a9ec-987b-46a3-9773-3cc1e34d0259","Type":"ContainerStarted","Data":"00bf32de0bd5db0f00a0a06432a58a5933c3ee9a1a1b185825b28698a63453c7"} Jul 21 01:37:53.484691 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:53.484653 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b6c89d7f8-7tjjt" event={"ID":"7fe2a9ec-987b-46a3-9773-3cc1e34d0259","Type":"ContainerStarted","Data":"9dc1a85522a1f379c80b73926808313091c7893a523f90eb4aad0e64ca0b03a9"} Jul 21 01:37:53.503824 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:53.503774 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6b6c89d7f8-7tjjt" podStartSLOduration=1.019558927 podStartE2EDuration="3.50375588s" totalImagesPullingTime="2.484196953s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 01:37:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:37:53.502596961 +0000 UTC m=+97.089762940" watchObservedRunningTime="2026-07-21 01:37:53.50375588 +0000 UTC m=+97.090922221" Jul 21 01:37:57.500232 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:57.500203 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-9ctn2_1e930d67-d711-419c-af02-86b43027a95d/cluster-monitoring-operator/0.log" Jul 21 01:37:57.500658 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:57.500244 2580 generic.go:356] "Generic (PLEG): container finished" podID="1e930d67-d711-419c-af02-86b43027a95d" containerID="416f862ba486338a6970a06cbdf5f9ede8c9484bccdb9767271ad7be2521f110" exitCode=2 Jul 21 01:37:57.500658 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:57.500303 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-9ctn2" event={"ID":"1e930d67-d711-419c-af02-86b43027a95d","Type":"ContainerDied","Data":"416f862ba486338a6970a06cbdf5f9ede8c9484bccdb9767271ad7be2521f110"} Jul 21 01:37:57.500734 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:57.500667 2580 scope.go:117] "RemoveContainer" containerID="416f862ba486338a6970a06cbdf5f9ede8c9484bccdb9767271ad7be2521f110" Jul 21 01:37:58.506125 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:58.506094 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-9ctn2_1e930d67-d711-419c-af02-86b43027a95d/cluster-monitoring-operator/0.log" Jul 21 01:37:58.506613 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:58.506202 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-9ctn2" event={"ID":"1e930d67-d711-419c-af02-86b43027a95d","Type":"ContainerStarted","Data":"5bb64d902b11ee94716c18926d31caebef6940e216583fa42e8a1a8dc4cb5183"} Jul 21 01:37:59.386506 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:37:59.386471 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-pczgw" Jul 21 01:38:00.535382 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:00.535346 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-b47d488dd-h2wcn"] Jul 21 01:38:00.721382 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:00.721344 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6b6c89d7f8-7tjjt" Jul 21 01:38:00.721592 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:00.721404 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6b6c89d7f8-7tjjt" Jul 21 01:38:00.726760 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:00.726735 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6b6c89d7f8-7tjjt" Jul 21 01:38:01.110768 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:01.110737 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-7gzg9"] Jul 21 01:38:01.113101 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:01.113082 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-7gzg9" Jul 21 01:38:01.115402 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:01.115380 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 21 01:38:01.115852 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:01.115829 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 21 01:38:01.115975 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:01.115888 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-q294w\"" Jul 21 01:38:01.116512 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:01.116494 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 21 01:38:01.116634 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:01.116586 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 21 01:38:01.127106 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:01.127085 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-7gzg9"] Jul 21 01:38:01.157605 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:01.156926 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/016c43a1-1619-4b20-aa34-37f9a61b0016-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-7gzg9\" (UID: \"016c43a1-1619-4b20-aa34-37f9a61b0016\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-7gzg9" Jul 21 01:38:01.157605 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:01.157010 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/016c43a1-1619-4b20-aa34-37f9a61b0016-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-7gzg9\" (UID: \"016c43a1-1619-4b20-aa34-37f9a61b0016\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-7gzg9" Jul 21 01:38:01.157605 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:01.157046 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/016c43a1-1619-4b20-aa34-37f9a61b0016-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-7gzg9\" (UID: \"016c43a1-1619-4b20-aa34-37f9a61b0016\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-7gzg9" Jul 21 01:38:01.157605 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:01.157088 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/016c43a1-1619-4b20-aa34-37f9a61b0016-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-7gzg9\" (UID: \"016c43a1-1619-4b20-aa34-37f9a61b0016\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-7gzg9" Jul 21 01:38:01.157605 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:01.157124 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/016c43a1-1619-4b20-aa34-37f9a61b0016-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-7gzg9\" (UID: \"016c43a1-1619-4b20-aa34-37f9a61b0016\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-7gzg9" Jul 21 01:38:01.157605 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:01.157183 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs2b5\" (UniqueName: \"kubernetes.io/projected/016c43a1-1619-4b20-aa34-37f9a61b0016-kube-api-access-cs2b5\") pod \"kube-state-metrics-56c5594674-7gzg9\" (UID: \"016c43a1-1619-4b20-aa34-37f9a61b0016\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-7gzg9" Jul 21 01:38:01.246349 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:01.246310 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-7vvrs"] Jul 21 01:38:01.249279 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:01.249256 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7vvrs" Jul 21 01:38:01.252093 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:01.252064 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 21 01:38:01.252360 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:01.252336 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 21 01:38:01.252491 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:01.252365 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 21 01:38:01.252491 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:01.252387 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-wpz6w\"" Jul 21 01:38:01.257591 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:01.257569 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/016c43a1-1619-4b20-aa34-37f9a61b0016-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-7gzg9\" (UID: \"016c43a1-1619-4b20-aa34-37f9a61b0016\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-7gzg9" Jul 21 01:38:01.257703 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:01.257603 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs2b5\" (UniqueName: \"kubernetes.io/projected/016c43a1-1619-4b20-aa34-37f9a61b0016-kube-api-access-cs2b5\") pod \"kube-state-metrics-56c5594674-7gzg9\" (UID: \"016c43a1-1619-4b20-aa34-37f9a61b0016\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-7gzg9" Jul 21 01:38:01.257703 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:01.257644 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/016c43a1-1619-4b20-aa34-37f9a61b0016-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-7gzg9\" (UID: \"016c43a1-1619-4b20-aa34-37f9a61b0016\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-7gzg9" Jul 21 01:38:01.257703 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:01.257672 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/016c43a1-1619-4b20-aa34-37f9a61b0016-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-7gzg9\" (UID: \"016c43a1-1619-4b20-aa34-37f9a61b0016\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-7gzg9" Jul 21 01:38:01.257703 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:01.257703 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/016c43a1-1619-4b20-aa34-37f9a61b0016-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-7gzg9\" (UID: \"016c43a1-1619-4b20-aa34-37f9a61b0016\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-7gzg9" Jul 21 01:38:01.257900 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:01.257724 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/016c43a1-1619-4b20-aa34-37f9a61b0016-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-7gzg9\" (UID: \"016c43a1-1619-4b20-aa34-37f9a61b0016\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-7gzg9" Jul 21 01:38:01.257900 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:38:01.257828 2580 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 21 01:38:01.257900 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:38:01.257897 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/016c43a1-1619-4b20-aa34-37f9a61b0016-kube-state-metrics-tls podName:016c43a1-1619-4b20-aa34-37f9a61b0016 nodeName:}" failed. No retries permitted until 2026-07-21 01:38:01.757876359 +0000 UTC m=+105.345042321 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/016c43a1-1619-4b20-aa34-37f9a61b0016-kube-state-metrics-tls") pod "kube-state-metrics-56c5594674-7gzg9" (UID: "016c43a1-1619-4b20-aa34-37f9a61b0016") : secret "kube-state-metrics-tls" not found Jul 21 01:38:01.258135 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:01.258118 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/016c43a1-1619-4b20-aa34-37f9a61b0016-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-7gzg9\" (UID: \"016c43a1-1619-4b20-aa34-37f9a61b0016\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-7gzg9" Jul 21 01:38:01.258371 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:01.258349 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/016c43a1-1619-4b20-aa34-37f9a61b0016-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-7gzg9\" (UID: \"016c43a1-1619-4b20-aa34-37f9a61b0016\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-7gzg9" Jul 21 01:38:01.258420 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:01.258358 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/016c43a1-1619-4b20-aa34-37f9a61b0016-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-7gzg9\" (UID: \"016c43a1-1619-4b20-aa34-37f9a61b0016\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-7gzg9" Jul 21 01:38:01.260262 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:01.260241 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/016c43a1-1619-4b20-aa34-37f9a61b0016-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-7gzg9\" (UID: \"016c43a1-1619-4b20-aa34-37f9a61b0016\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-7gzg9" Jul 21 01:38:01.276775 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:01.276750 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs2b5\" (UniqueName: \"kubernetes.io/projected/016c43a1-1619-4b20-aa34-37f9a61b0016-kube-api-access-cs2b5\") pod \"kube-state-metrics-56c5594674-7gzg9\" (UID: \"016c43a1-1619-4b20-aa34-37f9a61b0016\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-7gzg9" Jul 21 01:38:01.359038 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:01.358999 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6484e153-4568-44ee-ba8d-4d4b13b5defd-node-exporter-accelerators-collector-config\") pod \"node-exporter-7vvrs\" (UID: \"6484e153-4568-44ee-ba8d-4d4b13b5defd\") " pod="openshift-monitoring/node-exporter-7vvrs" Jul 21 01:38:01.359235 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:01.359050 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6484e153-4568-44ee-ba8d-4d4b13b5defd-node-exporter-textfile\") pod \"node-exporter-7vvrs\" (UID: \"6484e153-4568-44ee-ba8d-4d4b13b5defd\") " pod="openshift-monitoring/node-exporter-7vvrs" Jul 21 01:38:01.359235 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:01.359068 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6484e153-4568-44ee-ba8d-4d4b13b5defd-sys\") pod \"node-exporter-7vvrs\" (UID: \"6484e153-4568-44ee-ba8d-4d4b13b5defd\") " pod="openshift-monitoring/node-exporter-7vvrs" Jul 21 01:38:01.359235 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:01.359085 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6484e153-4568-44ee-ba8d-4d4b13b5defd-root\") pod \"node-exporter-7vvrs\" (UID: \"6484e153-4568-44ee-ba8d-4d4b13b5defd\") " pod="openshift-monitoring/node-exporter-7vvrs" Jul 21 01:38:01.359235 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:01.359189 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6484e153-4568-44ee-ba8d-4d4b13b5defd-node-exporter-wtmp\") pod \"node-exporter-7vvrs\" (UID: \"6484e153-4568-44ee-ba8d-4d4b13b5defd\") " pod="openshift-monitoring/node-exporter-7vvrs" Jul 21 01:38:01.359235 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:01.359225 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfk6n\" (UniqueName: \"kubernetes.io/projected/6484e153-4568-44ee-ba8d-4d4b13b5defd-kube-api-access-zfk6n\") pod \"node-exporter-7vvrs\" (UID: \"6484e153-4568-44ee-ba8d-4d4b13b5defd\") " pod="openshift-monitoring/node-exporter-7vvrs" Jul 21 01:38:01.359509 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:01.359271 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6484e153-4568-44ee-ba8d-4d4b13b5defd-node-exporter-tls\") pod \"node-exporter-7vvrs\" (UID: \"6484e153-4568-44ee-ba8d-4d4b13b5defd\") " pod="openshift-monitoring/node-exporter-7vvrs" Jul 21 01:38:01.359509 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:01.359308 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6484e153-4568-44ee-ba8d-4d4b13b5defd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7vvrs\" (UID: \"6484e153-4568-44ee-ba8d-4d4b13b5defd\") " pod="openshift-monitoring/node-exporter-7vvrs" Jul 21 01:38:01.359509 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:01.359364 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6484e153-4568-44ee-ba8d-4d4b13b5defd-metrics-client-ca\") pod \"node-exporter-7vvrs\" (UID: \"6484e153-4568-44ee-ba8d-4d4b13b5defd\") " pod="openshift-monitoring/node-exporter-7vvrs" Jul 21 01:38:01.460337 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:01.460251 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6484e153-4568-44ee-ba8d-4d4b13b5defd-sys\") pod \"node-exporter-7vvrs\" (UID: \"6484e153-4568-44ee-ba8d-4d4b13b5defd\") " pod="openshift-monitoring/node-exporter-7vvrs" Jul 21 01:38:01.460337 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:01.460288 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6484e153-4568-44ee-ba8d-4d4b13b5defd-root\") pod \"node-exporter-7vvrs\" (UID: \"6484e153-4568-44ee-ba8d-4d4b13b5defd\") " pod="openshift-monitoring/node-exporter-7vvrs" Jul 21 01:38:01.460615 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:01.460347 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6484e153-4568-44ee-ba8d-4d4b13b5defd-root\") pod \"node-exporter-7vvrs\" (UID: \"6484e153-4568-44ee-ba8d-4d4b13b5defd\") " pod="openshift-monitoring/node-exporter-7vvrs" Jul 21 01:38:01.460615 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:01.460355 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6484e153-4568-44ee-ba8d-4d4b13b5defd-sys\") pod \"node-exporter-7vvrs\" (UID: \"6484e153-4568-44ee-ba8d-4d4b13b5defd\") " pod="openshift-monitoring/node-exporter-7vvrs" Jul 21 01:38:01.460615 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:01.460374 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6484e153-4568-44ee-ba8d-4d4b13b5defd-node-exporter-wtmp\") pod \"node-exporter-7vvrs\" (UID: \"6484e153-4568-44ee-ba8d-4d4b13b5defd\") " pod="openshift-monitoring/node-exporter-7vvrs" Jul 21 01:38:01.460615 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:01.460401 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfk6n\" (UniqueName: \"kubernetes.io/projected/6484e153-4568-44ee-ba8d-4d4b13b5defd-kube-api-access-zfk6n\") pod \"node-exporter-7vvrs\" (UID: \"6484e153-4568-44ee-ba8d-4d4b13b5defd\") " pod="openshift-monitoring/node-exporter-7vvrs" Jul 21 01:38:01.460615 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:01.460418 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6484e153-4568-44ee-ba8d-4d4b13b5defd-node-exporter-tls\") pod \"node-exporter-7vvrs\" (UID: \"6484e153-4568-44ee-ba8d-4d4b13b5defd\") " pod="openshift-monitoring/node-exporter-7vvrs" Jul 21 01:38:01.460615 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:01.460438 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6484e153-4568-44ee-ba8d-4d4b13b5defd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7vvrs\" (UID: \"6484e153-4568-44ee-ba8d-4d4b13b5defd\") " pod="openshift-monitoring/node-exporter-7vvrs" Jul 21 01:38:01.460615 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:01.460488 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6484e153-4568-44ee-ba8d-4d4b13b5defd-metrics-client-ca\") pod \"node-exporter-7vvrs\" (UID: \"6484e153-4568-44ee-ba8d-4d4b13b5defd\") " pod="openshift-monitoring/node-exporter-7vvrs" Jul 21 01:38:01.460615 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:01.460494 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6484e153-4568-44ee-ba8d-4d4b13b5defd-node-exporter-wtmp\") pod \"node-exporter-7vvrs\" (UID: \"6484e153-4568-44ee-ba8d-4d4b13b5defd\") " pod="openshift-monitoring/node-exporter-7vvrs" Jul 21 01:38:01.460615 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:38:01.460579 2580 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 21 01:38:01.460615 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:01.460589 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6484e153-4568-44ee-ba8d-4d4b13b5defd-node-exporter-accelerators-collector-config\") pod \"node-exporter-7vvrs\" (UID: \"6484e153-4568-44ee-ba8d-4d4b13b5defd\") " pod="openshift-monitoring/node-exporter-7vvrs" Jul 21 01:38:01.461190 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:38:01.460637 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6484e153-4568-44ee-ba8d-4d4b13b5defd-node-exporter-tls podName:6484e153-4568-44ee-ba8d-4d4b13b5defd nodeName:}" failed. No retries permitted until 2026-07-21 01:38:01.960617736 +0000 UTC m=+105.547783698 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/6484e153-4568-44ee-ba8d-4d4b13b5defd-node-exporter-tls") pod "node-exporter-7vvrs" (UID: "6484e153-4568-44ee-ba8d-4d4b13b5defd") : secret "node-exporter-tls" not found Jul 21 01:38:01.461190 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:01.460663 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6484e153-4568-44ee-ba8d-4d4b13b5defd-node-exporter-textfile\") pod \"node-exporter-7vvrs\" (UID: \"6484e153-4568-44ee-ba8d-4d4b13b5defd\") " pod="openshift-monitoring/node-exporter-7vvrs" Jul 21 01:38:01.461190 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:01.461011 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6484e153-4568-44ee-ba8d-4d4b13b5defd-node-exporter-textfile\") pod \"node-exporter-7vvrs\" (UID: \"6484e153-4568-44ee-ba8d-4d4b13b5defd\") " pod="openshift-monitoring/node-exporter-7vvrs" Jul 21 01:38:01.461338 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:01.461205 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6484e153-4568-44ee-ba8d-4d4b13b5defd-metrics-client-ca\") pod \"node-exporter-7vvrs\" (UID: \"6484e153-4568-44ee-ba8d-4d4b13b5defd\") " pod="openshift-monitoring/node-exporter-7vvrs" Jul 21 01:38:01.461338 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:01.461254 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6484e153-4568-44ee-ba8d-4d4b13b5defd-node-exporter-accelerators-collector-config\") pod \"node-exporter-7vvrs\" (UID: \"6484e153-4568-44ee-ba8d-4d4b13b5defd\") " pod="openshift-monitoring/node-exporter-7vvrs" Jul 21 01:38:01.462775 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:01.462753 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6484e153-4568-44ee-ba8d-4d4b13b5defd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7vvrs\" (UID: \"6484e153-4568-44ee-ba8d-4d4b13b5defd\") " pod="openshift-monitoring/node-exporter-7vvrs" Jul 21 01:38:01.470755 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:01.470729 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfk6n\" (UniqueName: \"kubernetes.io/projected/6484e153-4568-44ee-ba8d-4d4b13b5defd-kube-api-access-zfk6n\") pod \"node-exporter-7vvrs\" (UID: \"6484e153-4568-44ee-ba8d-4d4b13b5defd\") " pod="openshift-monitoring/node-exporter-7vvrs" Jul 21 01:38:01.522135 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:01.522106 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6b6c89d7f8-7tjjt" Jul 21 01:38:01.763718 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:01.763683 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/016c43a1-1619-4b20-aa34-37f9a61b0016-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-7gzg9\" (UID: \"016c43a1-1619-4b20-aa34-37f9a61b0016\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-7gzg9" Jul 21 01:38:01.766118 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:01.766086 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/016c43a1-1619-4b20-aa34-37f9a61b0016-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-7gzg9\" (UID: \"016c43a1-1619-4b20-aa34-37f9a61b0016\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-7gzg9" Jul 21 01:38:01.965522 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:01.965482 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6484e153-4568-44ee-ba8d-4d4b13b5defd-node-exporter-tls\") pod \"node-exporter-7vvrs\" (UID: \"6484e153-4568-44ee-ba8d-4d4b13b5defd\") " pod="openshift-monitoring/node-exporter-7vvrs" Jul 21 01:38:01.965728 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:38:01.965630 2580 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 21 01:38:01.965728 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:38:01.965708 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6484e153-4568-44ee-ba8d-4d4b13b5defd-node-exporter-tls podName:6484e153-4568-44ee-ba8d-4d4b13b5defd nodeName:}" failed. No retries permitted until 2026-07-21 01:38:02.965687052 +0000 UTC m=+106.552853012 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/6484e153-4568-44ee-ba8d-4d4b13b5defd-node-exporter-tls") pod "node-exporter-7vvrs" (UID: "6484e153-4568-44ee-ba8d-4d4b13b5defd") : secret "node-exporter-tls" not found Jul 21 01:38:02.022923 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:02.022845 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-7gzg9" Jul 21 01:38:02.184367 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:02.184330 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-7gzg9"] Jul 21 01:38:02.190242 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:38:02.190212 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod016c43a1_1619_4b20_aa34_37f9a61b0016.slice/crio-d13e96d851702cdeb83724876631975a60e2f21cd156e17050013439c5ddca2f WatchSource:0}: Error finding container d13e96d851702cdeb83724876631975a60e2f21cd156e17050013439c5ddca2f: Status 404 returned error can't find the container with id d13e96d851702cdeb83724876631975a60e2f21cd156e17050013439c5ddca2f Jul 21 01:38:02.523684 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:02.523644 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-7gzg9" event={"ID":"016c43a1-1619-4b20-aa34-37f9a61b0016","Type":"ContainerStarted","Data":"d13e96d851702cdeb83724876631975a60e2f21cd156e17050013439c5ddca2f"} Jul 21 01:38:02.976839 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:02.976751 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6484e153-4568-44ee-ba8d-4d4b13b5defd-node-exporter-tls\") pod \"node-exporter-7vvrs\" (UID: \"6484e153-4568-44ee-ba8d-4d4b13b5defd\") " pod="openshift-monitoring/node-exporter-7vvrs" Jul 21 01:38:02.979412 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:02.979384 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6484e153-4568-44ee-ba8d-4d4b13b5defd-node-exporter-tls\") pod \"node-exporter-7vvrs\" (UID: \"6484e153-4568-44ee-ba8d-4d4b13b5defd\") " pod="openshift-monitoring/node-exporter-7vvrs" Jul 21 01:38:03.060348 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:03.060305 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7vvrs" Jul 21 01:38:03.083559 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:38:03.083512 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6484e153_4568_44ee_ba8d_4d4b13b5defd.slice/crio-65af48cf6d056df833d778d34aa255daccfa840a70f5b812f578635794429bb7 WatchSource:0}: Error finding container 65af48cf6d056df833d778d34aa255daccfa840a70f5b812f578635794429bb7: Status 404 returned error can't find the container with id 65af48cf6d056df833d778d34aa255daccfa840a70f5b812f578635794429bb7 Jul 21 01:38:03.182002 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:03.181968 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7655d55f5f-rhfvc"] Jul 21 01:38:03.206146 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:03.206115 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7655d55f5f-rhfvc"] Jul 21 01:38:03.206324 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:03.206228 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7655d55f5f-rhfvc" Jul 21 01:38:03.209066 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:03.209030 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 21 01:38:03.209214 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:03.209101 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-982g7\"" Jul 21 01:38:03.209214 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:03.209200 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 21 01:38:03.209367 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:03.209319 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 21 01:38:03.209661 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:03.209551 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 21 01:38:03.209661 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:03.209567 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-2c02uja58dsqf\"" Jul 21 01:38:03.209661 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:03.209589 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 21 01:38:03.280298 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:03.280266 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0b59c9f0-3074-40cb-9188-14225a950785-metrics-client-ca\") pod \"thanos-querier-7655d55f5f-rhfvc\" (UID: \"0b59c9f0-3074-40cb-9188-14225a950785\") " pod="openshift-monitoring/thanos-querier-7655d55f5f-rhfvc" Jul 21 01:38:03.280507 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:03.280305 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/0b59c9f0-3074-40cb-9188-14225a950785-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7655d55f5f-rhfvc\" (UID: \"0b59c9f0-3074-40cb-9188-14225a950785\") " pod="openshift-monitoring/thanos-querier-7655d55f5f-rhfvc" Jul 21 01:38:03.280507 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:03.280381 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z96tt\" (UniqueName: \"kubernetes.io/projected/0b59c9f0-3074-40cb-9188-14225a950785-kube-api-access-z96tt\") pod \"thanos-querier-7655d55f5f-rhfvc\" (UID: \"0b59c9f0-3074-40cb-9188-14225a950785\") " pod="openshift-monitoring/thanos-querier-7655d55f5f-rhfvc" Jul 21 01:38:03.280507 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:03.280419 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0b59c9f0-3074-40cb-9188-14225a950785-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7655d55f5f-rhfvc\" (UID: \"0b59c9f0-3074-40cb-9188-14225a950785\") " pod="openshift-monitoring/thanos-querier-7655d55f5f-rhfvc" Jul 21 01:38:03.280664 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:03.280512 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/0b59c9f0-3074-40cb-9188-14225a950785-secret-thanos-querier-tls\") pod \"thanos-querier-7655d55f5f-rhfvc\" (UID: \"0b59c9f0-3074-40cb-9188-14225a950785\") " pod="openshift-monitoring/thanos-querier-7655d55f5f-rhfvc" Jul 21 01:38:03.280664 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:03.280555 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0b59c9f0-3074-40cb-9188-14225a950785-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7655d55f5f-rhfvc\" (UID: \"0b59c9f0-3074-40cb-9188-14225a950785\") " pod="openshift-monitoring/thanos-querier-7655d55f5f-rhfvc" Jul 21 01:38:03.280664 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:03.280586 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/0b59c9f0-3074-40cb-9188-14225a950785-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7655d55f5f-rhfvc\" (UID: \"0b59c9f0-3074-40cb-9188-14225a950785\") " pod="openshift-monitoring/thanos-querier-7655d55f5f-rhfvc" Jul 21 01:38:03.280664 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:03.280623 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0b59c9f0-3074-40cb-9188-14225a950785-secret-grpc-tls\") pod \"thanos-querier-7655d55f5f-rhfvc\" (UID: \"0b59c9f0-3074-40cb-9188-14225a950785\") " pod="openshift-monitoring/thanos-querier-7655d55f5f-rhfvc" Jul 21 01:38:03.381409 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:03.381372 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z96tt\" (UniqueName: \"kubernetes.io/projected/0b59c9f0-3074-40cb-9188-14225a950785-kube-api-access-z96tt\") pod \"thanos-querier-7655d55f5f-rhfvc\" (UID: \"0b59c9f0-3074-40cb-9188-14225a950785\") " pod="openshift-monitoring/thanos-querier-7655d55f5f-rhfvc" Jul 21 01:38:03.381606 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:03.381419 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0b59c9f0-3074-40cb-9188-14225a950785-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7655d55f5f-rhfvc\" (UID: \"0b59c9f0-3074-40cb-9188-14225a950785\") " pod="openshift-monitoring/thanos-querier-7655d55f5f-rhfvc" Jul 21 01:38:03.381606 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:03.381502 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/0b59c9f0-3074-40cb-9188-14225a950785-secret-thanos-querier-tls\") pod \"thanos-querier-7655d55f5f-rhfvc\" (UID: \"0b59c9f0-3074-40cb-9188-14225a950785\") " pod="openshift-monitoring/thanos-querier-7655d55f5f-rhfvc" Jul 21 01:38:03.381606 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:03.381527 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0b59c9f0-3074-40cb-9188-14225a950785-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7655d55f5f-rhfvc\" (UID: \"0b59c9f0-3074-40cb-9188-14225a950785\") " pod="openshift-monitoring/thanos-querier-7655d55f5f-rhfvc" Jul 21 01:38:03.381606 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:03.381553 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/0b59c9f0-3074-40cb-9188-14225a950785-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7655d55f5f-rhfvc\" (UID: \"0b59c9f0-3074-40cb-9188-14225a950785\") " pod="openshift-monitoring/thanos-querier-7655d55f5f-rhfvc" Jul 21 01:38:03.381606 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:03.381581 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0b59c9f0-3074-40cb-9188-14225a950785-secret-grpc-tls\") pod \"thanos-querier-7655d55f5f-rhfvc\" (UID: \"0b59c9f0-3074-40cb-9188-14225a950785\") " pod="openshift-monitoring/thanos-querier-7655d55f5f-rhfvc" Jul 21 01:38:03.381879 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:03.381622 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0b59c9f0-3074-40cb-9188-14225a950785-metrics-client-ca\") pod \"thanos-querier-7655d55f5f-rhfvc\" (UID: \"0b59c9f0-3074-40cb-9188-14225a950785\") " pod="openshift-monitoring/thanos-querier-7655d55f5f-rhfvc" Jul 21 01:38:03.381879 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:03.381655 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/0b59c9f0-3074-40cb-9188-14225a950785-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7655d55f5f-rhfvc\" (UID: \"0b59c9f0-3074-40cb-9188-14225a950785\") " pod="openshift-monitoring/thanos-querier-7655d55f5f-rhfvc" Jul 21 01:38:03.382413 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:03.382385 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0b59c9f0-3074-40cb-9188-14225a950785-metrics-client-ca\") pod \"thanos-querier-7655d55f5f-rhfvc\" (UID: \"0b59c9f0-3074-40cb-9188-14225a950785\") " pod="openshift-monitoring/thanos-querier-7655d55f5f-rhfvc" Jul 21 01:38:03.385136 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:03.385114 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/0b59c9f0-3074-40cb-9188-14225a950785-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7655d55f5f-rhfvc\" (UID: \"0b59c9f0-3074-40cb-9188-14225a950785\") " pod="openshift-monitoring/thanos-querier-7655d55f5f-rhfvc" Jul 21 01:38:03.385352 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:03.385328 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/0b59c9f0-3074-40cb-9188-14225a950785-secret-thanos-querier-tls\") pod \"thanos-querier-7655d55f5f-rhfvc\" (UID: \"0b59c9f0-3074-40cb-9188-14225a950785\") " pod="openshift-monitoring/thanos-querier-7655d55f5f-rhfvc" Jul 21 01:38:03.385352 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:03.385341 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0b59c9f0-3074-40cb-9188-14225a950785-secret-grpc-tls\") pod \"thanos-querier-7655d55f5f-rhfvc\" (UID: \"0b59c9f0-3074-40cb-9188-14225a950785\") " pod="openshift-monitoring/thanos-querier-7655d55f5f-rhfvc" Jul 21 01:38:03.385482 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:03.385377 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0b59c9f0-3074-40cb-9188-14225a950785-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7655d55f5f-rhfvc\" (UID: \"0b59c9f0-3074-40cb-9188-14225a950785\") " pod="openshift-monitoring/thanos-querier-7655d55f5f-rhfvc" Jul 21 01:38:03.385482 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:03.385384 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/0b59c9f0-3074-40cb-9188-14225a950785-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7655d55f5f-rhfvc\" (UID: \"0b59c9f0-3074-40cb-9188-14225a950785\") " pod="openshift-monitoring/thanos-querier-7655d55f5f-rhfvc" Jul 21 01:38:03.385626 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:03.385600 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0b59c9f0-3074-40cb-9188-14225a950785-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7655d55f5f-rhfvc\" (UID: \"0b59c9f0-3074-40cb-9188-14225a950785\") " pod="openshift-monitoring/thanos-querier-7655d55f5f-rhfvc" Jul 21 01:38:03.392856 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:03.392836 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z96tt\" (UniqueName: \"kubernetes.io/projected/0b59c9f0-3074-40cb-9188-14225a950785-kube-api-access-z96tt\") pod \"thanos-querier-7655d55f5f-rhfvc\" (UID: \"0b59c9f0-3074-40cb-9188-14225a950785\") " pod="openshift-monitoring/thanos-querier-7655d55f5f-rhfvc" Jul 21 01:38:03.517781 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:03.517736 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7655d55f5f-rhfvc" Jul 21 01:38:03.529496 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:03.529442 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7vvrs" event={"ID":"6484e153-4568-44ee-ba8d-4d4b13b5defd","Type":"ContainerStarted","Data":"65af48cf6d056df833d778d34aa255daccfa840a70f5b812f578635794429bb7"} Jul 21 01:38:04.722782 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:04.722757 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7655d55f5f-rhfvc"] Jul 21 01:38:04.723935 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:38:04.723907 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b59c9f0_3074_40cb_9188_14225a950785.slice/crio-9d8f80986d4e816db8d4fb78a91aec5f8c2fa1204b2f5a076ffaa455fce27223 WatchSource:0}: Error finding container 9d8f80986d4e816db8d4fb78a91aec5f8c2fa1204b2f5a076ffaa455fce27223: Status 404 returned error can't find the container with id 9d8f80986d4e816db8d4fb78a91aec5f8c2fa1204b2f5a076ffaa455fce27223 Jul 21 01:38:05.540526 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:05.540464 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-7gzg9" event={"ID":"016c43a1-1619-4b20-aa34-37f9a61b0016","Type":"ContainerStarted","Data":"b16c43532ee0eae6c1dff5ed91f5ca0395795a83c03920178ecacf4cfb91d2a7"} Jul 21 01:38:05.540752 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:05.540544 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-7gzg9" event={"ID":"016c43a1-1619-4b20-aa34-37f9a61b0016","Type":"ContainerStarted","Data":"caa551fd21cad66cc2866ca48e5c05e2f52852c3aadf89afab76b73c6ae7bd05"} Jul 21 01:38:05.540752 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:05.540573 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-7gzg9" event={"ID":"016c43a1-1619-4b20-aa34-37f9a61b0016","Type":"ContainerStarted","Data":"9ed0eb388a6e1463cbe1d5e80a7eb8c60ffd9b791af9d788f89677eb4f5f146f"} Jul 21 01:38:05.542591 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:05.542561 2580 generic.go:356] "Generic (PLEG): container finished" podID="6484e153-4568-44ee-ba8d-4d4b13b5defd" containerID="75a92099711023686ace99f52dc877959cbf9be093d5635e400b367ee6992fa2" exitCode=0 Jul 21 01:38:05.542709 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:05.542648 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7vvrs" event={"ID":"6484e153-4568-44ee-ba8d-4d4b13b5defd","Type":"ContainerDied","Data":"75a92099711023686ace99f52dc877959cbf9be093d5635e400b367ee6992fa2"} Jul 21 01:38:05.544160 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:05.544137 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7655d55f5f-rhfvc" event={"ID":"0b59c9f0-3074-40cb-9188-14225a950785","Type":"ContainerStarted","Data":"9d8f80986d4e816db8d4fb78a91aec5f8c2fa1204b2f5a076ffaa455fce27223"} Jul 21 01:38:05.566833 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:05.566807 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6b98d6767c-27dq7"] Jul 21 01:38:05.586481 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:05.586432 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6b98d6767c-27dq7" Jul 21 01:38:05.587891 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:05.587863 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6b98d6767c-27dq7"] Jul 21 01:38:05.588292 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:05.588197 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56c5594674-7gzg9" podStartSLOduration=2.188751389 podStartE2EDuration="4.588179588s" totalImagesPullingTime="2.399428199s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 01:38:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:38:05.583811031 +0000 UTC m=+109.170977027" watchObservedRunningTime="2026-07-21 01:38:05.588179588 +0000 UTC m=+109.175345568" Jul 21 01:38:05.589243 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:05.589164 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 21 01:38:05.589243 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:05.589201 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 21 01:38:05.589575 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:05.589519 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 21 01:38:05.589634 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:05.589595 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-be32bf92c9ldn\"" Jul 21 01:38:05.589634 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:05.589603 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 21 01:38:05.589827 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:05.589807 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-dh598\"" Jul 21 01:38:05.701272 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:05.701244 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7244033a-1bcd-4621-9e12-7cead4e87eea-client-ca-bundle\") pod \"metrics-server-6b98d6767c-27dq7\" (UID: \"7244033a-1bcd-4621-9e12-7cead4e87eea\") " pod="openshift-monitoring/metrics-server-6b98d6767c-27dq7" Jul 21 01:38:05.701414 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:05.701295 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm67p\" (UniqueName: \"kubernetes.io/projected/7244033a-1bcd-4621-9e12-7cead4e87eea-kube-api-access-nm67p\") pod \"metrics-server-6b98d6767c-27dq7\" (UID: \"7244033a-1bcd-4621-9e12-7cead4e87eea\") " pod="openshift-monitoring/metrics-server-6b98d6767c-27dq7" Jul 21 01:38:05.701414 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:05.701349 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7244033a-1bcd-4621-9e12-7cead4e87eea-secret-metrics-server-tls\") pod \"metrics-server-6b98d6767c-27dq7\" (UID: \"7244033a-1bcd-4621-9e12-7cead4e87eea\") " pod="openshift-monitoring/metrics-server-6b98d6767c-27dq7" Jul 21 01:38:05.701414 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:05.701368 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7244033a-1bcd-4621-9e12-7cead4e87eea-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6b98d6767c-27dq7\" (UID: \"7244033a-1bcd-4621-9e12-7cead4e87eea\") " pod="openshift-monitoring/metrics-server-6b98d6767c-27dq7" Jul 21 01:38:05.701607 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:05.701427 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7244033a-1bcd-4621-9e12-7cead4e87eea-metrics-server-audit-profiles\") pod \"metrics-server-6b98d6767c-27dq7\" (UID: \"7244033a-1bcd-4621-9e12-7cead4e87eea\") " pod="openshift-monitoring/metrics-server-6b98d6767c-27dq7" Jul 21 01:38:05.701607 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:05.701585 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7244033a-1bcd-4621-9e12-7cead4e87eea-audit-log\") pod \"metrics-server-6b98d6767c-27dq7\" (UID: \"7244033a-1bcd-4621-9e12-7cead4e87eea\") " pod="openshift-monitoring/metrics-server-6b98d6767c-27dq7" Jul 21 01:38:05.701811 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:05.701781 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7244033a-1bcd-4621-9e12-7cead4e87eea-secret-metrics-server-client-certs\") pod \"metrics-server-6b98d6767c-27dq7\" (UID: \"7244033a-1bcd-4621-9e12-7cead4e87eea\") " pod="openshift-monitoring/metrics-server-6b98d6767c-27dq7" Jul 21 01:38:05.802919 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:05.802836 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7244033a-1bcd-4621-9e12-7cead4e87eea-secret-metrics-server-tls\") pod \"metrics-server-6b98d6767c-27dq7\" (UID: \"7244033a-1bcd-4621-9e12-7cead4e87eea\") " pod="openshift-monitoring/metrics-server-6b98d6767c-27dq7" Jul 21 01:38:05.802919 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:05.802880 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7244033a-1bcd-4621-9e12-7cead4e87eea-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6b98d6767c-27dq7\" (UID: \"7244033a-1bcd-4621-9e12-7cead4e87eea\") " pod="openshift-monitoring/metrics-server-6b98d6767c-27dq7" Jul 21 01:38:05.802919 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:05.802905 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7244033a-1bcd-4621-9e12-7cead4e87eea-metrics-server-audit-profiles\") pod \"metrics-server-6b98d6767c-27dq7\" (UID: \"7244033a-1bcd-4621-9e12-7cead4e87eea\") " pod="openshift-monitoring/metrics-server-6b98d6767c-27dq7" Jul 21 01:38:05.803436 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:05.802935 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7244033a-1bcd-4621-9e12-7cead4e87eea-audit-log\") pod \"metrics-server-6b98d6767c-27dq7\" (UID: \"7244033a-1bcd-4621-9e12-7cead4e87eea\") " pod="openshift-monitoring/metrics-server-6b98d6767c-27dq7" Jul 21 01:38:05.803436 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:05.802990 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7244033a-1bcd-4621-9e12-7cead4e87eea-secret-metrics-server-client-certs\") pod \"metrics-server-6b98d6767c-27dq7\" (UID: \"7244033a-1bcd-4621-9e12-7cead4e87eea\") " pod="openshift-monitoring/metrics-server-6b98d6767c-27dq7" Jul 21 01:38:05.803436 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:05.803034 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7244033a-1bcd-4621-9e12-7cead4e87eea-client-ca-bundle\") pod \"metrics-server-6b98d6767c-27dq7\" (UID: \"7244033a-1bcd-4621-9e12-7cead4e87eea\") " pod="openshift-monitoring/metrics-server-6b98d6767c-27dq7" Jul 21 01:38:05.803436 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:05.803082 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm67p\" (UniqueName: \"kubernetes.io/projected/7244033a-1bcd-4621-9e12-7cead4e87eea-kube-api-access-nm67p\") pod \"metrics-server-6b98d6767c-27dq7\" (UID: \"7244033a-1bcd-4621-9e12-7cead4e87eea\") " pod="openshift-monitoring/metrics-server-6b98d6767c-27dq7" Jul 21 01:38:05.803662 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:05.803441 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7244033a-1bcd-4621-9e12-7cead4e87eea-audit-log\") pod \"metrics-server-6b98d6767c-27dq7\" (UID: \"7244033a-1bcd-4621-9e12-7cead4e87eea\") " pod="openshift-monitoring/metrics-server-6b98d6767c-27dq7" Jul 21 01:38:05.803921 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:05.803842 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7244033a-1bcd-4621-9e12-7cead4e87eea-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6b98d6767c-27dq7\" (UID: \"7244033a-1bcd-4621-9e12-7cead4e87eea\") " pod="openshift-monitoring/metrics-server-6b98d6767c-27dq7" Jul 21 01:38:05.804607 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:05.804273 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7244033a-1bcd-4621-9e12-7cead4e87eea-metrics-server-audit-profiles\") pod \"metrics-server-6b98d6767c-27dq7\" (UID: \"7244033a-1bcd-4621-9e12-7cead4e87eea\") " pod="openshift-monitoring/metrics-server-6b98d6767c-27dq7" Jul 21 01:38:05.806690 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:05.806663 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7244033a-1bcd-4621-9e12-7cead4e87eea-secret-metrics-server-client-certs\") pod \"metrics-server-6b98d6767c-27dq7\" (UID: \"7244033a-1bcd-4621-9e12-7cead4e87eea\") " pod="openshift-monitoring/metrics-server-6b98d6767c-27dq7" Jul 21 01:38:05.806794 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:05.806709 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7244033a-1bcd-4621-9e12-7cead4e87eea-secret-metrics-server-tls\") pod \"metrics-server-6b98d6767c-27dq7\" (UID: \"7244033a-1bcd-4621-9e12-7cead4e87eea\") " pod="openshift-monitoring/metrics-server-6b98d6767c-27dq7" Jul 21 01:38:05.806794 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:05.806729 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7244033a-1bcd-4621-9e12-7cead4e87eea-client-ca-bundle\") pod \"metrics-server-6b98d6767c-27dq7\" (UID: \"7244033a-1bcd-4621-9e12-7cead4e87eea\") " pod="openshift-monitoring/metrics-server-6b98d6767c-27dq7" Jul 21 01:38:05.813878 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:05.813856 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm67p\" (UniqueName: \"kubernetes.io/projected/7244033a-1bcd-4621-9e12-7cead4e87eea-kube-api-access-nm67p\") pod \"metrics-server-6b98d6767c-27dq7\" (UID: \"7244033a-1bcd-4621-9e12-7cead4e87eea\") " pod="openshift-monitoring/metrics-server-6b98d6767c-27dq7" Jul 21 01:38:05.923415 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:05.923382 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6b98d6767c-27dq7" Jul 21 01:38:05.972269 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:05.972224 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-68gpx"] Jul 21 01:38:06.007170 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:06.007038 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-68gpx"] Jul 21 01:38:06.007170 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:06.007144 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-68gpx" Jul 21 01:38:06.012624 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:06.011307 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 21 01:38:06.012624 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:06.011580 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-8gn4z\"" Jul 21 01:38:06.096823 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:06.096748 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6b98d6767c-27dq7"] Jul 21 01:38:06.099634 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:38:06.099606 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7244033a_1bcd_4621_9e12_7cead4e87eea.slice/crio-c7658065023fd315a21b5ed81c21e7301a73bdb6372a50cd2b730cab3dd413da WatchSource:0}: Error finding container c7658065023fd315a21b5ed81c21e7301a73bdb6372a50cd2b730cab3dd413da: Status 404 returned error can't find the container with id c7658065023fd315a21b5ed81c21e7301a73bdb6372a50cd2b730cab3dd413da Jul 21 01:38:06.105663 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:06.105641 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zstq\" (UniqueName: \"kubernetes.io/projected/d04afbe3-2081-44fc-985b-2ca888bfb6f3-kube-api-access-6zstq\") pod \"monitoring-plugin-7b7b7bc8bf-68gpx\" (UID: \"d04afbe3-2081-44fc-985b-2ca888bfb6f3\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-68gpx" Jul 21 01:38:06.105767 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:06.105706 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d04afbe3-2081-44fc-985b-2ca888bfb6f3-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-68gpx\" (UID: \"d04afbe3-2081-44fc-985b-2ca888bfb6f3\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-68gpx" Jul 21 01:38:06.206539 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:06.206421 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zstq\" (UniqueName: \"kubernetes.io/projected/d04afbe3-2081-44fc-985b-2ca888bfb6f3-kube-api-access-6zstq\") pod \"monitoring-plugin-7b7b7bc8bf-68gpx\" (UID: \"d04afbe3-2081-44fc-985b-2ca888bfb6f3\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-68gpx" Jul 21 01:38:06.206705 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:06.206561 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d04afbe3-2081-44fc-985b-2ca888bfb6f3-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-68gpx\" (UID: \"d04afbe3-2081-44fc-985b-2ca888bfb6f3\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-68gpx" Jul 21 01:38:06.209184 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:06.209152 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d04afbe3-2081-44fc-985b-2ca888bfb6f3-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-68gpx\" (UID: \"d04afbe3-2081-44fc-985b-2ca888bfb6f3\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-68gpx" Jul 21 01:38:06.214491 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:06.214443 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zstq\" (UniqueName: \"kubernetes.io/projected/d04afbe3-2081-44fc-985b-2ca888bfb6f3-kube-api-access-6zstq\") pod \"monitoring-plugin-7b7b7bc8bf-68gpx\" (UID: \"d04afbe3-2081-44fc-985b-2ca888bfb6f3\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-68gpx" Jul 21 01:38:06.325396 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:06.325356 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-68gpx" Jul 21 01:38:06.347565 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:06.347484 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-d976688db-wk24w"] Jul 21 01:38:06.363106 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:06.363080 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-d976688db-wk24w" Jul 21 01:38:06.364077 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:06.364045 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-d976688db-wk24w"] Jul 21 01:38:06.366280 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:06.365966 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 21 01:38:06.366280 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:06.366029 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 21 01:38:06.366280 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:06.366051 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 21 01:38:06.366280 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:06.366104 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 21 01:38:06.366280 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:06.366050 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 21 01:38:06.366280 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:06.365971 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-jb4kt\"" Jul 21 01:38:06.373115 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:06.372903 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 21 01:38:06.408544 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:06.408503 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7312b0ab-bfff-4bce-976e-ba6e6167ba3e-serving-certs-ca-bundle\") pod \"telemeter-client-d976688db-wk24w\" (UID: \"7312b0ab-bfff-4bce-976e-ba6e6167ba3e\") " pod="openshift-monitoring/telemeter-client-d976688db-wk24w" Jul 21 01:38:06.408716 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:06.408574 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/7312b0ab-bfff-4bce-976e-ba6e6167ba3e-secret-telemeter-client\") pod \"telemeter-client-d976688db-wk24w\" (UID: \"7312b0ab-bfff-4bce-976e-ba6e6167ba3e\") " pod="openshift-monitoring/telemeter-client-d976688db-wk24w" Jul 21 01:38:06.408716 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:06.408657 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/7312b0ab-bfff-4bce-976e-ba6e6167ba3e-federate-client-tls\") pod \"telemeter-client-d976688db-wk24w\" (UID: \"7312b0ab-bfff-4bce-976e-ba6e6167ba3e\") " pod="openshift-monitoring/telemeter-client-d976688db-wk24w" Jul 21 01:38:06.408716 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:06.408703 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7312b0ab-bfff-4bce-976e-ba6e6167ba3e-telemeter-trusted-ca-bundle\") pod \"telemeter-client-d976688db-wk24w\" (UID: \"7312b0ab-bfff-4bce-976e-ba6e6167ba3e\") " pod="openshift-monitoring/telemeter-client-d976688db-wk24w" Jul 21 01:38:06.408877 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:06.408731 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qr2vv\" (UniqueName: \"kubernetes.io/projected/7312b0ab-bfff-4bce-976e-ba6e6167ba3e-kube-api-access-qr2vv\") pod \"telemeter-client-d976688db-wk24w\" (UID: \"7312b0ab-bfff-4bce-976e-ba6e6167ba3e\") " pod="openshift-monitoring/telemeter-client-d976688db-wk24w" Jul 21 01:38:06.408877 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:06.408806 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7312b0ab-bfff-4bce-976e-ba6e6167ba3e-metrics-client-ca\") pod \"telemeter-client-d976688db-wk24w\" (UID: \"7312b0ab-bfff-4bce-976e-ba6e6167ba3e\") " pod="openshift-monitoring/telemeter-client-d976688db-wk24w" Jul 21 01:38:06.408877 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:06.408865 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/7312b0ab-bfff-4bce-976e-ba6e6167ba3e-telemeter-client-tls\") pod \"telemeter-client-d976688db-wk24w\" (UID: \"7312b0ab-bfff-4bce-976e-ba6e6167ba3e\") " pod="openshift-monitoring/telemeter-client-d976688db-wk24w" Jul 21 01:38:06.409019 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:06.408923 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7312b0ab-bfff-4bce-976e-ba6e6167ba3e-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-d976688db-wk24w\" (UID: \"7312b0ab-bfff-4bce-976e-ba6e6167ba3e\") " pod="openshift-monitoring/telemeter-client-d976688db-wk24w" Jul 21 01:38:06.509842 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:06.509739 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/7312b0ab-bfff-4bce-976e-ba6e6167ba3e-federate-client-tls\") pod \"telemeter-client-d976688db-wk24w\" (UID: \"7312b0ab-bfff-4bce-976e-ba6e6167ba3e\") " pod="openshift-monitoring/telemeter-client-d976688db-wk24w" Jul 21 01:38:06.509842 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:06.509812 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7312b0ab-bfff-4bce-976e-ba6e6167ba3e-telemeter-trusted-ca-bundle\") pod \"telemeter-client-d976688db-wk24w\" (UID: \"7312b0ab-bfff-4bce-976e-ba6e6167ba3e\") " pod="openshift-monitoring/telemeter-client-d976688db-wk24w" Jul 21 01:38:06.509842 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:06.509839 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qr2vv\" (UniqueName: \"kubernetes.io/projected/7312b0ab-bfff-4bce-976e-ba6e6167ba3e-kube-api-access-qr2vv\") pod \"telemeter-client-d976688db-wk24w\" (UID: \"7312b0ab-bfff-4bce-976e-ba6e6167ba3e\") " pod="openshift-monitoring/telemeter-client-d976688db-wk24w" Jul 21 01:38:06.510118 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:06.509999 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7312b0ab-bfff-4bce-976e-ba6e6167ba3e-metrics-client-ca\") pod \"telemeter-client-d976688db-wk24w\" (UID: \"7312b0ab-bfff-4bce-976e-ba6e6167ba3e\") " pod="openshift-monitoring/telemeter-client-d976688db-wk24w" Jul 21 01:38:06.510118 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:06.510056 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/7312b0ab-bfff-4bce-976e-ba6e6167ba3e-telemeter-client-tls\") pod \"telemeter-client-d976688db-wk24w\" (UID: \"7312b0ab-bfff-4bce-976e-ba6e6167ba3e\") " pod="openshift-monitoring/telemeter-client-d976688db-wk24w" Jul 21 01:38:06.510118 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:06.510085 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7312b0ab-bfff-4bce-976e-ba6e6167ba3e-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-d976688db-wk24w\" (UID: \"7312b0ab-bfff-4bce-976e-ba6e6167ba3e\") " pod="openshift-monitoring/telemeter-client-d976688db-wk24w" Jul 21 01:38:06.510299 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:06.510149 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7312b0ab-bfff-4bce-976e-ba6e6167ba3e-serving-certs-ca-bundle\") pod \"telemeter-client-d976688db-wk24w\" (UID: \"7312b0ab-bfff-4bce-976e-ba6e6167ba3e\") " pod="openshift-monitoring/telemeter-client-d976688db-wk24w" Jul 21 01:38:06.510299 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:06.510187 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/7312b0ab-bfff-4bce-976e-ba6e6167ba3e-secret-telemeter-client\") pod \"telemeter-client-d976688db-wk24w\" (UID: \"7312b0ab-bfff-4bce-976e-ba6e6167ba3e\") " pod="openshift-monitoring/telemeter-client-d976688db-wk24w" Jul 21 01:38:06.510826 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:06.510764 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7312b0ab-bfff-4bce-976e-ba6e6167ba3e-metrics-client-ca\") pod \"telemeter-client-d976688db-wk24w\" (UID: \"7312b0ab-bfff-4bce-976e-ba6e6167ba3e\") " pod="openshift-monitoring/telemeter-client-d976688db-wk24w" Jul 21 01:38:06.510826 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:06.510802 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7312b0ab-bfff-4bce-976e-ba6e6167ba3e-telemeter-trusted-ca-bundle\") pod \"telemeter-client-d976688db-wk24w\" (UID: \"7312b0ab-bfff-4bce-976e-ba6e6167ba3e\") " pod="openshift-monitoring/telemeter-client-d976688db-wk24w" Jul 21 01:38:06.511032 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:06.510832 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7312b0ab-bfff-4bce-976e-ba6e6167ba3e-serving-certs-ca-bundle\") pod \"telemeter-client-d976688db-wk24w\" (UID: \"7312b0ab-bfff-4bce-976e-ba6e6167ba3e\") " pod="openshift-monitoring/telemeter-client-d976688db-wk24w" Jul 21 01:38:06.512890 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:06.512826 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/7312b0ab-bfff-4bce-976e-ba6e6167ba3e-federate-client-tls\") pod \"telemeter-client-d976688db-wk24w\" (UID: \"7312b0ab-bfff-4bce-976e-ba6e6167ba3e\") " pod="openshift-monitoring/telemeter-client-d976688db-wk24w" Jul 21 01:38:06.513145 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:06.513114 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/7312b0ab-bfff-4bce-976e-ba6e6167ba3e-secret-telemeter-client\") pod \"telemeter-client-d976688db-wk24w\" (UID: \"7312b0ab-bfff-4bce-976e-ba6e6167ba3e\") " pod="openshift-monitoring/telemeter-client-d976688db-wk24w" Jul 21 01:38:06.513325 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:06.513281 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/7312b0ab-bfff-4bce-976e-ba6e6167ba3e-telemeter-client-tls\") pod \"telemeter-client-d976688db-wk24w\" (UID: \"7312b0ab-bfff-4bce-976e-ba6e6167ba3e\") " pod="openshift-monitoring/telemeter-client-d976688db-wk24w" Jul 21 01:38:06.513535 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:06.513517 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7312b0ab-bfff-4bce-976e-ba6e6167ba3e-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-d976688db-wk24w\" (UID: \"7312b0ab-bfff-4bce-976e-ba6e6167ba3e\") " pod="openshift-monitoring/telemeter-client-d976688db-wk24w" Jul 21 01:38:06.520326 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:06.520303 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qr2vv\" (UniqueName: \"kubernetes.io/projected/7312b0ab-bfff-4bce-976e-ba6e6167ba3e-kube-api-access-qr2vv\") pod \"telemeter-client-d976688db-wk24w\" (UID: \"7312b0ab-bfff-4bce-976e-ba6e6167ba3e\") " pod="openshift-monitoring/telemeter-client-d976688db-wk24w" Jul 21 01:38:06.552151 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:06.552108 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6b98d6767c-27dq7" event={"ID":"7244033a-1bcd-4621-9e12-7cead4e87eea","Type":"ContainerStarted","Data":"c7658065023fd315a21b5ed81c21e7301a73bdb6372a50cd2b730cab3dd413da"} Jul 21 01:38:06.554769 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:06.554734 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7vvrs" event={"ID":"6484e153-4568-44ee-ba8d-4d4b13b5defd","Type":"ContainerStarted","Data":"e7e16cffabbcd038020603e85eba3a639ca18a30fac3467b0b1d22b4d8871bc9"} Jul 21 01:38:06.554922 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:06.554798 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7vvrs" event={"ID":"6484e153-4568-44ee-ba8d-4d4b13b5defd","Type":"ContainerStarted","Data":"e5bae7024576da622972d6e66467b506b6bbde8eaf9ab1ef509464061d528aad"} Jul 21 01:38:06.573695 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:06.573567 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-7vvrs" podStartSLOduration=3.949732917 podStartE2EDuration="5.573546602s" totalImagesPullingTime="1.508115956s" totalInitContainerRuntime="115.697729ms" isStatefulPod=true podCreationTimestamp="2026-07-21 01:38:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:38:06.572881759 +0000 UTC m=+110.160047738" watchObservedRunningTime="2026-07-21 01:38:06.573546602 +0000 UTC m=+110.160712579" Jul 21 01:38:06.679812 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:06.679723 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-d976688db-wk24w" Jul 21 01:38:06.878168 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:06.878134 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-68gpx"] Jul 21 01:38:06.900770 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:06.900746 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-d976688db-wk24w"] Jul 21 01:38:06.949014 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:38:06.948930 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd04afbe3_2081_44fc_985b_2ca888bfb6f3.slice/crio-eb8e17ab4c32171ccb97accdee7820b1b112d28d6fae93bb68eb46ae7eb878d5 WatchSource:0}: Error finding container eb8e17ab4c32171ccb97accdee7820b1b112d28d6fae93bb68eb46ae7eb878d5: Status 404 returned error can't find the container with id eb8e17ab4c32171ccb97accdee7820b1b112d28d6fae93bb68eb46ae7eb878d5 Jul 21 01:38:06.949757 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:38:06.949733 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7312b0ab_bfff_4bce_976e_ba6e6167ba3e.slice/crio-deb977d1d9e2e5652a898f954092856ad10c1b6dccf32c00a15baa6a6343fc85 WatchSource:0}: Error finding container deb977d1d9e2e5652a898f954092856ad10c1b6dccf32c00a15baa6a6343fc85: Status 404 returned error can't find the container with id deb977d1d9e2e5652a898f954092856ad10c1b6dccf32c00a15baa6a6343fc85 Jul 21 01:38:07.561693 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:07.561646 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-d976688db-wk24w" event={"ID":"7312b0ab-bfff-4bce-976e-ba6e6167ba3e","Type":"ContainerStarted","Data":"deb977d1d9e2e5652a898f954092856ad10c1b6dccf32c00a15baa6a6343fc85"} Jul 21 01:38:07.563470 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:07.563327 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-68gpx" event={"ID":"d04afbe3-2081-44fc-985b-2ca888bfb6f3","Type":"ContainerStarted","Data":"eb8e17ab4c32171ccb97accdee7820b1b112d28d6fae93bb68eb46ae7eb878d5"} Jul 21 01:38:07.566037 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:07.566009 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7655d55f5f-rhfvc" event={"ID":"0b59c9f0-3074-40cb-9188-14225a950785","Type":"ContainerStarted","Data":"6746a526829dedb4a8389f21c3111b2fbe0e5fe2791daaa47e13400529aed4fa"} Jul 21 01:38:07.566168 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:07.566084 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7655d55f5f-rhfvc" event={"ID":"0b59c9f0-3074-40cb-9188-14225a950785","Type":"ContainerStarted","Data":"d5d466a1c50b98abf3cd88b06001cff24c39f9664ed706e420ace80cc0576d70"} Jul 21 01:38:07.566168 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:07.566127 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7655d55f5f-rhfvc" event={"ID":"0b59c9f0-3074-40cb-9188-14225a950785","Type":"ContainerStarted","Data":"37633931de32afe5d56e7e989cb9b6d9cbe1afcbab2c521612e9ef1407232c9e"} Jul 21 01:38:10.583489 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:10.583425 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7655d55f5f-rhfvc" event={"ID":"0b59c9f0-3074-40cb-9188-14225a950785","Type":"ContainerStarted","Data":"37b640e6b35df3756d80b8d661815118cd355c7a06dbdfd68130a49daca77fe7"} Jul 21 01:38:10.583923 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:10.583539 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7655d55f5f-rhfvc" event={"ID":"0b59c9f0-3074-40cb-9188-14225a950785","Type":"ContainerStarted","Data":"28c11311321feb72d608c984e33dca7825e9f3b1e809e63fa0c983d63993959e"} Jul 21 01:38:10.583923 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:10.583601 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7655d55f5f-rhfvc" Jul 21 01:38:10.583923 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:10.583646 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7655d55f5f-rhfvc" event={"ID":"0b59c9f0-3074-40cb-9188-14225a950785","Type":"ContainerStarted","Data":"76876c13838bcddfdf050fa614641ba8d1fc8feb363687b62f261cc6aab97427"} Jul 21 01:38:10.585075 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:10.585054 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-d976688db-wk24w" event={"ID":"7312b0ab-bfff-4bce-976e-ba6e6167ba3e","Type":"ContainerStarted","Data":"5e52179bbb93fd96e0f05e08bad7ba47f40d958a4476087cf3c310f425b14fce"} Jul 21 01:38:10.586487 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:10.586468 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6b98d6767c-27dq7" event={"ID":"7244033a-1bcd-4621-9e12-7cead4e87eea","Type":"ContainerStarted","Data":"87c5b7dbb063d81dfc36409dc23c0b2ce92201fd474fb1f0431c50a78bedf93b"} Jul 21 01:38:10.587933 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:10.587909 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-68gpx" event={"ID":"d04afbe3-2081-44fc-985b-2ca888bfb6f3","Type":"ContainerStarted","Data":"023a6fe0a66acee0a83b7050314d3c9f4110a7f874b2ea243117817aa0c79e26"} Jul 21 01:38:10.588115 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:10.588094 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-68gpx" Jul 21 01:38:10.593393 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:10.593367 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-68gpx" Jul 21 01:38:10.609876 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:10.609833 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7655d55f5f-rhfvc" podStartSLOduration=2.510444073 podStartE2EDuration="7.609816823s" totalImagesPullingTime="5.09937275s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 01:38:03 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:38:10.608428763 +0000 UTC m=+114.195594793" watchObservedRunningTime="2026-07-21 01:38:10.609816823 +0000 UTC m=+114.196982800" Jul 21 01:38:10.631135 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:10.631080 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6b98d6767c-27dq7" podStartSLOduration=1.821789979 podStartE2EDuration="5.631066038s" totalImagesPullingTime="3.809276059s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 01:38:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:38:10.629617975 +0000 UTC m=+114.216783979" watchObservedRunningTime="2026-07-21 01:38:10.631066038 +0000 UTC m=+114.218231994" Jul 21 01:38:10.645389 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:10.645338 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-68gpx" podStartSLOduration=2.690340718 podStartE2EDuration="5.645320444s" totalImagesPullingTime="2.954979726s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 01:38:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:38:10.643718517 +0000 UTC m=+114.230884513" watchObservedRunningTime="2026-07-21 01:38:10.645320444 +0000 UTC m=+114.232486456" Jul 21 01:38:11.594322 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:11.594235 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-d976688db-wk24w" event={"ID":"7312b0ab-bfff-4bce-976e-ba6e6167ba3e","Type":"ContainerStarted","Data":"05a515945afb5c614a4a288d79216a8f902c9fc0415312e311cae30e47b167fa"} Jul 21 01:38:11.594322 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:11.594319 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-d976688db-wk24w" event={"ID":"7312b0ab-bfff-4bce-976e-ba6e6167ba3e","Type":"ContainerStarted","Data":"dc2291fd6e4dcf391640178d7a2434616bdad39162dac4134ad4972a01dfdc11"} Jul 21 01:38:11.618890 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:11.618834 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-d976688db-wk24w" podStartSLOduration=1.5532667770000002 podStartE2EDuration="5.618813213s" totalImagesPullingTime="4.065546436s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 01:38:06 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:38:11.617729042 +0000 UTC m=+115.204895038" watchObservedRunningTime="2026-07-21 01:38:11.618813213 +0000 UTC m=+115.205979192" Jul 21 01:38:12.594882 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:12.594851 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6fd7fd45f8-bf2k7"] Jul 21 01:38:12.598534 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:12.598511 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fd7fd45f8-bf2k7" Jul 21 01:38:12.610231 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:12.610205 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6fd7fd45f8-bf2k7"] Jul 21 01:38:12.668854 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:12.668808 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1b0a7fe3-f91b-4b27-8e60-2ff95099f475-console-config\") pod \"console-6fd7fd45f8-bf2k7\" (UID: \"1b0a7fe3-f91b-4b27-8e60-2ff95099f475\") " pod="openshift-console/console-6fd7fd45f8-bf2k7" Jul 21 01:38:12.668854 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:12.668856 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b0a7fe3-f91b-4b27-8e60-2ff95099f475-trusted-ca-bundle\") pod \"console-6fd7fd45f8-bf2k7\" (UID: \"1b0a7fe3-f91b-4b27-8e60-2ff95099f475\") " pod="openshift-console/console-6fd7fd45f8-bf2k7" Jul 21 01:38:12.669101 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:12.668966 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1b0a7fe3-f91b-4b27-8e60-2ff95099f475-oauth-serving-cert\") pod \"console-6fd7fd45f8-bf2k7\" (UID: \"1b0a7fe3-f91b-4b27-8e60-2ff95099f475\") " pod="openshift-console/console-6fd7fd45f8-bf2k7" Jul 21 01:38:12.669101 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:12.669011 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1b0a7fe3-f91b-4b27-8e60-2ff95099f475-service-ca\") pod \"console-6fd7fd45f8-bf2k7\" (UID: \"1b0a7fe3-f91b-4b27-8e60-2ff95099f475\") " pod="openshift-console/console-6fd7fd45f8-bf2k7" Jul 21 01:38:12.669176 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:12.669143 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1b0a7fe3-f91b-4b27-8e60-2ff95099f475-console-oauth-config\") pod \"console-6fd7fd45f8-bf2k7\" (UID: \"1b0a7fe3-f91b-4b27-8e60-2ff95099f475\") " pod="openshift-console/console-6fd7fd45f8-bf2k7" Jul 21 01:38:12.669251 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:12.669233 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1b0a7fe3-f91b-4b27-8e60-2ff95099f475-console-serving-cert\") pod \"console-6fd7fd45f8-bf2k7\" (UID: \"1b0a7fe3-f91b-4b27-8e60-2ff95099f475\") " pod="openshift-console/console-6fd7fd45f8-bf2k7" Jul 21 01:38:12.669306 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:12.669284 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l877d\" (UniqueName: \"kubernetes.io/projected/1b0a7fe3-f91b-4b27-8e60-2ff95099f475-kube-api-access-l877d\") pod \"console-6fd7fd45f8-bf2k7\" (UID: \"1b0a7fe3-f91b-4b27-8e60-2ff95099f475\") " pod="openshift-console/console-6fd7fd45f8-bf2k7" Jul 21 01:38:12.770174 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:12.770136 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1b0a7fe3-f91b-4b27-8e60-2ff95099f475-console-serving-cert\") pod \"console-6fd7fd45f8-bf2k7\" (UID: \"1b0a7fe3-f91b-4b27-8e60-2ff95099f475\") " pod="openshift-console/console-6fd7fd45f8-bf2k7" Jul 21 01:38:12.770174 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:12.770173 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l877d\" (UniqueName: \"kubernetes.io/projected/1b0a7fe3-f91b-4b27-8e60-2ff95099f475-kube-api-access-l877d\") pod \"console-6fd7fd45f8-bf2k7\" (UID: \"1b0a7fe3-f91b-4b27-8e60-2ff95099f475\") " pod="openshift-console/console-6fd7fd45f8-bf2k7" Jul 21 01:38:12.770386 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:12.770214 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1b0a7fe3-f91b-4b27-8e60-2ff95099f475-console-config\") pod \"console-6fd7fd45f8-bf2k7\" (UID: \"1b0a7fe3-f91b-4b27-8e60-2ff95099f475\") " pod="openshift-console/console-6fd7fd45f8-bf2k7" Jul 21 01:38:12.770386 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:12.770228 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b0a7fe3-f91b-4b27-8e60-2ff95099f475-trusted-ca-bundle\") pod \"console-6fd7fd45f8-bf2k7\" (UID: \"1b0a7fe3-f91b-4b27-8e60-2ff95099f475\") " pod="openshift-console/console-6fd7fd45f8-bf2k7" Jul 21 01:38:12.770386 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:12.770245 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1b0a7fe3-f91b-4b27-8e60-2ff95099f475-oauth-serving-cert\") pod \"console-6fd7fd45f8-bf2k7\" (UID: \"1b0a7fe3-f91b-4b27-8e60-2ff95099f475\") " pod="openshift-console/console-6fd7fd45f8-bf2k7" Jul 21 01:38:12.770386 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:12.770266 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1b0a7fe3-f91b-4b27-8e60-2ff95099f475-service-ca\") pod \"console-6fd7fd45f8-bf2k7\" (UID: \"1b0a7fe3-f91b-4b27-8e60-2ff95099f475\") " pod="openshift-console/console-6fd7fd45f8-bf2k7" Jul 21 01:38:12.770386 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:12.770315 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1b0a7fe3-f91b-4b27-8e60-2ff95099f475-console-oauth-config\") pod \"console-6fd7fd45f8-bf2k7\" (UID: \"1b0a7fe3-f91b-4b27-8e60-2ff95099f475\") " pod="openshift-console/console-6fd7fd45f8-bf2k7" Jul 21 01:38:12.771043 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:12.771018 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1b0a7fe3-f91b-4b27-8e60-2ff95099f475-service-ca\") pod \"console-6fd7fd45f8-bf2k7\" (UID: \"1b0a7fe3-f91b-4b27-8e60-2ff95099f475\") " pod="openshift-console/console-6fd7fd45f8-bf2k7" Jul 21 01:38:12.771157 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:12.771042 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1b0a7fe3-f91b-4b27-8e60-2ff95099f475-console-config\") pod \"console-6fd7fd45f8-bf2k7\" (UID: \"1b0a7fe3-f91b-4b27-8e60-2ff95099f475\") " pod="openshift-console/console-6fd7fd45f8-bf2k7" Jul 21 01:38:12.771198 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:12.771156 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1b0a7fe3-f91b-4b27-8e60-2ff95099f475-oauth-serving-cert\") pod \"console-6fd7fd45f8-bf2k7\" (UID: \"1b0a7fe3-f91b-4b27-8e60-2ff95099f475\") " pod="openshift-console/console-6fd7fd45f8-bf2k7" Jul 21 01:38:12.771270 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:12.771247 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b0a7fe3-f91b-4b27-8e60-2ff95099f475-trusted-ca-bundle\") pod \"console-6fd7fd45f8-bf2k7\" (UID: \"1b0a7fe3-f91b-4b27-8e60-2ff95099f475\") " pod="openshift-console/console-6fd7fd45f8-bf2k7" Jul 21 01:38:12.772784 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:12.772757 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1b0a7fe3-f91b-4b27-8e60-2ff95099f475-console-oauth-config\") pod \"console-6fd7fd45f8-bf2k7\" (UID: \"1b0a7fe3-f91b-4b27-8e60-2ff95099f475\") " pod="openshift-console/console-6fd7fd45f8-bf2k7" Jul 21 01:38:12.772899 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:12.772838 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1b0a7fe3-f91b-4b27-8e60-2ff95099f475-console-serving-cert\") pod \"console-6fd7fd45f8-bf2k7\" (UID: \"1b0a7fe3-f91b-4b27-8e60-2ff95099f475\") " pod="openshift-console/console-6fd7fd45f8-bf2k7" Jul 21 01:38:12.777595 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:12.777573 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l877d\" (UniqueName: \"kubernetes.io/projected/1b0a7fe3-f91b-4b27-8e60-2ff95099f475-kube-api-access-l877d\") pod \"console-6fd7fd45f8-bf2k7\" (UID: \"1b0a7fe3-f91b-4b27-8e60-2ff95099f475\") " pod="openshift-console/console-6fd7fd45f8-bf2k7" Jul 21 01:38:12.909037 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:12.908931 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fd7fd45f8-bf2k7" Jul 21 01:38:13.052368 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:13.052343 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6fd7fd45f8-bf2k7"] Jul 21 01:38:13.054957 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:38:13.054926 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b0a7fe3_f91b_4b27_8e60_2ff95099f475.slice/crio-2c2af01602eed96b7db214ae328bdc69573760218e069db7879d7decd6324e1e WatchSource:0}: Error finding container 2c2af01602eed96b7db214ae328bdc69573760218e069db7879d7decd6324e1e: Status 404 returned error can't find the container with id 2c2af01602eed96b7db214ae328bdc69573760218e069db7879d7decd6324e1e Jul 21 01:38:13.603947 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:13.603913 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fd7fd45f8-bf2k7" event={"ID":"1b0a7fe3-f91b-4b27-8e60-2ff95099f475","Type":"ContainerStarted","Data":"fe91cb72ae3485bddcf6544319cc0caccd81552da47e55f5e5faf61e8e5e3d75"} Jul 21 01:38:13.604346 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:13.603957 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fd7fd45f8-bf2k7" event={"ID":"1b0a7fe3-f91b-4b27-8e60-2ff95099f475","Type":"ContainerStarted","Data":"2c2af01602eed96b7db214ae328bdc69573760218e069db7879d7decd6324e1e"} Jul 21 01:38:13.622740 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:13.622692 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6fd7fd45f8-bf2k7" podStartSLOduration=1.6226756450000002 podStartE2EDuration="1.622675645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 01:38:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:38:13.620691101 +0000 UTC m=+117.207857078" watchObservedRunningTime="2026-07-21 01:38:13.622675645 +0000 UTC m=+117.209841622" Jul 21 01:38:16.601120 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:16.601093 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7655d55f5f-rhfvc" Jul 21 01:38:19.633927 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:19.633890 2580 generic.go:356] "Generic (PLEG): container finished" podID="6e08e8c3-dd3e-4792-8096-cf150dbcb58b" containerID="ec01f40376f16ad3f84e510f5eabbc39394c2c589eae3feb4f2a337eefe47385" exitCode=0 Jul 21 01:38:19.634407 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:19.633969 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-9h5pm" event={"ID":"6e08e8c3-dd3e-4792-8096-cf150dbcb58b","Type":"ContainerDied","Data":"ec01f40376f16ad3f84e510f5eabbc39394c2c589eae3feb4f2a337eefe47385"} Jul 21 01:38:19.634407 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:19.634390 2580 scope.go:117] "RemoveContainer" containerID="ec01f40376f16ad3f84e510f5eabbc39394c2c589eae3feb4f2a337eefe47385" Jul 21 01:38:20.639352 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:20.639318 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-9h5pm" event={"ID":"6e08e8c3-dd3e-4792-8096-cf150dbcb58b","Type":"ContainerStarted","Data":"203d18d65bba6fcf2afce5df05212eb962a187845c0695f03a9de6179508af8f"} Jul 21 01:38:22.909317 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:22.909277 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6fd7fd45f8-bf2k7" Jul 21 01:38:22.909735 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:22.909344 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6fd7fd45f8-bf2k7" Jul 21 01:38:22.914241 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:22.914217 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6fd7fd45f8-bf2k7" Jul 21 01:38:23.652368 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:23.652332 2580 generic.go:356] "Generic (PLEG): container finished" podID="d67dd362-e3a6-4ad1-9f9f-8f628ffce9a2" containerID="d0f1b677a4dd6cb2c846088f8f818252d3b5d122139ed6f40e3a3cef0e70bf8e" exitCode=0 Jul 21 01:38:23.652580 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:23.652407 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-wnjpr" event={"ID":"d67dd362-e3a6-4ad1-9f9f-8f628ffce9a2","Type":"ContainerDied","Data":"d0f1b677a4dd6cb2c846088f8f818252d3b5d122139ed6f40e3a3cef0e70bf8e"} Jul 21 01:38:23.652814 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:23.652796 2580 scope.go:117] "RemoveContainer" containerID="d0f1b677a4dd6cb2c846088f8f818252d3b5d122139ed6f40e3a3cef0e70bf8e" Jul 21 01:38:23.656703 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:23.656682 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6fd7fd45f8-bf2k7" Jul 21 01:38:23.716575 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:23.716371 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b6c89d7f8-7tjjt"] Jul 21 01:38:24.383291 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:24.383262 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6f7df7d7cb-gcx5w_e6d9baf0-743b-48f5-9374-4443c65ea048/router/0.log" Jul 21 01:38:24.394741 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:24.394714 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bmgfc_1d402931-b366-48df-acda-2e144e7c6c70/serve-healthcheck-canary/0.log" Jul 21 01:38:24.658758 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:24.658674 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-wnjpr" event={"ID":"d67dd362-e3a6-4ad1-9f9f-8f628ffce9a2","Type":"ContainerStarted","Data":"dea7c33c36b51a94f0e0f22f63c60b019446a645fc962c766c536480044f4184"} Jul 21 01:38:25.555870 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:25.555827 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-b47d488dd-h2wcn" podUID="fd13d7ca-7c61-4c40-a86e-a58c1a7d1390" containerName="registry" containerID="cri-o://298e0cb45bdb396913774f6b76b2f4bd244c1e2ff2049dfd86e0212576ef9885" gracePeriod=30 Jul 21 01:38:25.595622 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:38:25.595579 2580 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd13d7ca_7c61_4c40_a86e_a58c1a7d1390.slice/crio-conmon-298e0cb45bdb396913774f6b76b2f4bd244c1e2ff2049dfd86e0212576ef9885.scope\": RecentStats: unable to find data in memory cache]" Jul 21 01:38:25.811104 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:25.811049 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b47d488dd-h2wcn" Jul 21 01:38:25.924592 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:25.924544 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6b98d6767c-27dq7" Jul 21 01:38:25.924759 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:25.924639 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6b98d6767c-27dq7" Jul 21 01:38:25.997419 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:25.997366 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mrb8x\" (UniqueName: \"kubernetes.io/projected/fd13d7ca-7c61-4c40-a86e-a58c1a7d1390-kube-api-access-mrb8x\") pod \"fd13d7ca-7c61-4c40-a86e-a58c1a7d1390\" (UID: \"fd13d7ca-7c61-4c40-a86e-a58c1a7d1390\") " Jul 21 01:38:25.997419 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:25.997425 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fd13d7ca-7c61-4c40-a86e-a58c1a7d1390-bound-sa-token\") pod \"fd13d7ca-7c61-4c40-a86e-a58c1a7d1390\" (UID: \"fd13d7ca-7c61-4c40-a86e-a58c1a7d1390\") " Jul 21 01:38:25.997668 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:25.997483 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fd13d7ca-7c61-4c40-a86e-a58c1a7d1390-installation-pull-secrets\") pod \"fd13d7ca-7c61-4c40-a86e-a58c1a7d1390\" (UID: \"fd13d7ca-7c61-4c40-a86e-a58c1a7d1390\") " Jul 21 01:38:25.997668 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:25.997516 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fd13d7ca-7c61-4c40-a86e-a58c1a7d1390-registry-certificates\") pod \"fd13d7ca-7c61-4c40-a86e-a58c1a7d1390\" (UID: \"fd13d7ca-7c61-4c40-a86e-a58c1a7d1390\") " Jul 21 01:38:25.997668 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:25.997620 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fd13d7ca-7c61-4c40-a86e-a58c1a7d1390-trusted-ca\") pod \"fd13d7ca-7c61-4c40-a86e-a58c1a7d1390\" (UID: \"fd13d7ca-7c61-4c40-a86e-a58c1a7d1390\") " Jul 21 01:38:25.997824 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:25.997724 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fd13d7ca-7c61-4c40-a86e-a58c1a7d1390-image-registry-private-configuration\") pod \"fd13d7ca-7c61-4c40-a86e-a58c1a7d1390\" (UID: \"fd13d7ca-7c61-4c40-a86e-a58c1a7d1390\") " Jul 21 01:38:25.997824 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:25.997778 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd13d7ca-7c61-4c40-a86e-a58c1a7d1390-registry-tls\") pod \"fd13d7ca-7c61-4c40-a86e-a58c1a7d1390\" (UID: \"fd13d7ca-7c61-4c40-a86e-a58c1a7d1390\") " Jul 21 01:38:25.997928 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:25.997835 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fd13d7ca-7c61-4c40-a86e-a58c1a7d1390-ca-trust-extracted\") pod \"fd13d7ca-7c61-4c40-a86e-a58c1a7d1390\" (UID: \"fd13d7ca-7c61-4c40-a86e-a58c1a7d1390\") " Jul 21 01:38:25.998009 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:25.997858 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd13d7ca-7c61-4c40-a86e-a58c1a7d1390-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "fd13d7ca-7c61-4c40-a86e-a58c1a7d1390" (UID: "fd13d7ca-7c61-4c40-a86e-a58c1a7d1390"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 01:38:25.998103 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:25.998080 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd13d7ca-7c61-4c40-a86e-a58c1a7d1390-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "fd13d7ca-7c61-4c40-a86e-a58c1a7d1390" (UID: "fd13d7ca-7c61-4c40-a86e-a58c1a7d1390"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 01:38:25.998170 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:25.998150 2580 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fd13d7ca-7c61-4c40-a86e-a58c1a7d1390-registry-certificates\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 21 01:38:25.998221 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:25.998170 2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fd13d7ca-7c61-4c40-a86e-a58c1a7d1390-trusted-ca\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 21 01:38:26.000232 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:26.000199 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd13d7ca-7c61-4c40-a86e-a58c1a7d1390-kube-api-access-mrb8x" (OuterVolumeSpecName: "kube-api-access-mrb8x") pod "fd13d7ca-7c61-4c40-a86e-a58c1a7d1390" (UID: "fd13d7ca-7c61-4c40-a86e-a58c1a7d1390"). InnerVolumeSpecName "kube-api-access-mrb8x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 01:38:26.000384 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:26.000277 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd13d7ca-7c61-4c40-a86e-a58c1a7d1390-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "fd13d7ca-7c61-4c40-a86e-a58c1a7d1390" (UID: "fd13d7ca-7c61-4c40-a86e-a58c1a7d1390"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 01:38:26.000384 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:26.000324 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd13d7ca-7c61-4c40-a86e-a58c1a7d1390-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "fd13d7ca-7c61-4c40-a86e-a58c1a7d1390" (UID: "fd13d7ca-7c61-4c40-a86e-a58c1a7d1390"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 01:38:26.000514 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:26.000410 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd13d7ca-7c61-4c40-a86e-a58c1a7d1390-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "fd13d7ca-7c61-4c40-a86e-a58c1a7d1390" (UID: "fd13d7ca-7c61-4c40-a86e-a58c1a7d1390"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 01:38:26.000514 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:26.000429 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd13d7ca-7c61-4c40-a86e-a58c1a7d1390-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "fd13d7ca-7c61-4c40-a86e-a58c1a7d1390" (UID: "fd13d7ca-7c61-4c40-a86e-a58c1a7d1390"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 01:38:26.006399 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:26.006376 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd13d7ca-7c61-4c40-a86e-a58c1a7d1390-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "fd13d7ca-7c61-4c40-a86e-a58c1a7d1390" (UID: "fd13d7ca-7c61-4c40-a86e-a58c1a7d1390"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 01:38:26.099222 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:26.099118 2580 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fd13d7ca-7c61-4c40-a86e-a58c1a7d1390-installation-pull-secrets\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 21 01:38:26.099222 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:26.099159 2580 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fd13d7ca-7c61-4c40-a86e-a58c1a7d1390-image-registry-private-configuration\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 21 01:38:26.099222 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:26.099174 2580 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd13d7ca-7c61-4c40-a86e-a58c1a7d1390-registry-tls\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 21 01:38:26.099222 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:26.099187 2580 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fd13d7ca-7c61-4c40-a86e-a58c1a7d1390-ca-trust-extracted\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 21 01:38:26.099222 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:26.099201 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrb8x\" (UniqueName: \"kubernetes.io/projected/fd13d7ca-7c61-4c40-a86e-a58c1a7d1390-kube-api-access-mrb8x\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 21 01:38:26.099222 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:26.099214 2580 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fd13d7ca-7c61-4c40-a86e-a58c1a7d1390-bound-sa-token\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 21 01:38:26.668348 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:26.668314 2580 generic.go:356] "Generic (PLEG): container finished" podID="fd13d7ca-7c61-4c40-a86e-a58c1a7d1390" containerID="298e0cb45bdb396913774f6b76b2f4bd244c1e2ff2049dfd86e0212576ef9885" exitCode=0 Jul 21 01:38:26.668832 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:26.668375 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b47d488dd-h2wcn" Jul 21 01:38:26.668832 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:26.668401 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b47d488dd-h2wcn" event={"ID":"fd13d7ca-7c61-4c40-a86e-a58c1a7d1390","Type":"ContainerDied","Data":"298e0cb45bdb396913774f6b76b2f4bd244c1e2ff2049dfd86e0212576ef9885"} Jul 21 01:38:26.668832 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:26.668481 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b47d488dd-h2wcn" event={"ID":"fd13d7ca-7c61-4c40-a86e-a58c1a7d1390","Type":"ContainerDied","Data":"4456b9b619688cbcc405888e7cfca728acc3794ac957252bdfad5b29578c84b9"} Jul 21 01:38:26.668832 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:26.668512 2580 scope.go:117] "RemoveContainer" containerID="298e0cb45bdb396913774f6b76b2f4bd244c1e2ff2049dfd86e0212576ef9885" Jul 21 01:38:26.677253 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:26.677233 2580 scope.go:117] "RemoveContainer" containerID="298e0cb45bdb396913774f6b76b2f4bd244c1e2ff2049dfd86e0212576ef9885" Jul 21 01:38:26.677537 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:38:26.677516 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"298e0cb45bdb396913774f6b76b2f4bd244c1e2ff2049dfd86e0212576ef9885\": container with ID starting with 298e0cb45bdb396913774f6b76b2f4bd244c1e2ff2049dfd86e0212576ef9885 not found: ID does not exist" containerID="298e0cb45bdb396913774f6b76b2f4bd244c1e2ff2049dfd86e0212576ef9885" Jul 21 01:38:26.677592 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:26.677546 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"298e0cb45bdb396913774f6b76b2f4bd244c1e2ff2049dfd86e0212576ef9885"} err="failed to get container status \"298e0cb45bdb396913774f6b76b2f4bd244c1e2ff2049dfd86e0212576ef9885\": rpc error: code = NotFound desc = could not find container \"298e0cb45bdb396913774f6b76b2f4bd244c1e2ff2049dfd86e0212576ef9885\": container with ID starting with 298e0cb45bdb396913774f6b76b2f4bd244c1e2ff2049dfd86e0212576ef9885 not found: ID does not exist" Jul 21 01:38:26.689530 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:26.689505 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-b47d488dd-h2wcn"] Jul 21 01:38:26.693304 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:26.693283 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-b47d488dd-h2wcn"] Jul 21 01:38:26.876207 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:26.876172 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd13d7ca-7c61-4c40-a86e-a58c1a7d1390" path="/var/lib/kubelet/pods/fd13d7ca-7c61-4c40-a86e-a58c1a7d1390/volumes" Jul 21 01:38:33.700388 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:33.700350 2580 generic.go:356] "Generic (PLEG): container finished" podID="de3854b8-6d90-46d9-9ab9-60a83f7a58be" containerID="e31e973327c99c790bb72616aa8255ab8582cf8f4448dd66cbe8b7e495d3a4e5" exitCode=0 Jul 21 01:38:33.700826 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:33.700406 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-2pscn" event={"ID":"de3854b8-6d90-46d9-9ab9-60a83f7a58be","Type":"ContainerDied","Data":"e31e973327c99c790bb72616aa8255ab8582cf8f4448dd66cbe8b7e495d3a4e5"} Jul 21 01:38:33.700826 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:33.700730 2580 scope.go:117] "RemoveContainer" containerID="e31e973327c99c790bb72616aa8255ab8582cf8f4448dd66cbe8b7e495d3a4e5" Jul 21 01:38:34.706268 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:34.706235 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-2pscn" event={"ID":"de3854b8-6d90-46d9-9ab9-60a83f7a58be","Type":"ContainerStarted","Data":"a9b0ff4268649a0932f9ebf33c41be6a9cf607d86f7be31404868345dc38ec23"} Jul 21 01:38:45.928783 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:45.928756 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6b98d6767c-27dq7" Jul 21 01:38:45.932433 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:45.932410 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6b98d6767c-27dq7" Jul 21 01:38:48.742503 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:48.742420 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6b6c89d7f8-7tjjt" podUID="7fe2a9ec-987b-46a3-9773-3cc1e34d0259" containerName="console" containerID="cri-o://9dc1a85522a1f379c80b73926808313091c7893a523f90eb4aad0e64ca0b03a9" gracePeriod=15 Jul 21 01:38:48.979700 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:48.979674 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b6c89d7f8-7tjjt" Jul 21 01:38:49.104692 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:49.104656 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7fe2a9ec-987b-46a3-9773-3cc1e34d0259-console-oauth-config\") pod \"7fe2a9ec-987b-46a3-9773-3cc1e34d0259\" (UID: \"7fe2a9ec-987b-46a3-9773-3cc1e34d0259\") " Jul 21 01:38:49.104861 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:49.104726 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7fe2a9ec-987b-46a3-9773-3cc1e34d0259-trusted-ca-bundle\") pod \"7fe2a9ec-987b-46a3-9773-3cc1e34d0259\" (UID: \"7fe2a9ec-987b-46a3-9773-3cc1e34d0259\") " Jul 21 01:38:49.104861 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:49.104763 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7fe2a9ec-987b-46a3-9773-3cc1e34d0259-service-ca\") pod \"7fe2a9ec-987b-46a3-9773-3cc1e34d0259\" (UID: \"7fe2a9ec-987b-46a3-9773-3cc1e34d0259\") " Jul 21 01:38:49.104861 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:49.104796 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nk26c\" (UniqueName: \"kubernetes.io/projected/7fe2a9ec-987b-46a3-9773-3cc1e34d0259-kube-api-access-nk26c\") pod \"7fe2a9ec-987b-46a3-9773-3cc1e34d0259\" (UID: \"7fe2a9ec-987b-46a3-9773-3cc1e34d0259\") " Jul 21 01:38:49.105004 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:49.104956 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7fe2a9ec-987b-46a3-9773-3cc1e34d0259-console-serving-cert\") pod \"7fe2a9ec-987b-46a3-9773-3cc1e34d0259\" (UID: \"7fe2a9ec-987b-46a3-9773-3cc1e34d0259\") " Jul 21 01:38:49.105062 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:49.105016 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7fe2a9ec-987b-46a3-9773-3cc1e34d0259-console-config\") pod \"7fe2a9ec-987b-46a3-9773-3cc1e34d0259\" (UID: \"7fe2a9ec-987b-46a3-9773-3cc1e34d0259\") " Jul 21 01:38:49.105114 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:49.105066 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7fe2a9ec-987b-46a3-9773-3cc1e34d0259-oauth-serving-cert\") pod \"7fe2a9ec-987b-46a3-9773-3cc1e34d0259\" (UID: \"7fe2a9ec-987b-46a3-9773-3cc1e34d0259\") " Jul 21 01:38:49.105203 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:49.105172 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fe2a9ec-987b-46a3-9773-3cc1e34d0259-service-ca" (OuterVolumeSpecName: "service-ca") pod "7fe2a9ec-987b-46a3-9773-3cc1e34d0259" (UID: "7fe2a9ec-987b-46a3-9773-3cc1e34d0259"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 01:38:49.105203 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:49.105194 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fe2a9ec-987b-46a3-9773-3cc1e34d0259-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "7fe2a9ec-987b-46a3-9773-3cc1e34d0259" (UID: "7fe2a9ec-987b-46a3-9773-3cc1e34d0259"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 01:38:49.105403 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:49.105344 2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7fe2a9ec-987b-46a3-9773-3cc1e34d0259-trusted-ca-bundle\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 21 01:38:49.105403 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:49.105364 2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7fe2a9ec-987b-46a3-9773-3cc1e34d0259-service-ca\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 21 01:38:49.105567 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:49.105546 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fe2a9ec-987b-46a3-9773-3cc1e34d0259-console-config" (OuterVolumeSpecName: "console-config") pod "7fe2a9ec-987b-46a3-9773-3cc1e34d0259" (UID: "7fe2a9ec-987b-46a3-9773-3cc1e34d0259"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 01:38:49.105567 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:49.105551 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fe2a9ec-987b-46a3-9773-3cc1e34d0259-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7fe2a9ec-987b-46a3-9773-3cc1e34d0259" (UID: "7fe2a9ec-987b-46a3-9773-3cc1e34d0259"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 01:38:49.107199 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:49.107175 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fe2a9ec-987b-46a3-9773-3cc1e34d0259-kube-api-access-nk26c" (OuterVolumeSpecName: "kube-api-access-nk26c") pod "7fe2a9ec-987b-46a3-9773-3cc1e34d0259" (UID: "7fe2a9ec-987b-46a3-9773-3cc1e34d0259"). InnerVolumeSpecName "kube-api-access-nk26c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 01:38:49.107344 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:49.107322 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fe2a9ec-987b-46a3-9773-3cc1e34d0259-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7fe2a9ec-987b-46a3-9773-3cc1e34d0259" (UID: "7fe2a9ec-987b-46a3-9773-3cc1e34d0259"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 01:38:49.107344 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:49.107331 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fe2a9ec-987b-46a3-9773-3cc1e34d0259-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7fe2a9ec-987b-46a3-9773-3cc1e34d0259" (UID: "7fe2a9ec-987b-46a3-9773-3cc1e34d0259"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 01:38:49.206358 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:49.206308 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nk26c\" (UniqueName: \"kubernetes.io/projected/7fe2a9ec-987b-46a3-9773-3cc1e34d0259-kube-api-access-nk26c\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 21 01:38:49.206358 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:49.206354 2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7fe2a9ec-987b-46a3-9773-3cc1e34d0259-console-serving-cert\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 21 01:38:49.206358 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:49.206366 2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7fe2a9ec-987b-46a3-9773-3cc1e34d0259-console-config\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 21 01:38:49.206358 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:49.206377 2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7fe2a9ec-987b-46a3-9773-3cc1e34d0259-oauth-serving-cert\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 21 01:38:49.206358 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:49.206385 2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7fe2a9ec-987b-46a3-9773-3cc1e34d0259-console-oauth-config\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 21 01:38:49.771524 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:49.771490 2580 generic.go:356] "Generic (PLEG): container finished" podID="7fe2a9ec-987b-46a3-9773-3cc1e34d0259" containerID="9dc1a85522a1f379c80b73926808313091c7893a523f90eb4aad0e64ca0b03a9" exitCode=0 Jul 21 01:38:49.771975 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:49.771532 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b6c89d7f8-7tjjt" event={"ID":"7fe2a9ec-987b-46a3-9773-3cc1e34d0259","Type":"ContainerDied","Data":"9dc1a85522a1f379c80b73926808313091c7893a523f90eb4aad0e64ca0b03a9"} Jul 21 01:38:49.771975 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:49.771535 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b6c89d7f8-7tjjt" Jul 21 01:38:49.771975 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:49.771571 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b6c89d7f8-7tjjt" event={"ID":"7fe2a9ec-987b-46a3-9773-3cc1e34d0259","Type":"ContainerDied","Data":"00bf32de0bd5db0f00a0a06432a58a5933c3ee9a1a1b185825b28698a63453c7"} Jul 21 01:38:49.771975 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:49.771591 2580 scope.go:117] "RemoveContainer" containerID="9dc1a85522a1f379c80b73926808313091c7893a523f90eb4aad0e64ca0b03a9" Jul 21 01:38:49.780398 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:49.780376 2580 scope.go:117] "RemoveContainer" containerID="9dc1a85522a1f379c80b73926808313091c7893a523f90eb4aad0e64ca0b03a9" Jul 21 01:38:49.780699 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:38:49.780681 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dc1a85522a1f379c80b73926808313091c7893a523f90eb4aad0e64ca0b03a9\": container with ID starting with 9dc1a85522a1f379c80b73926808313091c7893a523f90eb4aad0e64ca0b03a9 not found: ID does not exist" containerID="9dc1a85522a1f379c80b73926808313091c7893a523f90eb4aad0e64ca0b03a9" Jul 21 01:38:49.780757 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:49.780710 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dc1a85522a1f379c80b73926808313091c7893a523f90eb4aad0e64ca0b03a9"} err="failed to get container status \"9dc1a85522a1f379c80b73926808313091c7893a523f90eb4aad0e64ca0b03a9\": rpc error: code = NotFound desc = could not find container \"9dc1a85522a1f379c80b73926808313091c7893a523f90eb4aad0e64ca0b03a9\": container with ID starting with 9dc1a85522a1f379c80b73926808313091c7893a523f90eb4aad0e64ca0b03a9 not found: ID does not exist" Jul 21 01:38:49.792227 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:49.792200 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b6c89d7f8-7tjjt"] Jul 21 01:38:49.800007 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:49.799985 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6b6c89d7f8-7tjjt"] Jul 21 01:38:50.875882 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:38:50.875847 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7fe2a9ec-987b-46a3-9773-3cc1e34d0259" path="/var/lib/kubelet/pods/7fe2a9ec-987b-46a3-9773-3cc1e34d0259/volumes" Jul 21 01:39:27.912224 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:39:27.912187 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-8dd66cfd6-9gbmt"] Jul 21 01:39:27.912793 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:39:27.912564 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd13d7ca-7c61-4c40-a86e-a58c1a7d1390" containerName="registry" Jul 21 01:39:27.912793 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:39:27.912577 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd13d7ca-7c61-4c40-a86e-a58c1a7d1390" containerName="registry" Jul 21 01:39:27.912793 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:39:27.912595 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fe2a9ec-987b-46a3-9773-3cc1e34d0259" containerName="console" Jul 21 01:39:27.912793 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:39:27.912600 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fe2a9ec-987b-46a3-9773-3cc1e34d0259" containerName="console" Jul 21 01:39:27.912793 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:39:27.912658 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fe2a9ec-987b-46a3-9773-3cc1e34d0259" containerName="console" Jul 21 01:39:27.912793 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:39:27.912666 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd13d7ca-7c61-4c40-a86e-a58c1a7d1390" containerName="registry" Jul 21 01:39:27.919854 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:39:27.919825 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8dd66cfd6-9gbmt" Jul 21 01:39:27.926124 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:39:27.926094 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8dd66cfd6-9gbmt"] Jul 21 01:39:28.015566 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:39:28.015523 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bm848\" (UniqueName: \"kubernetes.io/projected/6796addb-d850-411b-80db-84e97535ca94-kube-api-access-bm848\") pod \"console-8dd66cfd6-9gbmt\" (UID: \"6796addb-d850-411b-80db-84e97535ca94\") " pod="openshift-console/console-8dd66cfd6-9gbmt" Jul 21 01:39:28.015779 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:39:28.015595 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6796addb-d850-411b-80db-84e97535ca94-console-serving-cert\") pod \"console-8dd66cfd6-9gbmt\" (UID: \"6796addb-d850-411b-80db-84e97535ca94\") " pod="openshift-console/console-8dd66cfd6-9gbmt" Jul 21 01:39:28.015779 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:39:28.015630 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6796addb-d850-411b-80db-84e97535ca94-oauth-serving-cert\") pod \"console-8dd66cfd6-9gbmt\" (UID: \"6796addb-d850-411b-80db-84e97535ca94\") " pod="openshift-console/console-8dd66cfd6-9gbmt" Jul 21 01:39:28.015779 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:39:28.015655 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6796addb-d850-411b-80db-84e97535ca94-service-ca\") pod \"console-8dd66cfd6-9gbmt\" (UID: \"6796addb-d850-411b-80db-84e97535ca94\") " pod="openshift-console/console-8dd66cfd6-9gbmt" Jul 21 01:39:28.015779 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:39:28.015678 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6796addb-d850-411b-80db-84e97535ca94-console-oauth-config\") pod \"console-8dd66cfd6-9gbmt\" (UID: \"6796addb-d850-411b-80db-84e97535ca94\") " pod="openshift-console/console-8dd66cfd6-9gbmt" Jul 21 01:39:28.015779 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:39:28.015726 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6796addb-d850-411b-80db-84e97535ca94-trusted-ca-bundle\") pod \"console-8dd66cfd6-9gbmt\" (UID: \"6796addb-d850-411b-80db-84e97535ca94\") " pod="openshift-console/console-8dd66cfd6-9gbmt" Jul 21 01:39:28.015968 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:39:28.015788 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6796addb-d850-411b-80db-84e97535ca94-console-config\") pod \"console-8dd66cfd6-9gbmt\" (UID: \"6796addb-d850-411b-80db-84e97535ca94\") " pod="openshift-console/console-8dd66cfd6-9gbmt" Jul 21 01:39:28.116758 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:39:28.116713 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6796addb-d850-411b-80db-84e97535ca94-trusted-ca-bundle\") pod \"console-8dd66cfd6-9gbmt\" (UID: \"6796addb-d850-411b-80db-84e97535ca94\") " pod="openshift-console/console-8dd66cfd6-9gbmt" Jul 21 01:39:28.116758 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:39:28.116756 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6796addb-d850-411b-80db-84e97535ca94-console-config\") pod \"console-8dd66cfd6-9gbmt\" (UID: \"6796addb-d850-411b-80db-84e97535ca94\") " pod="openshift-console/console-8dd66cfd6-9gbmt" Jul 21 01:39:28.117030 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:39:28.116785 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm848\" (UniqueName: \"kubernetes.io/projected/6796addb-d850-411b-80db-84e97535ca94-kube-api-access-bm848\") pod \"console-8dd66cfd6-9gbmt\" (UID: \"6796addb-d850-411b-80db-84e97535ca94\") " pod="openshift-console/console-8dd66cfd6-9gbmt" Jul 21 01:39:28.117030 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:39:28.116839 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6796addb-d850-411b-80db-84e97535ca94-console-serving-cert\") pod \"console-8dd66cfd6-9gbmt\" (UID: \"6796addb-d850-411b-80db-84e97535ca94\") " pod="openshift-console/console-8dd66cfd6-9gbmt" Jul 21 01:39:28.117030 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:39:28.116876 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6796addb-d850-411b-80db-84e97535ca94-oauth-serving-cert\") pod \"console-8dd66cfd6-9gbmt\" (UID: \"6796addb-d850-411b-80db-84e97535ca94\") " pod="openshift-console/console-8dd66cfd6-9gbmt" Jul 21 01:39:28.117030 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:39:28.116904 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6796addb-d850-411b-80db-84e97535ca94-service-ca\") pod \"console-8dd66cfd6-9gbmt\" (UID: \"6796addb-d850-411b-80db-84e97535ca94\") " pod="openshift-console/console-8dd66cfd6-9gbmt" Jul 21 01:39:28.117030 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:39:28.116930 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6796addb-d850-411b-80db-84e97535ca94-console-oauth-config\") pod \"console-8dd66cfd6-9gbmt\" (UID: \"6796addb-d850-411b-80db-84e97535ca94\") " pod="openshift-console/console-8dd66cfd6-9gbmt" Jul 21 01:39:28.117647 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:39:28.117620 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6796addb-d850-411b-80db-84e97535ca94-console-config\") pod \"console-8dd66cfd6-9gbmt\" (UID: \"6796addb-d850-411b-80db-84e97535ca94\") " pod="openshift-console/console-8dd66cfd6-9gbmt" Jul 21 01:39:28.118071 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:39:28.117746 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6796addb-d850-411b-80db-84e97535ca94-trusted-ca-bundle\") pod \"console-8dd66cfd6-9gbmt\" (UID: \"6796addb-d850-411b-80db-84e97535ca94\") " pod="openshift-console/console-8dd66cfd6-9gbmt" Jul 21 01:39:28.118071 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:39:28.117764 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6796addb-d850-411b-80db-84e97535ca94-oauth-serving-cert\") pod \"console-8dd66cfd6-9gbmt\" (UID: \"6796addb-d850-411b-80db-84e97535ca94\") " pod="openshift-console/console-8dd66cfd6-9gbmt" Jul 21 01:39:28.118198 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:39:28.117792 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6796addb-d850-411b-80db-84e97535ca94-service-ca\") pod \"console-8dd66cfd6-9gbmt\" (UID: \"6796addb-d850-411b-80db-84e97535ca94\") " pod="openshift-console/console-8dd66cfd6-9gbmt" Jul 21 01:39:28.119314 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:39:28.119280 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6796addb-d850-411b-80db-84e97535ca94-console-serving-cert\") pod \"console-8dd66cfd6-9gbmt\" (UID: \"6796addb-d850-411b-80db-84e97535ca94\") " pod="openshift-console/console-8dd66cfd6-9gbmt" Jul 21 01:39:28.119430 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:39:28.119380 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6796addb-d850-411b-80db-84e97535ca94-console-oauth-config\") pod \"console-8dd66cfd6-9gbmt\" (UID: \"6796addb-d850-411b-80db-84e97535ca94\") " pod="openshift-console/console-8dd66cfd6-9gbmt" Jul 21 01:39:28.124287 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:39:28.124264 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm848\" (UniqueName: \"kubernetes.io/projected/6796addb-d850-411b-80db-84e97535ca94-kube-api-access-bm848\") pod \"console-8dd66cfd6-9gbmt\" (UID: \"6796addb-d850-411b-80db-84e97535ca94\") " pod="openshift-console/console-8dd66cfd6-9gbmt" Jul 21 01:39:28.231900 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:39:28.231809 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8dd66cfd6-9gbmt" Jul 21 01:39:28.363493 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:39:28.363461 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8dd66cfd6-9gbmt"] Jul 21 01:39:28.365941 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:39:28.365915 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6796addb_d850_411b_80db_84e97535ca94.slice/crio-99fbdf7efa16cbd518295c0d38574ff4da06be558d70c09d03c26cd54257bcca WatchSource:0}: Error finding container 99fbdf7efa16cbd518295c0d38574ff4da06be558d70c09d03c26cd54257bcca: Status 404 returned error can't find the container with id 99fbdf7efa16cbd518295c0d38574ff4da06be558d70c09d03c26cd54257bcca Jul 21 01:39:28.930889 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:39:28.930852 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8dd66cfd6-9gbmt" event={"ID":"6796addb-d850-411b-80db-84e97535ca94","Type":"ContainerStarted","Data":"dcc1038581097533a22b68e48cbacecf8edbb00fb4a4022add2e1be14aab3abe"} Jul 21 01:39:28.931292 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:39:28.930900 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8dd66cfd6-9gbmt" event={"ID":"6796addb-d850-411b-80db-84e97535ca94","Type":"ContainerStarted","Data":"99fbdf7efa16cbd518295c0d38574ff4da06be558d70c09d03c26cd54257bcca"} Jul 21 01:39:28.948215 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:39:28.948166 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-8dd66cfd6-9gbmt" podStartSLOduration=1.948149353 podStartE2EDuration="1.948149353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 01:39:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:39:28.946409741 +0000 UTC m=+192.533575718" watchObservedRunningTime="2026-07-21 01:39:28.948149353 +0000 UTC m=+192.535315387" Jul 21 01:39:38.232461 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:39:38.232427 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-8dd66cfd6-9gbmt" Jul 21 01:39:38.232916 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:39:38.232503 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8dd66cfd6-9gbmt" Jul 21 01:39:38.237241 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:39:38.237219 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-8dd66cfd6-9gbmt" Jul 21 01:39:38.976744 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:39:38.976715 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-8dd66cfd6-9gbmt" Jul 21 01:39:39.020933 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:39:39.020902 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6fd7fd45f8-bf2k7"] Jul 21 01:40:04.045009 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:40:04.044944 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6fd7fd45f8-bf2k7" podUID="1b0a7fe3-f91b-4b27-8e60-2ff95099f475" containerName="console" containerID="cri-o://fe91cb72ae3485bddcf6544319cc0caccd81552da47e55f5e5faf61e8e5e3d75" gracePeriod=15 Jul 21 01:40:04.297742 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:40:04.297680 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fd7fd45f8-bf2k7" Jul 21 01:40:04.431818 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:40:04.431770 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1b0a7fe3-f91b-4b27-8e60-2ff95099f475-service-ca\") pod \"1b0a7fe3-f91b-4b27-8e60-2ff95099f475\" (UID: \"1b0a7fe3-f91b-4b27-8e60-2ff95099f475\") " Jul 21 01:40:04.432036 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:40:04.431844 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b0a7fe3-f91b-4b27-8e60-2ff95099f475-trusted-ca-bundle\") pod \"1b0a7fe3-f91b-4b27-8e60-2ff95099f475\" (UID: \"1b0a7fe3-f91b-4b27-8e60-2ff95099f475\") " Jul 21 01:40:04.432036 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:40:04.431870 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1b0a7fe3-f91b-4b27-8e60-2ff95099f475-console-config\") pod \"1b0a7fe3-f91b-4b27-8e60-2ff95099f475\" (UID: \"1b0a7fe3-f91b-4b27-8e60-2ff95099f475\") " Jul 21 01:40:04.432036 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:40:04.431904 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l877d\" (UniqueName: \"kubernetes.io/projected/1b0a7fe3-f91b-4b27-8e60-2ff95099f475-kube-api-access-l877d\") pod \"1b0a7fe3-f91b-4b27-8e60-2ff95099f475\" (UID: \"1b0a7fe3-f91b-4b27-8e60-2ff95099f475\") " Jul 21 01:40:04.432036 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:40:04.431941 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1b0a7fe3-f91b-4b27-8e60-2ff95099f475-console-serving-cert\") pod \"1b0a7fe3-f91b-4b27-8e60-2ff95099f475\" (UID: \"1b0a7fe3-f91b-4b27-8e60-2ff95099f475\") " Jul 21 01:40:04.432036 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:40:04.431985 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1b0a7fe3-f91b-4b27-8e60-2ff95099f475-oauth-serving-cert\") pod \"1b0a7fe3-f91b-4b27-8e60-2ff95099f475\" (UID: \"1b0a7fe3-f91b-4b27-8e60-2ff95099f475\") " Jul 21 01:40:04.432296 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:40:04.432208 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b0a7fe3-f91b-4b27-8e60-2ff95099f475-service-ca" (OuterVolumeSpecName: "service-ca") pod "1b0a7fe3-f91b-4b27-8e60-2ff95099f475" (UID: "1b0a7fe3-f91b-4b27-8e60-2ff95099f475"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 01:40:04.432296 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:40:04.432186 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1b0a7fe3-f91b-4b27-8e60-2ff95099f475-console-oauth-config\") pod \"1b0a7fe3-f91b-4b27-8e60-2ff95099f475\" (UID: \"1b0a7fe3-f91b-4b27-8e60-2ff95099f475\") " Jul 21 01:40:04.432296 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:40:04.432280 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b0a7fe3-f91b-4b27-8e60-2ff95099f475-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "1b0a7fe3-f91b-4b27-8e60-2ff95099f475" (UID: "1b0a7fe3-f91b-4b27-8e60-2ff95099f475"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 01:40:04.432487 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:40:04.432340 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b0a7fe3-f91b-4b27-8e60-2ff95099f475-console-config" (OuterVolumeSpecName: "console-config") pod "1b0a7fe3-f91b-4b27-8e60-2ff95099f475" (UID: "1b0a7fe3-f91b-4b27-8e60-2ff95099f475"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 01:40:04.432487 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:40:04.432398 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b0a7fe3-f91b-4b27-8e60-2ff95099f475-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "1b0a7fe3-f91b-4b27-8e60-2ff95099f475" (UID: "1b0a7fe3-f91b-4b27-8e60-2ff95099f475"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 01:40:04.432609 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:40:04.432588 2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1b0a7fe3-f91b-4b27-8e60-2ff95099f475-oauth-serving-cert\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 21 01:40:04.432663 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:40:04.432614 2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1b0a7fe3-f91b-4b27-8e60-2ff95099f475-service-ca\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 21 01:40:04.432663 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:40:04.432628 2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b0a7fe3-f91b-4b27-8e60-2ff95099f475-trusted-ca-bundle\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 21 01:40:04.432663 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:40:04.432645 2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1b0a7fe3-f91b-4b27-8e60-2ff95099f475-console-config\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 21 01:40:04.434147 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:40:04.434122 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b0a7fe3-f91b-4b27-8e60-2ff95099f475-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "1b0a7fe3-f91b-4b27-8e60-2ff95099f475" (UID: "1b0a7fe3-f91b-4b27-8e60-2ff95099f475"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 01:40:04.434243 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:40:04.434156 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b0a7fe3-f91b-4b27-8e60-2ff95099f475-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "1b0a7fe3-f91b-4b27-8e60-2ff95099f475" (UID: "1b0a7fe3-f91b-4b27-8e60-2ff95099f475"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 01:40:04.434243 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:40:04.434169 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b0a7fe3-f91b-4b27-8e60-2ff95099f475-kube-api-access-l877d" (OuterVolumeSpecName: "kube-api-access-l877d") pod "1b0a7fe3-f91b-4b27-8e60-2ff95099f475" (UID: "1b0a7fe3-f91b-4b27-8e60-2ff95099f475"). InnerVolumeSpecName "kube-api-access-l877d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 01:40:04.533076 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:40:04.533032 2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1b0a7fe3-f91b-4b27-8e60-2ff95099f475-console-oauth-config\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 21 01:40:04.533076 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:40:04.533068 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l877d\" (UniqueName: \"kubernetes.io/projected/1b0a7fe3-f91b-4b27-8e60-2ff95099f475-kube-api-access-l877d\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 21 01:40:04.533076 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:40:04.533082 2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1b0a7fe3-f91b-4b27-8e60-2ff95099f475-console-serving-cert\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 21 01:40:05.087350 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:40:05.087313 2580 generic.go:356] "Generic (PLEG): container finished" podID="1b0a7fe3-f91b-4b27-8e60-2ff95099f475" containerID="fe91cb72ae3485bddcf6544319cc0caccd81552da47e55f5e5faf61e8e5e3d75" exitCode=0 Jul 21 01:40:05.087779 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:40:05.087357 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fd7fd45f8-bf2k7" Jul 21 01:40:05.087779 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:40:05.087397 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fd7fd45f8-bf2k7" event={"ID":"1b0a7fe3-f91b-4b27-8e60-2ff95099f475","Type":"ContainerDied","Data":"fe91cb72ae3485bddcf6544319cc0caccd81552da47e55f5e5faf61e8e5e3d75"} Jul 21 01:40:05.087779 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:40:05.087442 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fd7fd45f8-bf2k7" event={"ID":"1b0a7fe3-f91b-4b27-8e60-2ff95099f475","Type":"ContainerDied","Data":"2c2af01602eed96b7db214ae328bdc69573760218e069db7879d7decd6324e1e"} Jul 21 01:40:05.087779 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:40:05.087489 2580 scope.go:117] "RemoveContainer" containerID="fe91cb72ae3485bddcf6544319cc0caccd81552da47e55f5e5faf61e8e5e3d75" Jul 21 01:40:05.096245 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:40:05.096225 2580 scope.go:117] "RemoveContainer" containerID="fe91cb72ae3485bddcf6544319cc0caccd81552da47e55f5e5faf61e8e5e3d75" Jul 21 01:40:05.096523 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:40:05.096504 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe91cb72ae3485bddcf6544319cc0caccd81552da47e55f5e5faf61e8e5e3d75\": container with ID starting with fe91cb72ae3485bddcf6544319cc0caccd81552da47e55f5e5faf61e8e5e3d75 not found: ID does not exist" containerID="fe91cb72ae3485bddcf6544319cc0caccd81552da47e55f5e5faf61e8e5e3d75" Jul 21 01:40:05.096620 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:40:05.096531 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe91cb72ae3485bddcf6544319cc0caccd81552da47e55f5e5faf61e8e5e3d75"} err="failed to get container status \"fe91cb72ae3485bddcf6544319cc0caccd81552da47e55f5e5faf61e8e5e3d75\": rpc error: code = NotFound desc = could not find container \"fe91cb72ae3485bddcf6544319cc0caccd81552da47e55f5e5faf61e8e5e3d75\": container with ID starting with fe91cb72ae3485bddcf6544319cc0caccd81552da47e55f5e5faf61e8e5e3d75 not found: ID does not exist" Jul 21 01:40:05.103015 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:40:05.102992 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6fd7fd45f8-bf2k7"] Jul 21 01:40:05.106366 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:40:05.106339 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6fd7fd45f8-bf2k7"] Jul 21 01:40:06.875589 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:40:06.875556 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b0a7fe3-f91b-4b27-8e60-2ff95099f475" path="/var/lib/kubelet/pods/1b0a7fe3-f91b-4b27-8e60-2ff95099f475/volumes" Jul 21 01:40:29.700425 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:40:29.700390 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-h8dv9"] Jul 21 01:40:29.700899 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:40:29.700734 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b0a7fe3-f91b-4b27-8e60-2ff95099f475" containerName="console" Jul 21 01:40:29.700899 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:40:29.700746 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b0a7fe3-f91b-4b27-8e60-2ff95099f475" containerName="console" Jul 21 01:40:29.700899 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:40:29.700819 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b0a7fe3-f91b-4b27-8e60-2ff95099f475" containerName="console" Jul 21 01:40:29.704840 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:40:29.704821 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-h8dv9" Jul 21 01:40:29.707312 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:40:29.707275 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 21 01:40:29.708113 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:40:29.708094 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-wzw25\"" Jul 21 01:40:29.708171 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:40:29.708142 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 21 01:40:29.713020 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:40:29.712996 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-h8dv9"] Jul 21 01:40:29.736720 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:40:29.736674 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f2336e8f-9a6b-4775-8aeb-ffeeb9190646-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-h8dv9\" (UID: \"f2336e8f-9a6b-4775-8aeb-ffeeb9190646\") " pod="cert-manager/cert-manager-webhook-6486ffd997-h8dv9" Jul 21 01:40:29.736720 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:40:29.736718 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stshj\" (UniqueName: \"kubernetes.io/projected/f2336e8f-9a6b-4775-8aeb-ffeeb9190646-kube-api-access-stshj\") pod \"cert-manager-webhook-6486ffd997-h8dv9\" (UID: \"f2336e8f-9a6b-4775-8aeb-ffeeb9190646\") " pod="cert-manager/cert-manager-webhook-6486ffd997-h8dv9" Jul 21 01:40:29.838042 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:40:29.838002 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f2336e8f-9a6b-4775-8aeb-ffeeb9190646-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-h8dv9\" (UID: \"f2336e8f-9a6b-4775-8aeb-ffeeb9190646\") " pod="cert-manager/cert-manager-webhook-6486ffd997-h8dv9" Jul 21 01:40:29.838042 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:40:29.838041 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stshj\" (UniqueName: \"kubernetes.io/projected/f2336e8f-9a6b-4775-8aeb-ffeeb9190646-kube-api-access-stshj\") pod \"cert-manager-webhook-6486ffd997-h8dv9\" (UID: \"f2336e8f-9a6b-4775-8aeb-ffeeb9190646\") " pod="cert-manager/cert-manager-webhook-6486ffd997-h8dv9" Jul 21 01:40:29.852136 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:40:29.852101 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f2336e8f-9a6b-4775-8aeb-ffeeb9190646-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-h8dv9\" (UID: \"f2336e8f-9a6b-4775-8aeb-ffeeb9190646\") " pod="cert-manager/cert-manager-webhook-6486ffd997-h8dv9" Jul 21 01:40:29.852136 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:40:29.852129 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-stshj\" (UniqueName: \"kubernetes.io/projected/f2336e8f-9a6b-4775-8aeb-ffeeb9190646-kube-api-access-stshj\") pod \"cert-manager-webhook-6486ffd997-h8dv9\" (UID: \"f2336e8f-9a6b-4775-8aeb-ffeeb9190646\") " pod="cert-manager/cert-manager-webhook-6486ffd997-h8dv9" Jul 21 01:40:30.024853 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:40:30.024816 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-h8dv9" Jul 21 01:40:30.165586 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:40:30.165559 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-h8dv9"] Jul 21 01:40:30.168068 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:40:30.168042 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf2336e8f_9a6b_4775_8aeb_ffeeb9190646.slice/crio-cff0d651696cfcbc9c128273274ec0faea7fd8db747f58c08c2bed370cf51210 WatchSource:0}: Error finding container cff0d651696cfcbc9c128273274ec0faea7fd8db747f58c08c2bed370cf51210: Status 404 returned error can't find the container with id cff0d651696cfcbc9c128273274ec0faea7fd8db747f58c08c2bed370cf51210 Jul 21 01:40:30.189555 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:40:30.189523 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-h8dv9" event={"ID":"f2336e8f-9a6b-4775-8aeb-ffeeb9190646","Type":"ContainerStarted","Data":"cff0d651696cfcbc9c128273274ec0faea7fd8db747f58c08c2bed370cf51210"} Jul 21 01:40:34.211273 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:40:34.211240 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-h8dv9" event={"ID":"f2336e8f-9a6b-4775-8aeb-ffeeb9190646","Type":"ContainerStarted","Data":"80330690b94362ecf77164f560bc22dea01d0b4cc59a22e1cb25c6c2023e4cda"} Jul 21 01:40:34.211695 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:40:34.211310 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-h8dv9" Jul 21 01:40:34.227501 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:40:34.227436 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-h8dv9" podStartSLOduration=1.916698617 podStartE2EDuration="5.227419017s" totalImagesPullingTime="3.3107204s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 01:40:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:40:34.225784492 +0000 UTC m=+257.812950470" watchObservedRunningTime="2026-07-21 01:40:34.227419017 +0000 UTC m=+257.814584994" Jul 21 01:40:40.121319 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:40:40.121238 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-7k56g"] Jul 21 01:40:40.124609 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:40:40.124588 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-7k56g" Jul 21 01:40:40.127045 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:40:40.127021 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-7vpcn\"" Jul 21 01:40:40.134055 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:40:40.134034 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-7k56g"] Jul 21 01:40:40.219109 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:40:40.219072 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-h8dv9" Jul 21 01:40:40.231496 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:40:40.231467 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/18ab9158-6d6c-4646-9b6a-5a4170721842-bound-sa-token\") pod \"cert-manager-55b4d954d5-7k56g\" (UID: \"18ab9158-6d6c-4646-9b6a-5a4170721842\") " pod="cert-manager/cert-manager-55b4d954d5-7k56g" Jul 21 01:40:40.231647 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:40:40.231503 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fs62c\" (UniqueName: \"kubernetes.io/projected/18ab9158-6d6c-4646-9b6a-5a4170721842-kube-api-access-fs62c\") pod \"cert-manager-55b4d954d5-7k56g\" (UID: \"18ab9158-6d6c-4646-9b6a-5a4170721842\") " pod="cert-manager/cert-manager-55b4d954d5-7k56g" Jul 21 01:40:40.332819 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:40:40.332722 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/18ab9158-6d6c-4646-9b6a-5a4170721842-bound-sa-token\") pod \"cert-manager-55b4d954d5-7k56g\" (UID: \"18ab9158-6d6c-4646-9b6a-5a4170721842\") " pod="cert-manager/cert-manager-55b4d954d5-7k56g" Jul 21 01:40:40.332987 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:40:40.332858 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs62c\" (UniqueName: \"kubernetes.io/projected/18ab9158-6d6c-4646-9b6a-5a4170721842-kube-api-access-fs62c\") pod \"cert-manager-55b4d954d5-7k56g\" (UID: \"18ab9158-6d6c-4646-9b6a-5a4170721842\") " pod="cert-manager/cert-manager-55b4d954d5-7k56g" Jul 21 01:40:40.340412 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:40:40.340382 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/18ab9158-6d6c-4646-9b6a-5a4170721842-bound-sa-token\") pod \"cert-manager-55b4d954d5-7k56g\" (UID: \"18ab9158-6d6c-4646-9b6a-5a4170721842\") " pod="cert-manager/cert-manager-55b4d954d5-7k56g" Jul 21 01:40:40.340624 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:40:40.340607 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs62c\" (UniqueName: \"kubernetes.io/projected/18ab9158-6d6c-4646-9b6a-5a4170721842-kube-api-access-fs62c\") pod \"cert-manager-55b4d954d5-7k56g\" (UID: \"18ab9158-6d6c-4646-9b6a-5a4170721842\") " pod="cert-manager/cert-manager-55b4d954d5-7k56g" Jul 21 01:40:40.433407 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:40:40.433313 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-7k56g" Jul 21 01:40:40.564871 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:40:40.564847 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-7k56g"] Jul 21 01:40:40.567443 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:40:40.567415 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod18ab9158_6d6c_4646_9b6a_5a4170721842.slice/crio-4a4fa6c6aac6999966d115826bcbf658e6a1fdfbc02486c4571db9082a1de6df WatchSource:0}: Error finding container 4a4fa6c6aac6999966d115826bcbf658e6a1fdfbc02486c4571db9082a1de6df: Status 404 returned error can't find the container with id 4a4fa6c6aac6999966d115826bcbf658e6a1fdfbc02486c4571db9082a1de6df Jul 21 01:40:41.241274 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:40:41.241229 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-7k56g" event={"ID":"18ab9158-6d6c-4646-9b6a-5a4170721842","Type":"ContainerStarted","Data":"62b3decc93aeed2bad108d9e87e7b72e515d85eb80e262616518874e502cdf21"} Jul 21 01:40:41.241274 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:40:41.241267 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-7k56g" event={"ID":"18ab9158-6d6c-4646-9b6a-5a4170721842","Type":"ContainerStarted","Data":"4a4fa6c6aac6999966d115826bcbf658e6a1fdfbc02486c4571db9082a1de6df"} Jul 21 01:40:41.255498 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:40:41.255417 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-7k56g" podStartSLOduration=1.255397905 podStartE2EDuration="1.255397905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 01:40:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:40:41.254681962 +0000 UTC m=+264.841847940" watchObservedRunningTime="2026-07-21 01:40:41.255397905 +0000 UTC m=+264.842563881" Jul 21 01:40:59.850157 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:40:59.850122 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7d7c89d5b6-722r9"] Jul 21 01:40:59.855424 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:40:59.855397 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7d7c89d5b6-722r9" Jul 21 01:40:59.858272 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:40:59.858242 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 21 01:40:59.858478 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:40:59.858316 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-wdvgf\"" Jul 21 01:40:59.858591 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:40:59.858513 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 21 01:40:59.858591 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:40:59.858561 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 21 01:40:59.858936 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:40:59.858919 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 21 01:40:59.867627 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:40:59.867603 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7d7c89d5b6-722r9"] Jul 21 01:41:00.003506 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:00.003420 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/60d07d21-eb68-49ee-b399-5e1acb9b11cd-manifests\") pod \"opendatahub-operator-controller-manager-7d7c89d5b6-722r9\" (UID: \"60d07d21-eb68-49ee-b399-5e1acb9b11cd\") " pod="opendatahub/opendatahub-operator-controller-manager-7d7c89d5b6-722r9" Jul 21 01:41:00.003705 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:00.003512 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/60d07d21-eb68-49ee-b399-5e1acb9b11cd-tmp\") pod \"opendatahub-operator-controller-manager-7d7c89d5b6-722r9\" (UID: \"60d07d21-eb68-49ee-b399-5e1acb9b11cd\") " pod="opendatahub/opendatahub-operator-controller-manager-7d7c89d5b6-722r9" Jul 21 01:41:00.003705 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:00.003582 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhwvw\" (UniqueName: \"kubernetes.io/projected/60d07d21-eb68-49ee-b399-5e1acb9b11cd-kube-api-access-xhwvw\") pod \"opendatahub-operator-controller-manager-7d7c89d5b6-722r9\" (UID: \"60d07d21-eb68-49ee-b399-5e1acb9b11cd\") " pod="opendatahub/opendatahub-operator-controller-manager-7d7c89d5b6-722r9" Jul 21 01:41:00.003705 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:00.003608 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/60d07d21-eb68-49ee-b399-5e1acb9b11cd-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d7c89d5b6-722r9\" (UID: \"60d07d21-eb68-49ee-b399-5e1acb9b11cd\") " pod="opendatahub/opendatahub-operator-controller-manager-7d7c89d5b6-722r9" Jul 21 01:41:00.003705 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:00.003633 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/60d07d21-eb68-49ee-b399-5e1acb9b11cd-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d7c89d5b6-722r9\" (UID: \"60d07d21-eb68-49ee-b399-5e1acb9b11cd\") " pod="opendatahub/opendatahub-operator-controller-manager-7d7c89d5b6-722r9" Jul 21 01:41:00.105132 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:00.105036 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/60d07d21-eb68-49ee-b399-5e1acb9b11cd-manifests\") pod \"opendatahub-operator-controller-manager-7d7c89d5b6-722r9\" (UID: \"60d07d21-eb68-49ee-b399-5e1acb9b11cd\") " pod="opendatahub/opendatahub-operator-controller-manager-7d7c89d5b6-722r9" Jul 21 01:41:00.105132 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:00.105072 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/60d07d21-eb68-49ee-b399-5e1acb9b11cd-tmp\") pod \"opendatahub-operator-controller-manager-7d7c89d5b6-722r9\" (UID: \"60d07d21-eb68-49ee-b399-5e1acb9b11cd\") " pod="opendatahub/opendatahub-operator-controller-manager-7d7c89d5b6-722r9" Jul 21 01:41:00.105132 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:00.105113 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhwvw\" (UniqueName: \"kubernetes.io/projected/60d07d21-eb68-49ee-b399-5e1acb9b11cd-kube-api-access-xhwvw\") pod \"opendatahub-operator-controller-manager-7d7c89d5b6-722r9\" (UID: \"60d07d21-eb68-49ee-b399-5e1acb9b11cd\") " pod="opendatahub/opendatahub-operator-controller-manager-7d7c89d5b6-722r9" Jul 21 01:41:00.105418 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:00.105137 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/60d07d21-eb68-49ee-b399-5e1acb9b11cd-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d7c89d5b6-722r9\" (UID: \"60d07d21-eb68-49ee-b399-5e1acb9b11cd\") " pod="opendatahub/opendatahub-operator-controller-manager-7d7c89d5b6-722r9" Jul 21 01:41:00.105418 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:00.105163 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/60d07d21-eb68-49ee-b399-5e1acb9b11cd-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d7c89d5b6-722r9\" (UID: \"60d07d21-eb68-49ee-b399-5e1acb9b11cd\") " pod="opendatahub/opendatahub-operator-controller-manager-7d7c89d5b6-722r9" Jul 21 01:41:00.105583 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:00.105558 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/60d07d21-eb68-49ee-b399-5e1acb9b11cd-manifests\") pod \"opendatahub-operator-controller-manager-7d7c89d5b6-722r9\" (UID: \"60d07d21-eb68-49ee-b399-5e1acb9b11cd\") " pod="opendatahub/opendatahub-operator-controller-manager-7d7c89d5b6-722r9" Jul 21 01:41:00.107595 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:00.107566 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/60d07d21-eb68-49ee-b399-5e1acb9b11cd-tmp\") pod \"opendatahub-operator-controller-manager-7d7c89d5b6-722r9\" (UID: \"60d07d21-eb68-49ee-b399-5e1acb9b11cd\") " pod="opendatahub/opendatahub-operator-controller-manager-7d7c89d5b6-722r9" Jul 21 01:41:00.107695 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:00.107615 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/60d07d21-eb68-49ee-b399-5e1acb9b11cd-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d7c89d5b6-722r9\" (UID: \"60d07d21-eb68-49ee-b399-5e1acb9b11cd\") " pod="opendatahub/opendatahub-operator-controller-manager-7d7c89d5b6-722r9" Jul 21 01:41:00.107888 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:00.107866 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/60d07d21-eb68-49ee-b399-5e1acb9b11cd-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d7c89d5b6-722r9\" (UID: \"60d07d21-eb68-49ee-b399-5e1acb9b11cd\") " pod="opendatahub/opendatahub-operator-controller-manager-7d7c89d5b6-722r9" Jul 21 01:41:00.113493 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:00.113422 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhwvw\" (UniqueName: \"kubernetes.io/projected/60d07d21-eb68-49ee-b399-5e1acb9b11cd-kube-api-access-xhwvw\") pod \"opendatahub-operator-controller-manager-7d7c89d5b6-722r9\" (UID: \"60d07d21-eb68-49ee-b399-5e1acb9b11cd\") " pod="opendatahub/opendatahub-operator-controller-manager-7d7c89d5b6-722r9" Jul 21 01:41:00.167873 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:00.167837 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7d7c89d5b6-722r9" Jul 21 01:41:00.309578 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:00.309552 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7d7c89d5b6-722r9"] Jul 21 01:41:00.312542 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:41:00.312511 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60d07d21_eb68_49ee_b399_5e1acb9b11cd.slice/crio-6c9f673cf75cfa850b043808919009e1b314fb5ea48438c7d54a7af102d104e7 WatchSource:0}: Error finding container 6c9f673cf75cfa850b043808919009e1b314fb5ea48438c7d54a7af102d104e7: Status 404 returned error can't find the container with id 6c9f673cf75cfa850b043808919009e1b314fb5ea48438c7d54a7af102d104e7 Jul 21 01:41:00.323832 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:00.323805 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7d7c89d5b6-722r9" event={"ID":"60d07d21-eb68-49ee-b399-5e1acb9b11cd","Type":"ContainerStarted","Data":"6c9f673cf75cfa850b043808919009e1b314fb5ea48438c7d54a7af102d104e7"} Jul 21 01:41:04.345707 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:04.345663 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7d7c89d5b6-722r9" event={"ID":"60d07d21-eb68-49ee-b399-5e1acb9b11cd","Type":"ContainerStarted","Data":"9a93677ca73cb6acee397bc95c00cb88715dfd40888e3bb4f8aac8299b68aefe"} Jul 21 01:41:05.351932 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:05.351892 2580 generic.go:356] "Generic (PLEG): container finished" podID="60d07d21-eb68-49ee-b399-5e1acb9b11cd" containerID="9a93677ca73cb6acee397bc95c00cb88715dfd40888e3bb4f8aac8299b68aefe" exitCode=0 Jul 21 01:41:05.352413 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:05.351939 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7d7c89d5b6-722r9" event={"ID":"60d07d21-eb68-49ee-b399-5e1acb9b11cd","Type":"ContainerDied","Data":"9a93677ca73cb6acee397bc95c00cb88715dfd40888e3bb4f8aac8299b68aefe"} Jul 21 01:41:06.363346 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:06.363308 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7d7c89d5b6-722r9" event={"ID":"60d07d21-eb68-49ee-b399-5e1acb9b11cd","Type":"ContainerStarted","Data":"cb760dc856005e8fbbb7168aeae0402925ba7903c1a3b15ea09f79170735fd59"} Jul 21 01:41:06.363801 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:06.363392 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7d7c89d5b6-722r9" Jul 21 01:41:06.384853 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:06.384802 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7d7c89d5b6-722r9" podStartSLOduration=2.913344913 podStartE2EDuration="7.384786869s" totalImagesPullingTime="3.298140105s" totalInitContainerRuntime="1.173301851s" isStatefulPod=false podCreationTimestamp="2026-07-21 01:40:59 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:41:06.382366918 +0000 UTC m=+289.969532923" watchObservedRunningTime="2026-07-21 01:41:06.384786869 +0000 UTC m=+289.971952848" Jul 21 01:41:16.756527 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:16.756495 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-9ctn2_1e930d67-d711-419c-af02-86b43027a95d/cluster-monitoring-operator/0.log" Jul 21 01:41:16.761441 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:16.761415 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-9ctn2_1e930d67-d711-419c-af02-86b43027a95d/cluster-monitoring-operator/0.log" Jul 21 01:41:16.762329 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:16.762307 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-4mtp7_9c380009-9bf5-4b73-803a-5a1ea9c1abf0/console-operator/1.log" Jul 21 01:41:16.767389 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:16.767363 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-4mtp7_9c380009-9bf5-4b73-803a-5a1ea9c1abf0/console-operator/1.log" Jul 21 01:41:16.774127 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:16.774106 2580 kubelet.go:1648] "Image garbage collection succeeded" Jul 21 01:41:17.370583 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:17.370558 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7d7c89d5b6-722r9" Jul 21 01:41:19.718006 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:19.717970 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5cc474c686-6vvdm"] Jul 21 01:41:19.725617 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:19.725582 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5cc474c686-6vvdm" Jul 21 01:41:19.728302 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:19.728277 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 21 01:41:19.728428 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:19.728312 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 21 01:41:19.737361 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:19.737333 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5cc474c686-6vvdm"] Jul 21 01:41:19.781033 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:19.781000 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gkhf\" (UniqueName: \"kubernetes.io/projected/f478270d-44e8-4bb0-bec0-fda93118abb3-kube-api-access-5gkhf\") pod \"kube-auth-proxy-5cc474c686-6vvdm\" (UID: \"f478270d-44e8-4bb0-bec0-fda93118abb3\") " pod="openshift-ingress/kube-auth-proxy-5cc474c686-6vvdm" Jul 21 01:41:19.781241 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:19.781060 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f478270d-44e8-4bb0-bec0-fda93118abb3-tmp\") pod \"kube-auth-proxy-5cc474c686-6vvdm\" (UID: \"f478270d-44e8-4bb0-bec0-fda93118abb3\") " pod="openshift-ingress/kube-auth-proxy-5cc474c686-6vvdm" Jul 21 01:41:19.781241 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:19.781132 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f478270d-44e8-4bb0-bec0-fda93118abb3-tls-certs\") pod \"kube-auth-proxy-5cc474c686-6vvdm\" (UID: \"f478270d-44e8-4bb0-bec0-fda93118abb3\") " pod="openshift-ingress/kube-auth-proxy-5cc474c686-6vvdm" Jul 21 01:41:19.882029 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:19.881999 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f478270d-44e8-4bb0-bec0-fda93118abb3-tls-certs\") pod \"kube-auth-proxy-5cc474c686-6vvdm\" (UID: \"f478270d-44e8-4bb0-bec0-fda93118abb3\") " pod="openshift-ingress/kube-auth-proxy-5cc474c686-6vvdm" Jul 21 01:41:19.882202 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:19.882048 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gkhf\" (UniqueName: \"kubernetes.io/projected/f478270d-44e8-4bb0-bec0-fda93118abb3-kube-api-access-5gkhf\") pod \"kube-auth-proxy-5cc474c686-6vvdm\" (UID: \"f478270d-44e8-4bb0-bec0-fda93118abb3\") " pod="openshift-ingress/kube-auth-proxy-5cc474c686-6vvdm" Jul 21 01:41:19.882202 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:19.882126 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f478270d-44e8-4bb0-bec0-fda93118abb3-tmp\") pod \"kube-auth-proxy-5cc474c686-6vvdm\" (UID: \"f478270d-44e8-4bb0-bec0-fda93118abb3\") " pod="openshift-ingress/kube-auth-proxy-5cc474c686-6vvdm" Jul 21 01:41:19.884361 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:19.884330 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f478270d-44e8-4bb0-bec0-fda93118abb3-tmp\") pod \"kube-auth-proxy-5cc474c686-6vvdm\" (UID: \"f478270d-44e8-4bb0-bec0-fda93118abb3\") " pod="openshift-ingress/kube-auth-proxy-5cc474c686-6vvdm" Jul 21 01:41:19.884501 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:19.884404 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f478270d-44e8-4bb0-bec0-fda93118abb3-tls-certs\") pod \"kube-auth-proxy-5cc474c686-6vvdm\" (UID: \"f478270d-44e8-4bb0-bec0-fda93118abb3\") " pod="openshift-ingress/kube-auth-proxy-5cc474c686-6vvdm" Jul 21 01:41:19.894079 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:19.894048 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gkhf\" (UniqueName: \"kubernetes.io/projected/f478270d-44e8-4bb0-bec0-fda93118abb3-kube-api-access-5gkhf\") pod \"kube-auth-proxy-5cc474c686-6vvdm\" (UID: \"f478270d-44e8-4bb0-bec0-fda93118abb3\") " pod="openshift-ingress/kube-auth-proxy-5cc474c686-6vvdm" Jul 21 01:41:20.035192 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:20.035152 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5cc474c686-6vvdm" Jul 21 01:41:20.084345 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:20.084311 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-bb4gc"] Jul 21 01:41:20.088224 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:20.088198 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-bb4gc" Jul 21 01:41:20.091648 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:20.091588 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 21 01:41:20.091903 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:20.091885 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-kptkb\"" Jul 21 01:41:20.104940 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:20.104747 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-bb4gc"] Jul 21 01:41:20.185460 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:20.185419 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zs97n\" (UniqueName: \"kubernetes.io/projected/6a7381ef-2c27-451a-9516-fab7f9c42e87-kube-api-access-zs97n\") pod \"odh-model-controller-7998c67dc4-bb4gc\" (UID: \"6a7381ef-2c27-451a-9516-fab7f9c42e87\") " pod="opendatahub/odh-model-controller-7998c67dc4-bb4gc" Jul 21 01:41:20.185638 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:20.185488 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6a7381ef-2c27-451a-9516-fab7f9c42e87-cert\") pod \"odh-model-controller-7998c67dc4-bb4gc\" (UID: \"6a7381ef-2c27-451a-9516-fab7f9c42e87\") " pod="opendatahub/odh-model-controller-7998c67dc4-bb4gc" Jul 21 01:41:20.189288 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:20.189260 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5cc474c686-6vvdm"] Jul 21 01:41:20.192804 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:41:20.192775 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf478270d_44e8_4bb0_bec0_fda93118abb3.slice/crio-3630af1755ada2301958b105e24404d4fe8500ef77b2e4e498fb38f6f7b95a6e WatchSource:0}: Error finding container 3630af1755ada2301958b105e24404d4fe8500ef77b2e4e498fb38f6f7b95a6e: Status 404 returned error can't find the container with id 3630af1755ada2301958b105e24404d4fe8500ef77b2e4e498fb38f6f7b95a6e Jul 21 01:41:20.194831 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:20.194812 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 01:41:20.286978 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:20.286889 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zs97n\" (UniqueName: \"kubernetes.io/projected/6a7381ef-2c27-451a-9516-fab7f9c42e87-kube-api-access-zs97n\") pod \"odh-model-controller-7998c67dc4-bb4gc\" (UID: \"6a7381ef-2c27-451a-9516-fab7f9c42e87\") " pod="opendatahub/odh-model-controller-7998c67dc4-bb4gc" Jul 21 01:41:20.286978 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:20.286942 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6a7381ef-2c27-451a-9516-fab7f9c42e87-cert\") pod \"odh-model-controller-7998c67dc4-bb4gc\" (UID: \"6a7381ef-2c27-451a-9516-fab7f9c42e87\") " pod="opendatahub/odh-model-controller-7998c67dc4-bb4gc" Jul 21 01:41:20.287174 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:41:20.287044 2580 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 21 01:41:20.287174 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:41:20.287098 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a7381ef-2c27-451a-9516-fab7f9c42e87-cert podName:6a7381ef-2c27-451a-9516-fab7f9c42e87 nodeName:}" failed. No retries permitted until 2026-07-21 01:41:20.787081856 +0000 UTC m=+304.374247812 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/6a7381ef-2c27-451a-9516-fab7f9c42e87-cert") pod "odh-model-controller-7998c67dc4-bb4gc" (UID: "6a7381ef-2c27-451a-9516-fab7f9c42e87") : secret "odh-model-controller-webhook-cert" not found Jul 21 01:41:20.298127 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:20.298100 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zs97n\" (UniqueName: \"kubernetes.io/projected/6a7381ef-2c27-451a-9516-fab7f9c42e87-kube-api-access-zs97n\") pod \"odh-model-controller-7998c67dc4-bb4gc\" (UID: \"6a7381ef-2c27-451a-9516-fab7f9c42e87\") " pod="opendatahub/odh-model-controller-7998c67dc4-bb4gc" Jul 21 01:41:20.434021 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:20.433984 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5cc474c686-6vvdm" event={"ID":"f478270d-44e8-4bb0-bec0-fda93118abb3","Type":"ContainerStarted","Data":"3630af1755ada2301958b105e24404d4fe8500ef77b2e4e498fb38f6f7b95a6e"} Jul 21 01:41:20.791408 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:20.791374 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6a7381ef-2c27-451a-9516-fab7f9c42e87-cert\") pod \"odh-model-controller-7998c67dc4-bb4gc\" (UID: \"6a7381ef-2c27-451a-9516-fab7f9c42e87\") " pod="opendatahub/odh-model-controller-7998c67dc4-bb4gc" Jul 21 01:41:20.793888 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:20.793859 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6a7381ef-2c27-451a-9516-fab7f9c42e87-cert\") pod \"odh-model-controller-7998c67dc4-bb4gc\" (UID: \"6a7381ef-2c27-451a-9516-fab7f9c42e87\") " pod="opendatahub/odh-model-controller-7998c67dc4-bb4gc" Jul 21 01:41:21.002599 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:21.002558 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-bb4gc" Jul 21 01:41:21.013581 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:21.011953 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-8dt74"] Jul 21 01:41:21.015688 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:21.015661 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-8dt74" Jul 21 01:41:21.018065 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:21.018022 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 21 01:41:21.032185 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:21.032008 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-8dt74"] Jul 21 01:41:21.093352 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:21.093322 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e079cc2b-d540-422b-a89c-f59f6c812680-tls-certs\") pod \"model-serving-api-54d89bd5fc-8dt74\" (UID: \"e079cc2b-d540-422b-a89c-f59f6c812680\") " pod="opendatahub/model-serving-api-54d89bd5fc-8dt74" Jul 21 01:41:21.093538 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:21.093374 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddnqq\" (UniqueName: \"kubernetes.io/projected/e079cc2b-d540-422b-a89c-f59f6c812680-kube-api-access-ddnqq\") pod \"model-serving-api-54d89bd5fc-8dt74\" (UID: \"e079cc2b-d540-422b-a89c-f59f6c812680\") " pod="opendatahub/model-serving-api-54d89bd5fc-8dt74" Jul 21 01:41:21.185068 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:21.185026 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-bb4gc"] Jul 21 01:41:21.186634 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:41:21.186607 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a7381ef_2c27_451a_9516_fab7f9c42e87.slice/crio-9add89755117dd9de774c0ef341f41aecfabc9b3cb445b3dffa7d4976b3ab11f WatchSource:0}: Error finding container 9add89755117dd9de774c0ef341f41aecfabc9b3cb445b3dffa7d4976b3ab11f: Status 404 returned error can't find the container with id 9add89755117dd9de774c0ef341f41aecfabc9b3cb445b3dffa7d4976b3ab11f Jul 21 01:41:21.194610 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:21.194580 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddnqq\" (UniqueName: \"kubernetes.io/projected/e079cc2b-d540-422b-a89c-f59f6c812680-kube-api-access-ddnqq\") pod \"model-serving-api-54d89bd5fc-8dt74\" (UID: \"e079cc2b-d540-422b-a89c-f59f6c812680\") " pod="opendatahub/model-serving-api-54d89bd5fc-8dt74" Jul 21 01:41:21.194718 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:21.194668 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e079cc2b-d540-422b-a89c-f59f6c812680-tls-certs\") pod \"model-serving-api-54d89bd5fc-8dt74\" (UID: \"e079cc2b-d540-422b-a89c-f59f6c812680\") " pod="opendatahub/model-serving-api-54d89bd5fc-8dt74" Jul 21 01:41:21.197822 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:21.197800 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e079cc2b-d540-422b-a89c-f59f6c812680-tls-certs\") pod \"model-serving-api-54d89bd5fc-8dt74\" (UID: \"e079cc2b-d540-422b-a89c-f59f6c812680\") " pod="opendatahub/model-serving-api-54d89bd5fc-8dt74" Jul 21 01:41:21.218233 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:21.218195 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddnqq\" (UniqueName: \"kubernetes.io/projected/e079cc2b-d540-422b-a89c-f59f6c812680-kube-api-access-ddnqq\") pod \"model-serving-api-54d89bd5fc-8dt74\" (UID: \"e079cc2b-d540-422b-a89c-f59f6c812680\") " pod="opendatahub/model-serving-api-54d89bd5fc-8dt74" Jul 21 01:41:21.304219 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:21.304132 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-z9jmt"] Jul 21 01:41:21.308431 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:21.308409 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-z9jmt" Jul 21 01:41:21.310945 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:21.310922 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 21 01:41:21.311058 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:21.310923 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 21 01:41:21.311058 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:21.310923 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 21 01:41:21.311058 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:21.310981 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-gkmq8\"" Jul 21 01:41:21.311058 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:21.310969 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 21 01:41:21.319158 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:21.319137 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-z9jmt"] Jul 21 01:41:21.353224 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:21.353180 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-8dt74" Jul 21 01:41:21.396569 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:21.396529 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a171a7ed-c69a-4b4f-939c-8181e00287ba-istio-token\") pod \"istiod-openshift-gateway-7574459656-z9jmt\" (UID: \"a171a7ed-c69a-4b4f-939c-8181e00287ba\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-z9jmt" Jul 21 01:41:21.396795 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:21.396584 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a171a7ed-c69a-4b4f-939c-8181e00287ba-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-z9jmt\" (UID: \"a171a7ed-c69a-4b4f-939c-8181e00287ba\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-z9jmt" Jul 21 01:41:21.396795 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:21.396625 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ssnh\" (UniqueName: \"kubernetes.io/projected/a171a7ed-c69a-4b4f-939c-8181e00287ba-kube-api-access-9ssnh\") pod \"istiod-openshift-gateway-7574459656-z9jmt\" (UID: \"a171a7ed-c69a-4b4f-939c-8181e00287ba\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-z9jmt" Jul 21 01:41:21.396795 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:21.396698 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a171a7ed-c69a-4b4f-939c-8181e00287ba-local-certs\") pod \"istiod-openshift-gateway-7574459656-z9jmt\" (UID: \"a171a7ed-c69a-4b4f-939c-8181e00287ba\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-z9jmt" Jul 21 01:41:21.396795 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:21.396726 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a171a7ed-c69a-4b4f-939c-8181e00287ba-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-z9jmt\" (UID: \"a171a7ed-c69a-4b4f-939c-8181e00287ba\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-z9jmt" Jul 21 01:41:21.396795 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:21.396757 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a171a7ed-c69a-4b4f-939c-8181e00287ba-cacerts\") pod \"istiod-openshift-gateway-7574459656-z9jmt\" (UID: \"a171a7ed-c69a-4b4f-939c-8181e00287ba\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-z9jmt" Jul 21 01:41:21.397161 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:21.396826 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a171a7ed-c69a-4b4f-939c-8181e00287ba-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-z9jmt\" (UID: \"a171a7ed-c69a-4b4f-939c-8181e00287ba\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-z9jmt" Jul 21 01:41:21.441835 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:21.441798 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-bb4gc" event={"ID":"6a7381ef-2c27-451a-9516-fab7f9c42e87","Type":"ContainerStarted","Data":"9add89755117dd9de774c0ef341f41aecfabc9b3cb445b3dffa7d4976b3ab11f"} Jul 21 01:41:21.498078 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:21.498035 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a171a7ed-c69a-4b4f-939c-8181e00287ba-local-certs\") pod \"istiod-openshift-gateway-7574459656-z9jmt\" (UID: \"a171a7ed-c69a-4b4f-939c-8181e00287ba\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-z9jmt" Jul 21 01:41:21.498078 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:21.498072 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a171a7ed-c69a-4b4f-939c-8181e00287ba-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-z9jmt\" (UID: \"a171a7ed-c69a-4b4f-939c-8181e00287ba\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-z9jmt" Jul 21 01:41:21.498318 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:21.498095 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a171a7ed-c69a-4b4f-939c-8181e00287ba-cacerts\") pod \"istiod-openshift-gateway-7574459656-z9jmt\" (UID: \"a171a7ed-c69a-4b4f-939c-8181e00287ba\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-z9jmt" Jul 21 01:41:21.498318 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:21.498145 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a171a7ed-c69a-4b4f-939c-8181e00287ba-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-z9jmt\" (UID: \"a171a7ed-c69a-4b4f-939c-8181e00287ba\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-z9jmt" Jul 21 01:41:21.498318 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:21.498172 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a171a7ed-c69a-4b4f-939c-8181e00287ba-istio-token\") pod \"istiod-openshift-gateway-7574459656-z9jmt\" (UID: \"a171a7ed-c69a-4b4f-939c-8181e00287ba\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-z9jmt" Jul 21 01:41:21.498318 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:21.498191 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a171a7ed-c69a-4b4f-939c-8181e00287ba-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-z9jmt\" (UID: \"a171a7ed-c69a-4b4f-939c-8181e00287ba\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-z9jmt" Jul 21 01:41:21.498318 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:21.498222 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ssnh\" (UniqueName: \"kubernetes.io/projected/a171a7ed-c69a-4b4f-939c-8181e00287ba-kube-api-access-9ssnh\") pod \"istiod-openshift-gateway-7574459656-z9jmt\" (UID: \"a171a7ed-c69a-4b4f-939c-8181e00287ba\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-z9jmt" Jul 21 01:41:21.498917 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:21.498849 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a171a7ed-c69a-4b4f-939c-8181e00287ba-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-z9jmt\" (UID: \"a171a7ed-c69a-4b4f-939c-8181e00287ba\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-z9jmt" Jul 21 01:41:21.500999 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:21.500972 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a171a7ed-c69a-4b4f-939c-8181e00287ba-local-certs\") pod \"istiod-openshift-gateway-7574459656-z9jmt\" (UID: \"a171a7ed-c69a-4b4f-939c-8181e00287ba\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-z9jmt" Jul 21 01:41:21.501203 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:21.501186 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a171a7ed-c69a-4b4f-939c-8181e00287ba-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-z9jmt\" (UID: \"a171a7ed-c69a-4b4f-939c-8181e00287ba\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-z9jmt" Jul 21 01:41:21.501245 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:21.501196 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a171a7ed-c69a-4b4f-939c-8181e00287ba-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-z9jmt\" (UID: \"a171a7ed-c69a-4b4f-939c-8181e00287ba\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-z9jmt" Jul 21 01:41:21.501279 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:21.501251 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a171a7ed-c69a-4b4f-939c-8181e00287ba-cacerts\") pod \"istiod-openshift-gateway-7574459656-z9jmt\" (UID: \"a171a7ed-c69a-4b4f-939c-8181e00287ba\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-z9jmt" Jul 21 01:41:21.506791 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:21.506764 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-8dt74"] Jul 21 01:41:21.508731 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:21.508712 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a171a7ed-c69a-4b4f-939c-8181e00287ba-istio-token\") pod \"istiod-openshift-gateway-7574459656-z9jmt\" (UID: \"a171a7ed-c69a-4b4f-939c-8181e00287ba\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-z9jmt" Jul 21 01:41:21.509302 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:21.509281 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ssnh\" (UniqueName: \"kubernetes.io/projected/a171a7ed-c69a-4b4f-939c-8181e00287ba-kube-api-access-9ssnh\") pod \"istiod-openshift-gateway-7574459656-z9jmt\" (UID: \"a171a7ed-c69a-4b4f-939c-8181e00287ba\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-z9jmt" Jul 21 01:41:21.509593 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:41:21.509573 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode079cc2b_d540_422b_a89c_f59f6c812680.slice/crio-8f6a09ca676d73a92f231a6231c55285a22d9a4d621235a16851dd5ec7ceb819 WatchSource:0}: Error finding container 8f6a09ca676d73a92f231a6231c55285a22d9a4d621235a16851dd5ec7ceb819: Status 404 returned error can't find the container with id 8f6a09ca676d73a92f231a6231c55285a22d9a4d621235a16851dd5ec7ceb819 Jul 21 01:41:21.618185 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:21.618104 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-z9jmt" Jul 21 01:41:21.767690 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:21.767658 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-z9jmt"] Jul 21 01:41:21.769849 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:41:21.769812 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda171a7ed_c69a_4b4f_939c_8181e00287ba.slice/crio-4889926b7aabdbe6a335d8059620f02f6f3f1f5326e67174b9213fb59e9f50b3 WatchSource:0}: Error finding container 4889926b7aabdbe6a335d8059620f02f6f3f1f5326e67174b9213fb59e9f50b3: Status 404 returned error can't find the container with id 4889926b7aabdbe6a335d8059620f02f6f3f1f5326e67174b9213fb59e9f50b3 Jul 21 01:41:22.452746 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:22.452654 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-8dt74" event={"ID":"e079cc2b-d540-422b-a89c-f59f6c812680","Type":"ContainerStarted","Data":"8f6a09ca676d73a92f231a6231c55285a22d9a4d621235a16851dd5ec7ceb819"} Jul 21 01:41:22.455711 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:22.455591 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-z9jmt" event={"ID":"a171a7ed-c69a-4b4f-939c-8181e00287ba","Type":"ContainerStarted","Data":"4889926b7aabdbe6a335d8059620f02f6f3f1f5326e67174b9213fb59e9f50b3"} Jul 21 01:41:26.104650 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:26.104614 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 01:41:26.105054 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:26.104681 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 01:41:26.493264 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:26.493167 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-8dt74" event={"ID":"e079cc2b-d540-422b-a89c-f59f6c812680","Type":"ContainerStarted","Data":"6023d6c26a59a96a2c116f7e43c81feaa6a1d0651682931d1f736c83cfaf7be5"} Jul 21 01:41:26.493432 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:26.493290 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-8dt74" Jul 21 01:41:26.494832 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:26.494802 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5cc474c686-6vvdm" event={"ID":"f478270d-44e8-4bb0-bec0-fda93118abb3","Type":"ContainerStarted","Data":"2f1c884120c6c8333a2ca91f7014e44fb066c4f6cdd48139dae1225ff58ecb01"} Jul 21 01:41:26.496436 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:26.496386 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-z9jmt" event={"ID":"a171a7ed-c69a-4b4f-939c-8181e00287ba","Type":"ContainerStarted","Data":"b11aac14992327b67f13f98fdf594eb52fa04557167d9b4be4f3f92f7824e939"} Jul 21 01:41:26.496557 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:26.496439 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-z9jmt" Jul 21 01:41:26.497903 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:26.497881 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-bb4gc" event={"ID":"6a7381ef-2c27-451a-9516-fab7f9c42e87","Type":"ContainerStarted","Data":"73b6ca6730136852f0a6e39a4420af1bf9a6de7a922b643746fd43b20b15ce1c"} Jul 21 01:41:26.497997 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:26.497973 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-bb4gc" Jul 21 01:41:26.509866 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:26.509818 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-8dt74" podStartSLOduration=1.981622714 podStartE2EDuration="6.509805094s" totalImagesPullingTime="4.52818238s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 01:41:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:41:26.507993817 +0000 UTC m=+310.095159826" watchObservedRunningTime="2026-07-21 01:41:26.509805094 +0000 UTC m=+310.096971071" Jul 21 01:41:26.522840 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:26.522796 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5cc474c686-6vvdm" podStartSLOduration=1.682227887 podStartE2EDuration="7.522780659s" totalImagesPullingTime="5.840552772s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 01:41:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:41:26.5217153 +0000 UTC m=+310.108881299" watchObservedRunningTime="2026-07-21 01:41:26.522780659 +0000 UTC m=+310.109946637" Jul 21 01:41:26.542549 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:26.539824 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-z9jmt" podStartSLOduration=1.208103345 podStartE2EDuration="5.539802047s" totalImagesPullingTime="4.331698702s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 01:41:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:41:26.538514913 +0000 UTC m=+310.125680917" watchObservedRunningTime="2026-07-21 01:41:26.539802047 +0000 UTC m=+310.126968026" Jul 21 01:41:26.555480 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:26.554848 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-bb4gc" podStartSLOduration=1.692370177 podStartE2EDuration="6.55482908s" totalImagesPullingTime="4.862458903s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 01:41:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:41:26.55437641 +0000 UTC m=+310.141542411" watchObservedRunningTime="2026-07-21 01:41:26.55482908 +0000 UTC m=+310.141995062" Jul 21 01:41:27.505380 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:27.505355 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-z9jmt" Jul 21 01:41:31.382555 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:31.382523 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-pkkpq"] Jul 21 01:41:31.386497 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:31.386474 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pkkpq" Jul 21 01:41:31.388730 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:31.388705 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-pgxz4"] Jul 21 01:41:31.389474 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:31.389434 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 21 01:41:31.389580 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:31.389471 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-zj68r\"" Jul 21 01:41:31.392429 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:31.392287 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pgxz4" Jul 21 01:41:31.395300 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:31.395282 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 21 01:41:31.395428 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:31.395411 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-wf462\"" Jul 21 01:41:31.398623 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:31.398599 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-pkkpq"] Jul 21 01:41:31.405540 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:31.405517 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-pgxz4"] Jul 21 01:41:31.495538 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:31.495484 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db4zg\" (UniqueName: \"kubernetes.io/projected/a5b32c0e-f65a-48ba-a752-1b78cb1950e3-kube-api-access-db4zg\") pod \"llmisvc-controller-manager-777c6bd7d5-pgxz4\" (UID: \"a5b32c0e-f65a-48ba-a752-1b78cb1950e3\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pgxz4" Jul 21 01:41:31.495708 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:31.495582 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a5b32c0e-f65a-48ba-a752-1b78cb1950e3-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-pgxz4\" (UID: \"a5b32c0e-f65a-48ba-a752-1b78cb1950e3\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pgxz4" Jul 21 01:41:31.495708 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:31.495608 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e604ba86-e513-4b2c-88e4-0e7b90974b72-cert\") pod \"kserve-controller-manager-d6f4b65b8-pkkpq\" (UID: \"e604ba86-e513-4b2c-88e4-0e7b90974b72\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pkkpq" Jul 21 01:41:31.495780 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:31.495708 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lzqg\" (UniqueName: \"kubernetes.io/projected/e604ba86-e513-4b2c-88e4-0e7b90974b72-kube-api-access-7lzqg\") pod \"kserve-controller-manager-d6f4b65b8-pkkpq\" (UID: \"e604ba86-e513-4b2c-88e4-0e7b90974b72\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pkkpq" Jul 21 01:41:31.596279 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:31.596248 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a5b32c0e-f65a-48ba-a752-1b78cb1950e3-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-pgxz4\" (UID: \"a5b32c0e-f65a-48ba-a752-1b78cb1950e3\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pgxz4" Jul 21 01:41:31.596511 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:31.596294 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e604ba86-e513-4b2c-88e4-0e7b90974b72-cert\") pod \"kserve-controller-manager-d6f4b65b8-pkkpq\" (UID: \"e604ba86-e513-4b2c-88e4-0e7b90974b72\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pkkpq" Jul 21 01:41:31.596511 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:31.596357 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lzqg\" (UniqueName: \"kubernetes.io/projected/e604ba86-e513-4b2c-88e4-0e7b90974b72-kube-api-access-7lzqg\") pod \"kserve-controller-manager-d6f4b65b8-pkkpq\" (UID: \"e604ba86-e513-4b2c-88e4-0e7b90974b72\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pkkpq" Jul 21 01:41:31.596511 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:41:31.596424 2580 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 21 01:41:31.596701 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:41:31.596526 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5b32c0e-f65a-48ba-a752-1b78cb1950e3-cert podName:a5b32c0e-f65a-48ba-a752-1b78cb1950e3 nodeName:}" failed. No retries permitted until 2026-07-21 01:41:32.096505638 +0000 UTC m=+315.683671608 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/a5b32c0e-f65a-48ba-a752-1b78cb1950e3-cert") pod "llmisvc-controller-manager-777c6bd7d5-pgxz4" (UID: "a5b32c0e-f65a-48ba-a752-1b78cb1950e3") : secret "llmisvc-webhook-server-cert" not found Jul 21 01:41:31.596701 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:31.596432 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db4zg\" (UniqueName: \"kubernetes.io/projected/a5b32c0e-f65a-48ba-a752-1b78cb1950e3-kube-api-access-db4zg\") pod \"llmisvc-controller-manager-777c6bd7d5-pgxz4\" (UID: \"a5b32c0e-f65a-48ba-a752-1b78cb1950e3\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pgxz4" Jul 21 01:41:31.599098 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:31.599074 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e604ba86-e513-4b2c-88e4-0e7b90974b72-cert\") pod \"kserve-controller-manager-d6f4b65b8-pkkpq\" (UID: \"e604ba86-e513-4b2c-88e4-0e7b90974b72\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pkkpq" Jul 21 01:41:31.624726 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:31.624689 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lzqg\" (UniqueName: \"kubernetes.io/projected/e604ba86-e513-4b2c-88e4-0e7b90974b72-kube-api-access-7lzqg\") pod \"kserve-controller-manager-d6f4b65b8-pkkpq\" (UID: \"e604ba86-e513-4b2c-88e4-0e7b90974b72\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pkkpq" Jul 21 01:41:31.625253 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:31.625235 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-db4zg\" (UniqueName: \"kubernetes.io/projected/a5b32c0e-f65a-48ba-a752-1b78cb1950e3-kube-api-access-db4zg\") pod \"llmisvc-controller-manager-777c6bd7d5-pgxz4\" (UID: \"a5b32c0e-f65a-48ba-a752-1b78cb1950e3\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pgxz4" Jul 21 01:41:31.700659 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:31.700567 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pkkpq" Jul 21 01:41:31.876855 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:31.876822 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-pkkpq"] Jul 21 01:41:31.881806 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:41:31.881778 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode604ba86_e513_4b2c_88e4_0e7b90974b72.slice/crio-19e8e3e097121effe11652fd0c8c08e094488a47a2740f9453566e1d8f1e9164 WatchSource:0}: Error finding container 19e8e3e097121effe11652fd0c8c08e094488a47a2740f9453566e1d8f1e9164: Status 404 returned error can't find the container with id 19e8e3e097121effe11652fd0c8c08e094488a47a2740f9453566e1d8f1e9164 Jul 21 01:41:32.101936 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:32.101896 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a5b32c0e-f65a-48ba-a752-1b78cb1950e3-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-pgxz4\" (UID: \"a5b32c0e-f65a-48ba-a752-1b78cb1950e3\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pgxz4" Jul 21 01:41:32.104467 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:32.104435 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a5b32c0e-f65a-48ba-a752-1b78cb1950e3-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-pgxz4\" (UID: \"a5b32c0e-f65a-48ba-a752-1b78cb1950e3\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pgxz4" Jul 21 01:41:32.307633 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:32.307594 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pgxz4" Jul 21 01:41:32.465253 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:32.465219 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-pgxz4"] Jul 21 01:41:32.467637 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:41:32.467605 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda5b32c0e_f65a_48ba_a752_1b78cb1950e3.slice/crio-a24e0999f87dc078f606f1d7c507e72122fcdbd3897518e85f38494b3f845550 WatchSource:0}: Error finding container a24e0999f87dc078f606f1d7c507e72122fcdbd3897518e85f38494b3f845550: Status 404 returned error can't find the container with id a24e0999f87dc078f606f1d7c507e72122fcdbd3897518e85f38494b3f845550 Jul 21 01:41:32.529136 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:32.529097 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pkkpq" event={"ID":"e604ba86-e513-4b2c-88e4-0e7b90974b72","Type":"ContainerStarted","Data":"19e8e3e097121effe11652fd0c8c08e094488a47a2740f9453566e1d8f1e9164"} Jul 21 01:41:32.530174 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:32.530150 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pgxz4" event={"ID":"a5b32c0e-f65a-48ba-a752-1b78cb1950e3","Type":"ContainerStarted","Data":"a24e0999f87dc078f606f1d7c507e72122fcdbd3897518e85f38494b3f845550"} Jul 21 01:41:33.876380 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:33.876339 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-z9jmt"] Jul 21 01:41:33.876840 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:33.876558 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-z9jmt" podUID="a171a7ed-c69a-4b4f-939c-8181e00287ba" containerName="discovery" containerID="cri-o://b11aac14992327b67f13f98fdf594eb52fa04557167d9b4be4f3f92f7824e939" gracePeriod=30 Jul 21 01:41:34.544473 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:34.544321 2580 generic.go:356] "Generic (PLEG): container finished" podID="a171a7ed-c69a-4b4f-939c-8181e00287ba" containerID="b11aac14992327b67f13f98fdf594eb52fa04557167d9b4be4f3f92f7824e939" exitCode=0 Jul 21 01:41:34.544473 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:34.544391 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-z9jmt" event={"ID":"a171a7ed-c69a-4b4f-939c-8181e00287ba","Type":"ContainerDied","Data":"b11aac14992327b67f13f98fdf594eb52fa04557167d9b4be4f3f92f7824e939"} Jul 21 01:41:34.755849 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:34.755823 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-z9jmt" Jul 21 01:41:34.833001 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:34.832974 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9ssnh\" (UniqueName: \"kubernetes.io/projected/a171a7ed-c69a-4b4f-939c-8181e00287ba-kube-api-access-9ssnh\") pod \"a171a7ed-c69a-4b4f-939c-8181e00287ba\" (UID: \"a171a7ed-c69a-4b4f-939c-8181e00287ba\") " Jul 21 01:41:34.833180 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:34.833045 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a171a7ed-c69a-4b4f-939c-8181e00287ba-istio-csr-ca-configmap\") pod \"a171a7ed-c69a-4b4f-939c-8181e00287ba\" (UID: \"a171a7ed-c69a-4b4f-939c-8181e00287ba\") " Jul 21 01:41:34.833180 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:34.833106 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a171a7ed-c69a-4b4f-939c-8181e00287ba-istio-csr-dns-cert\") pod \"a171a7ed-c69a-4b4f-939c-8181e00287ba\" (UID: \"a171a7ed-c69a-4b4f-939c-8181e00287ba\") " Jul 21 01:41:34.833180 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:34.833141 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a171a7ed-c69a-4b4f-939c-8181e00287ba-local-certs\") pod \"a171a7ed-c69a-4b4f-939c-8181e00287ba\" (UID: \"a171a7ed-c69a-4b4f-939c-8181e00287ba\") " Jul 21 01:41:34.833180 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:34.833179 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a171a7ed-c69a-4b4f-939c-8181e00287ba-istio-token\") pod \"a171a7ed-c69a-4b4f-939c-8181e00287ba\" (UID: \"a171a7ed-c69a-4b4f-939c-8181e00287ba\") " Jul 21 01:41:34.833375 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:34.833207 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a171a7ed-c69a-4b4f-939c-8181e00287ba-istio-kubeconfig\") pod \"a171a7ed-c69a-4b4f-939c-8181e00287ba\" (UID: \"a171a7ed-c69a-4b4f-939c-8181e00287ba\") " Jul 21 01:41:34.833375 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:34.833238 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a171a7ed-c69a-4b4f-939c-8181e00287ba-cacerts\") pod \"a171a7ed-c69a-4b4f-939c-8181e00287ba\" (UID: \"a171a7ed-c69a-4b4f-939c-8181e00287ba\") " Jul 21 01:41:34.833375 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:34.833350 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a171a7ed-c69a-4b4f-939c-8181e00287ba-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "a171a7ed-c69a-4b4f-939c-8181e00287ba" (UID: "a171a7ed-c69a-4b4f-939c-8181e00287ba"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 01:41:34.833613 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:34.833594 2580 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a171a7ed-c69a-4b4f-939c-8181e00287ba-istio-csr-ca-configmap\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 21 01:41:34.835441 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:34.835413 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a171a7ed-c69a-4b4f-939c-8181e00287ba-local-certs" (OuterVolumeSpecName: "local-certs") pod "a171a7ed-c69a-4b4f-939c-8181e00287ba" (UID: "a171a7ed-c69a-4b4f-939c-8181e00287ba"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 01:41:34.835568 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:34.835505 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a171a7ed-c69a-4b4f-939c-8181e00287ba-cacerts" (OuterVolumeSpecName: "cacerts") pod "a171a7ed-c69a-4b4f-939c-8181e00287ba" (UID: "a171a7ed-c69a-4b4f-939c-8181e00287ba"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 01:41:34.835654 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:34.835629 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a171a7ed-c69a-4b4f-939c-8181e00287ba-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "a171a7ed-c69a-4b4f-939c-8181e00287ba" (UID: "a171a7ed-c69a-4b4f-939c-8181e00287ba"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 01:41:34.835731 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:34.835665 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a171a7ed-c69a-4b4f-939c-8181e00287ba-istio-token" (OuterVolumeSpecName: "istio-token") pod "a171a7ed-c69a-4b4f-939c-8181e00287ba" (UID: "a171a7ed-c69a-4b4f-939c-8181e00287ba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 01:41:34.835789 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:34.835756 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a171a7ed-c69a-4b4f-939c-8181e00287ba-kube-api-access-9ssnh" (OuterVolumeSpecName: "kube-api-access-9ssnh") pod "a171a7ed-c69a-4b4f-939c-8181e00287ba" (UID: "a171a7ed-c69a-4b4f-939c-8181e00287ba"). InnerVolumeSpecName "kube-api-access-9ssnh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 01:41:34.835941 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:34.835825 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a171a7ed-c69a-4b4f-939c-8181e00287ba-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "a171a7ed-c69a-4b4f-939c-8181e00287ba" (UID: "a171a7ed-c69a-4b4f-939c-8181e00287ba"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 01:41:34.934365 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:34.934329 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ssnh\" (UniqueName: \"kubernetes.io/projected/a171a7ed-c69a-4b4f-939c-8181e00287ba-kube-api-access-9ssnh\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 21 01:41:34.934365 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:34.934365 2580 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a171a7ed-c69a-4b4f-939c-8181e00287ba-istio-csr-dns-cert\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 21 01:41:34.934365 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:34.934376 2580 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a171a7ed-c69a-4b4f-939c-8181e00287ba-local-certs\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 21 01:41:34.934901 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:34.934385 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a171a7ed-c69a-4b4f-939c-8181e00287ba-istio-token\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 21 01:41:34.934901 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:34.934393 2580 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a171a7ed-c69a-4b4f-939c-8181e00287ba-istio-kubeconfig\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 21 01:41:34.934901 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:34.934400 2580 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a171a7ed-c69a-4b4f-939c-8181e00287ba-cacerts\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 21 01:41:35.551695 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:35.551673 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-z9jmt" Jul 21 01:41:35.551695 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:35.551678 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-z9jmt" event={"ID":"a171a7ed-c69a-4b4f-939c-8181e00287ba","Type":"ContainerDied","Data":"4889926b7aabdbe6a335d8059620f02f6f3f1f5326e67174b9213fb59e9f50b3"} Jul 21 01:41:35.551930 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:35.551741 2580 scope.go:117] "RemoveContainer" containerID="b11aac14992327b67f13f98fdf594eb52fa04557167d9b4be4f3f92f7824e939" Jul 21 01:41:35.553354 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:35.553319 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pkkpq" event={"ID":"e604ba86-e513-4b2c-88e4-0e7b90974b72","Type":"ContainerStarted","Data":"397e2e472386664e1edc990d9f5fd06d0716f4e0497a0f15b6a00a9635a29aea"} Jul 21 01:41:35.553501 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:35.553483 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pkkpq" Jul 21 01:41:35.568367 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:35.568343 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-z9jmt"] Jul 21 01:41:35.574381 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:35.574358 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-z9jmt"] Jul 21 01:41:35.588620 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:35.588571 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pkkpq" podStartSLOduration=1.6809214190000001 podStartE2EDuration="4.588556495s" totalImagesPullingTime="2.907635076s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 01:41:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:41:35.586494899 +0000 UTC m=+319.173660904" watchObservedRunningTime="2026-07-21 01:41:35.588556495 +0000 UTC m=+319.175722491" Jul 21 01:41:36.561632 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:36.561598 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pgxz4" event={"ID":"a5b32c0e-f65a-48ba-a752-1b78cb1950e3","Type":"ContainerStarted","Data":"290d83f6745998eed32951aa80c93762fdb05c029a4f5519981978e8bb388113"} Jul 21 01:41:36.562106 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:36.561716 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pgxz4" Jul 21 01:41:36.578930 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:36.578878 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pgxz4" podStartSLOduration=2.596057503 podStartE2EDuration="5.578859668s" totalImagesPullingTime="2.982802165s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 01:41:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:41:36.578020422 +0000 UTC m=+320.165186420" watchObservedRunningTime="2026-07-21 01:41:36.578859668 +0000 UTC m=+320.166025649" Jul 21 01:41:36.877921 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:36.877843 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a171a7ed-c69a-4b4f-939c-8181e00287ba" path="/var/lib/kubelet/pods/a171a7ed-c69a-4b4f-939c-8181e00287ba/volumes" Jul 21 01:41:37.505890 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:37.505860 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-bb4gc" Jul 21 01:41:37.507562 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:41:37.507540 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-8dt74" Jul 21 01:42:06.567235 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:42:06.567205 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pkkpq" Jul 21 01:42:07.571059 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:42:07.571030 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pgxz4" Jul 21 01:42:21.080867 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:42:21.080828 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-lmx5z"] Jul 21 01:42:21.081527 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:42:21.081487 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a171a7ed-c69a-4b4f-939c-8181e00287ba" containerName="discovery" Jul 21 01:42:21.081527 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:42:21.081514 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a171a7ed-c69a-4b4f-939c-8181e00287ba" containerName="discovery" Jul 21 01:42:21.081748 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:42:21.081646 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a171a7ed-c69a-4b4f-939c-8181e00287ba" containerName="discovery" Jul 21 01:42:21.084700 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:42:21.084669 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-lmx5z" Jul 21 01:42:21.086888 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:42:21.086867 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 21 01:42:21.087175 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:42:21.087154 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 21 01:42:21.087769 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:42:21.087744 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-s9qfb\"" Jul 21 01:42:21.094140 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:42:21.093650 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-lmx5z"] Jul 21 01:42:21.248372 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:42:21.248338 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jjqs\" (UniqueName: \"kubernetes.io/projected/07277352-fdd4-4e30-9ccb-e39393711132-kube-api-access-6jjqs\") pod \"kuadrant-operator-catalog-lmx5z\" (UID: \"07277352-fdd4-4e30-9ccb-e39393711132\") " pod="kuadrant-system/kuadrant-operator-catalog-lmx5z" Jul 21 01:42:21.349692 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:42:21.349604 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jjqs\" (UniqueName: \"kubernetes.io/projected/07277352-fdd4-4e30-9ccb-e39393711132-kube-api-access-6jjqs\") pod \"kuadrant-operator-catalog-lmx5z\" (UID: \"07277352-fdd4-4e30-9ccb-e39393711132\") " pod="kuadrant-system/kuadrant-operator-catalog-lmx5z" Jul 21 01:42:21.358064 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:42:21.358039 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jjqs\" (UniqueName: \"kubernetes.io/projected/07277352-fdd4-4e30-9ccb-e39393711132-kube-api-access-6jjqs\") pod \"kuadrant-operator-catalog-lmx5z\" (UID: \"07277352-fdd4-4e30-9ccb-e39393711132\") " pod="kuadrant-system/kuadrant-operator-catalog-lmx5z" Jul 21 01:42:21.397336 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:42:21.397295 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-lmx5z" Jul 21 01:42:21.441444 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:42:21.441411 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-lmx5z"] Jul 21 01:42:21.529410 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:42:21.529387 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-lmx5z"] Jul 21 01:42:21.531869 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:42:21.531843 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07277352_fdd4_4e30_9ccb_e39393711132.slice/crio-77d01bca6d8323684427dcc4dc6ff4b1258a7736394c531a91128613a650342f WatchSource:0}: Error finding container 77d01bca6d8323684427dcc4dc6ff4b1258a7736394c531a91128613a650342f: Status 404 returned error can't find the container with id 77d01bca6d8323684427dcc4dc6ff4b1258a7736394c531a91128613a650342f Jul 21 01:42:21.651470 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:42:21.651372 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-r79j5"] Jul 21 01:42:21.654423 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:42:21.654408 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-r79j5" Jul 21 01:42:21.662784 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:42:21.662760 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-r79j5"] Jul 21 01:42:21.753948 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:42:21.753918 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hklqs\" (UniqueName: \"kubernetes.io/projected/e8374b93-553a-4491-9eb4-e8282bb527b1-kube-api-access-hklqs\") pod \"kuadrant-operator-catalog-r79j5\" (UID: \"e8374b93-553a-4491-9eb4-e8282bb527b1\") " pod="kuadrant-system/kuadrant-operator-catalog-r79j5" Jul 21 01:42:21.780529 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:42:21.780494 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-lmx5z" event={"ID":"07277352-fdd4-4e30-9ccb-e39393711132","Type":"ContainerStarted","Data":"77d01bca6d8323684427dcc4dc6ff4b1258a7736394c531a91128613a650342f"} Jul 21 01:42:21.854698 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:42:21.854667 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hklqs\" (UniqueName: \"kubernetes.io/projected/e8374b93-553a-4491-9eb4-e8282bb527b1-kube-api-access-hklqs\") pod \"kuadrant-operator-catalog-r79j5\" (UID: \"e8374b93-553a-4491-9eb4-e8282bb527b1\") " pod="kuadrant-system/kuadrant-operator-catalog-r79j5" Jul 21 01:42:21.862276 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:42:21.862253 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hklqs\" (UniqueName: \"kubernetes.io/projected/e8374b93-553a-4491-9eb4-e8282bb527b1-kube-api-access-hklqs\") pod \"kuadrant-operator-catalog-r79j5\" (UID: \"e8374b93-553a-4491-9eb4-e8282bb527b1\") " pod="kuadrant-system/kuadrant-operator-catalog-r79j5" Jul 21 01:42:21.965507 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:42:21.965408 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-r79j5" Jul 21 01:42:22.111989 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:42:22.111911 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-r79j5"] Jul 21 01:42:22.151228 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:42:22.151185 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8374b93_553a_4491_9eb4_e8282bb527b1.slice/crio-60640da0d870a92cdeccb431c29360dea7f9e9f6d586f4cdcdfe2977af2228dc WatchSource:0}: Error finding container 60640da0d870a92cdeccb431c29360dea7f9e9f6d586f4cdcdfe2977af2228dc: Status 404 returned error can't find the container with id 60640da0d870a92cdeccb431c29360dea7f9e9f6d586f4cdcdfe2977af2228dc Jul 21 01:42:22.789172 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:42:22.789128 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-r79j5" event={"ID":"e8374b93-553a-4491-9eb4-e8282bb527b1","Type":"ContainerStarted","Data":"60640da0d870a92cdeccb431c29360dea7f9e9f6d586f4cdcdfe2977af2228dc"} Jul 21 01:42:23.796305 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:42:23.796273 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-lmx5z" event={"ID":"07277352-fdd4-4e30-9ccb-e39393711132","Type":"ContainerStarted","Data":"9381e31e4bb5fe8a13b7a3f7ee6a4a6f146e9b1cacabc4a5714a128c11bf0523"} Jul 21 01:42:23.796780 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:42:23.796339 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-lmx5z" podUID="07277352-fdd4-4e30-9ccb-e39393711132" containerName="registry-server" containerID="cri-o://9381e31e4bb5fe8a13b7a3f7ee6a4a6f146e9b1cacabc4a5714a128c11bf0523" gracePeriod=2 Jul 21 01:42:23.797939 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:42:23.797904 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-r79j5" event={"ID":"e8374b93-553a-4491-9eb4-e8282bb527b1","Type":"ContainerStarted","Data":"f3569048e58572b69e33e056bfcdacee0413849048e926f92074b252129a30eb"} Jul 21 01:42:23.812330 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:42:23.812282 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-lmx5z" podStartSLOduration=0.844606723 podStartE2EDuration="2.812263679s" totalImagesPullingTime="1.967656956s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 01:42:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:42:23.810461749 +0000 UTC m=+367.397627742" watchObservedRunningTime="2026-07-21 01:42:23.812263679 +0000 UTC m=+367.399429660" Jul 21 01:42:23.825472 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:42:23.825407 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-r79j5" podStartSLOduration=1.476327253 podStartE2EDuration="2.825390811s" totalImagesPullingTime="1.349063558s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 01:42:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:42:23.824005408 +0000 UTC m=+367.411171386" watchObservedRunningTime="2026-07-21 01:42:23.825390811 +0000 UTC m=+367.412556788" Jul 21 01:42:24.041792 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:42:24.041767 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-lmx5z" Jul 21 01:42:24.177370 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:42:24.177275 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6jjqs\" (UniqueName: \"kubernetes.io/projected/07277352-fdd4-4e30-9ccb-e39393711132-kube-api-access-6jjqs\") pod \"07277352-fdd4-4e30-9ccb-e39393711132\" (UID: \"07277352-fdd4-4e30-9ccb-e39393711132\") " Jul 21 01:42:24.179485 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:42:24.179439 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07277352-fdd4-4e30-9ccb-e39393711132-kube-api-access-6jjqs" (OuterVolumeSpecName: "kube-api-access-6jjqs") pod "07277352-fdd4-4e30-9ccb-e39393711132" (UID: "07277352-fdd4-4e30-9ccb-e39393711132"). InnerVolumeSpecName "kube-api-access-6jjqs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 01:42:24.278357 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:42:24.278322 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jjqs\" (UniqueName: \"kubernetes.io/projected/07277352-fdd4-4e30-9ccb-e39393711132-kube-api-access-6jjqs\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 21 01:42:24.804162 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:42:24.804076 2580 generic.go:356] "Generic (PLEG): container finished" podID="07277352-fdd4-4e30-9ccb-e39393711132" containerID="9381e31e4bb5fe8a13b7a3f7ee6a4a6f146e9b1cacabc4a5714a128c11bf0523" exitCode=0 Jul 21 01:42:24.804162 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:42:24.804113 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-lmx5z" Jul 21 01:42:24.804669 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:42:24.804160 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-lmx5z" event={"ID":"07277352-fdd4-4e30-9ccb-e39393711132","Type":"ContainerDied","Data":"9381e31e4bb5fe8a13b7a3f7ee6a4a6f146e9b1cacabc4a5714a128c11bf0523"} Jul 21 01:42:24.804669 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:42:24.804204 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-lmx5z" event={"ID":"07277352-fdd4-4e30-9ccb-e39393711132","Type":"ContainerDied","Data":"77d01bca6d8323684427dcc4dc6ff4b1258a7736394c531a91128613a650342f"} Jul 21 01:42:24.804669 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:42:24.804226 2580 scope.go:117] "RemoveContainer" containerID="9381e31e4bb5fe8a13b7a3f7ee6a4a6f146e9b1cacabc4a5714a128c11bf0523" Jul 21 01:42:24.814017 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:42:24.814001 2580 scope.go:117] "RemoveContainer" containerID="9381e31e4bb5fe8a13b7a3f7ee6a4a6f146e9b1cacabc4a5714a128c11bf0523" Jul 21 01:42:24.814282 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:42:24.814263 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9381e31e4bb5fe8a13b7a3f7ee6a4a6f146e9b1cacabc4a5714a128c11bf0523\": container with ID starting with 9381e31e4bb5fe8a13b7a3f7ee6a4a6f146e9b1cacabc4a5714a128c11bf0523 not found: ID does not exist" containerID="9381e31e4bb5fe8a13b7a3f7ee6a4a6f146e9b1cacabc4a5714a128c11bf0523" Jul 21 01:42:24.814349 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:42:24.814295 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9381e31e4bb5fe8a13b7a3f7ee6a4a6f146e9b1cacabc4a5714a128c11bf0523"} err="failed to get container status \"9381e31e4bb5fe8a13b7a3f7ee6a4a6f146e9b1cacabc4a5714a128c11bf0523\": rpc error: code = NotFound desc = could not find container \"9381e31e4bb5fe8a13b7a3f7ee6a4a6f146e9b1cacabc4a5714a128c11bf0523\": container with ID starting with 9381e31e4bb5fe8a13b7a3f7ee6a4a6f146e9b1cacabc4a5714a128c11bf0523 not found: ID does not exist" Jul 21 01:42:24.823777 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:42:24.823750 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-lmx5z"] Jul 21 01:42:24.827776 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:42:24.827752 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-lmx5z"] Jul 21 01:42:24.880928 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:42:24.879011 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07277352-fdd4-4e30-9ccb-e39393711132" path="/var/lib/kubelet/pods/07277352-fdd4-4e30-9ccb-e39393711132/volumes" Jul 21 01:42:31.966133 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:42:31.966094 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-r79j5" Jul 21 01:42:31.966713 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:42:31.966324 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-r79j5" Jul 21 01:42:31.989765 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:42:31.989736 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-r79j5" Jul 21 01:42:32.865994 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:42:32.865963 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-r79j5" Jul 21 01:42:52.514374 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:42:52.514338 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-7gfwm"] Jul 21 01:42:52.514890 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:42:52.514736 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07277352-fdd4-4e30-9ccb-e39393711132" containerName="registry-server" Jul 21 01:42:52.514890 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:42:52.514747 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="07277352-fdd4-4e30-9ccb-e39393711132" containerName="registry-server" Jul 21 01:42:52.514890 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:42:52.514815 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="07277352-fdd4-4e30-9ccb-e39393711132" containerName="registry-server" Jul 21 01:42:52.518197 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:42:52.518177 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-7gfwm" Jul 21 01:42:52.520826 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:42:52.520804 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-2qhv9\"" Jul 21 01:42:52.520935 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:42:52.520869 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 21 01:42:52.547470 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:42:52.527757 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-7gfwm"] Jul 21 01:42:52.631349 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:42:52.631311 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqdnl\" (UniqueName: \"kubernetes.io/projected/3f0675f5-b65c-4a80-acda-d1744325c0e9-kube-api-access-qqdnl\") pod \"dns-operator-controller-manager-864dd78c4c-7gfwm\" (UID: \"3f0675f5-b65c-4a80-acda-d1744325c0e9\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-7gfwm" Jul 21 01:42:52.732495 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:42:52.732444 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqdnl\" (UniqueName: \"kubernetes.io/projected/3f0675f5-b65c-4a80-acda-d1744325c0e9-kube-api-access-qqdnl\") pod \"dns-operator-controller-manager-864dd78c4c-7gfwm\" (UID: \"3f0675f5-b65c-4a80-acda-d1744325c0e9\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-7gfwm" Jul 21 01:42:52.739945 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:42:52.739915 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqdnl\" (UniqueName: \"kubernetes.io/projected/3f0675f5-b65c-4a80-acda-d1744325c0e9-kube-api-access-qqdnl\") pod \"dns-operator-controller-manager-864dd78c4c-7gfwm\" (UID: \"3f0675f5-b65c-4a80-acda-d1744325c0e9\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-7gfwm" Jul 21 01:42:52.831737 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:42:52.831652 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-7gfwm" Jul 21 01:42:52.991803 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:42:52.991776 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-7gfwm"] Jul 21 01:42:52.995004 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:42:52.994972 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f0675f5_b65c_4a80_acda_d1744325c0e9.slice/crio-45d24298ddef47da7ba983f088d7c1bece26f96353edcd653bf6cae47ea2ead7 WatchSource:0}: Error finding container 45d24298ddef47da7ba983f088d7c1bece26f96353edcd653bf6cae47ea2ead7: Status 404 returned error can't find the container with id 45d24298ddef47da7ba983f088d7c1bece26f96353edcd653bf6cae47ea2ead7 Jul 21 01:42:53.951807 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:42:53.951770 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-7gfwm" event={"ID":"3f0675f5-b65c-4a80-acda-d1744325c0e9","Type":"ContainerStarted","Data":"45d24298ddef47da7ba983f088d7c1bece26f96353edcd653bf6cae47ea2ead7"} Jul 21 01:42:54.958400 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:42:54.958370 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-7gfwm" event={"ID":"3f0675f5-b65c-4a80-acda-d1744325c0e9","Type":"ContainerStarted","Data":"6be72be8085caa1da5f3ed042552229bde3e14744b8e706f6ffc36c881c0fa2c"} Jul 21 01:42:54.958821 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:42:54.958561 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-7gfwm" Jul 21 01:42:54.975340 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:42:54.975287 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-7gfwm" podStartSLOduration=1.1686608490000001 podStartE2EDuration="2.975265425s" totalImagesPullingTime="1.806604576s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 01:42:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:42:54.97502031 +0000 UTC m=+398.562186315" watchObservedRunningTime="2026-07-21 01:42:54.975265425 +0000 UTC m=+398.562431403" Jul 21 01:42:55.931390 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:42:55.931348 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gb22k"] Jul 21 01:42:55.935030 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:42:55.935013 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gb22k" Jul 21 01:42:55.939077 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:42:55.939049 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-7nz7g\"" Jul 21 01:42:55.950985 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:42:55.950961 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gb22k"] Jul 21 01:42:56.062893 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:42:56.062846 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wq2qt\" (UniqueName: \"kubernetes.io/projected/3faa50b6-8d97-43a1-844d-91962b724e8f-kube-api-access-wq2qt\") pod \"limitador-operator-controller-manager-859d7b78bd-gb22k\" (UID: \"3faa50b6-8d97-43a1-844d-91962b724e8f\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gb22k" Jul 21 01:42:56.164500 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:42:56.164436 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq2qt\" (UniqueName: \"kubernetes.io/projected/3faa50b6-8d97-43a1-844d-91962b724e8f-kube-api-access-wq2qt\") pod \"limitador-operator-controller-manager-859d7b78bd-gb22k\" (UID: \"3faa50b6-8d97-43a1-844d-91962b724e8f\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gb22k" Jul 21 01:42:56.177619 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:42:56.177582 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wq2qt\" (UniqueName: \"kubernetes.io/projected/3faa50b6-8d97-43a1-844d-91962b724e8f-kube-api-access-wq2qt\") pod \"limitador-operator-controller-manager-859d7b78bd-gb22k\" (UID: \"3faa50b6-8d97-43a1-844d-91962b724e8f\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gb22k" Jul 21 01:42:56.245650 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:42:56.245561 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gb22k" Jul 21 01:42:56.378970 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:42:56.378948 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gb22k"] Jul 21 01:42:56.381293 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:42:56.381261 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3faa50b6_8d97_43a1_844d_91962b724e8f.slice/crio-4fd001fb495bedd0877f55ab3e4c90d278295bd5621b64175cc810a2d29910a3 WatchSource:0}: Error finding container 4fd001fb495bedd0877f55ab3e4c90d278295bd5621b64175cc810a2d29910a3: Status 404 returned error can't find the container with id 4fd001fb495bedd0877f55ab3e4c90d278295bd5621b64175cc810a2d29910a3 Jul 21 01:42:56.970213 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:42:56.970177 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gb22k" event={"ID":"3faa50b6-8d97-43a1-844d-91962b724e8f","Type":"ContainerStarted","Data":"4fd001fb495bedd0877f55ab3e4c90d278295bd5621b64175cc810a2d29910a3"} Jul 21 01:42:58.984481 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:42:58.984342 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gb22k" event={"ID":"3faa50b6-8d97-43a1-844d-91962b724e8f","Type":"ContainerStarted","Data":"bca458d627abd46c8520548563f212402c3aa297fe550a1d67e06964a10e9d8f"} Jul 21 01:42:58.984481 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:42:58.984422 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gb22k" Jul 21 01:42:59.001958 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:42:59.001899 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gb22k" podStartSLOduration=2.133217674 podStartE2EDuration="4.001875605s" totalImagesPullingTime="1.868657931s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 01:42:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:42:59.000694057 +0000 UTC m=+402.587860047" watchObservedRunningTime="2026-07-21 01:42:59.001875605 +0000 UTC m=+402.589041584" Jul 21 01:42:59.887307 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:42:59.887272 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-kxb76"] Jul 21 01:42:59.890847 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:42:59.890829 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-kxb76" Jul 21 01:42:59.893653 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:42:59.893631 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-jcqth\"" Jul 21 01:42:59.903661 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:42:59.903637 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-kxb76"] Jul 21 01:43:00.000341 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:00.000306 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87lcp\" (UniqueName: \"kubernetes.io/projected/270a391d-e4cd-4d02-8b67-66f6c9715e8e-kube-api-access-87lcp\") pod \"authorino-operator-75cc8d5cb7-kxb76\" (UID: \"270a391d-e4cd-4d02-8b67-66f6c9715e8e\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-kxb76" Jul 21 01:43:00.101816 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:00.101775 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87lcp\" (UniqueName: \"kubernetes.io/projected/270a391d-e4cd-4d02-8b67-66f6c9715e8e-kube-api-access-87lcp\") pod \"authorino-operator-75cc8d5cb7-kxb76\" (UID: \"270a391d-e4cd-4d02-8b67-66f6c9715e8e\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-kxb76" Jul 21 01:43:00.110104 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:00.110077 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87lcp\" (UniqueName: \"kubernetes.io/projected/270a391d-e4cd-4d02-8b67-66f6c9715e8e-kube-api-access-87lcp\") pod \"authorino-operator-75cc8d5cb7-kxb76\" (UID: \"270a391d-e4cd-4d02-8b67-66f6c9715e8e\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-kxb76" Jul 21 01:43:00.201294 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:00.201204 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-kxb76" Jul 21 01:43:00.338341 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:00.338317 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-kxb76"] Jul 21 01:43:00.340551 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:43:00.340518 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod270a391d_e4cd_4d02_8b67_66f6c9715e8e.slice/crio-146dff166761386042e609b1635ec53a55e69248b074638002b6bc95533b8f84 WatchSource:0}: Error finding container 146dff166761386042e609b1635ec53a55e69248b074638002b6bc95533b8f84: Status 404 returned error can't find the container with id 146dff166761386042e609b1635ec53a55e69248b074638002b6bc95533b8f84 Jul 21 01:43:00.997546 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:00.997500 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-kxb76" event={"ID":"270a391d-e4cd-4d02-8b67-66f6c9715e8e","Type":"ContainerStarted","Data":"146dff166761386042e609b1635ec53a55e69248b074638002b6bc95533b8f84"} Jul 21 01:43:03.010811 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:03.010772 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-kxb76" event={"ID":"270a391d-e4cd-4d02-8b67-66f6c9715e8e","Type":"ContainerStarted","Data":"1dfe0d3e94c4f887063a0cf399353b085fd8fbd3b7ca785bbc1a1717f093a832"} Jul 21 01:43:03.011234 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:03.010889 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-kxb76" Jul 21 01:43:03.027966 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:03.027913 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-kxb76" podStartSLOduration=1.738808291 podStartE2EDuration="4.027896387s" totalImagesPullingTime="2.289088096s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 01:42:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:43:03.025430878 +0000 UTC m=+406.612596857" watchObservedRunningTime="2026-07-21 01:43:03.027896387 +0000 UTC m=+406.615062365" Jul 21 01:43:04.611996 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:04.611962 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-db4ck"] Jul 21 01:43:04.615662 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:04.615639 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-db4ck" Jul 21 01:43:04.618000 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:04.617979 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-l692r\"" Jul 21 01:43:04.628302 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:04.628278 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-db4ck"] Jul 21 01:43:04.748902 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:04.748867 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mr9sq\" (UniqueName: \"kubernetes.io/projected/42789c7f-7215-4f03-896f-67b73cbd82dd-kube-api-access-mr9sq\") pod \"kuadrant-operator-controller-manager-8684c9ff49-db4ck\" (UID: \"42789c7f-7215-4f03-896f-67b73cbd82dd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-db4ck" Jul 21 01:43:04.749070 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:04.748917 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/42789c7f-7215-4f03-896f-67b73cbd82dd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-db4ck\" (UID: \"42789c7f-7215-4f03-896f-67b73cbd82dd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-db4ck" Jul 21 01:43:04.850359 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:04.850321 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mr9sq\" (UniqueName: \"kubernetes.io/projected/42789c7f-7215-4f03-896f-67b73cbd82dd-kube-api-access-mr9sq\") pod \"kuadrant-operator-controller-manager-8684c9ff49-db4ck\" (UID: \"42789c7f-7215-4f03-896f-67b73cbd82dd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-db4ck" Jul 21 01:43:04.850553 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:04.850369 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/42789c7f-7215-4f03-896f-67b73cbd82dd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-db4ck\" (UID: \"42789c7f-7215-4f03-896f-67b73cbd82dd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-db4ck" Jul 21 01:43:04.850726 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:04.850708 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/42789c7f-7215-4f03-896f-67b73cbd82dd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-db4ck\" (UID: \"42789c7f-7215-4f03-896f-67b73cbd82dd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-db4ck" Jul 21 01:43:04.858391 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:04.858369 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mr9sq\" (UniqueName: \"kubernetes.io/projected/42789c7f-7215-4f03-896f-67b73cbd82dd-kube-api-access-mr9sq\") pod \"kuadrant-operator-controller-manager-8684c9ff49-db4ck\" (UID: \"42789c7f-7215-4f03-896f-67b73cbd82dd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-db4ck" Jul 21 01:43:04.926592 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:04.926490 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-db4ck" Jul 21 01:43:05.084060 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:05.084040 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-db4ck"] Jul 21 01:43:05.085949 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:43:05.085918 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42789c7f_7215_4f03_896f_67b73cbd82dd.slice/crio-317141411c8fa349bd50dd0738b0c3393cf076219fadbea86026880c498904c0 WatchSource:0}: Error finding container 317141411c8fa349bd50dd0738b0c3393cf076219fadbea86026880c498904c0: Status 404 returned error can't find the container with id 317141411c8fa349bd50dd0738b0c3393cf076219fadbea86026880c498904c0 Jul 21 01:43:05.966178 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:05.966150 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-7gfwm" Jul 21 01:43:06.030590 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:06.030551 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-db4ck" event={"ID":"42789c7f-7215-4f03-896f-67b73cbd82dd","Type":"ContainerStarted","Data":"317141411c8fa349bd50dd0738b0c3393cf076219fadbea86026880c498904c0"} Jul 21 01:43:09.991676 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:09.991645 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gb22k" Jul 21 01:43:11.058858 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:11.058822 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-db4ck" event={"ID":"42789c7f-7215-4f03-896f-67b73cbd82dd","Type":"ContainerStarted","Data":"a8c8eb75d4e4eee1e86b78adea372bca5953f2bb6215bdee39c3a106dc202dc1"} Jul 21 01:43:11.059255 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:11.058961 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-db4ck" Jul 21 01:43:11.077789 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:11.077744 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-db4ck" podStartSLOduration=1.6653855389999999 podStartE2EDuration="7.077726749s" totalImagesPullingTime="5.41234121s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 01:43:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:43:11.075957308 +0000 UTC m=+414.663123288" watchObservedRunningTime="2026-07-21 01:43:11.077726749 +0000 UTC m=+414.664892796" Jul 21 01:43:13.167718 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:13.167685 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-xtgcn"] Jul 21 01:43:13.171182 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:13.171160 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xtgcn" Jul 21 01:43:13.174203 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:13.174184 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 21 01:43:13.174326 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:13.174188 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-7xkt9\"" Jul 21 01:43:13.174326 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:13.174223 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 21 01:43:13.178965 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:13.178937 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-xtgcn"] Jul 21 01:43:13.229609 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:13.229574 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0a12d9c8-e70f-4f03-8465-926cdc4645f5-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-xtgcn\" (UID: \"0a12d9c8-e70f-4f03-8465-926cdc4645f5\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xtgcn" Jul 21 01:43:13.229609 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:13.229615 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0a12d9c8-e70f-4f03-8465-926cdc4645f5-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-xtgcn\" (UID: \"0a12d9c8-e70f-4f03-8465-926cdc4645f5\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xtgcn" Jul 21 01:43:13.229832 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:13.229652 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tv7hm\" (UniqueName: \"kubernetes.io/projected/0a12d9c8-e70f-4f03-8465-926cdc4645f5-kube-api-access-tv7hm\") pod \"kuadrant-console-plugin-66b5774985-xtgcn\" (UID: \"0a12d9c8-e70f-4f03-8465-926cdc4645f5\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xtgcn" Jul 21 01:43:13.330410 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:13.330367 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0a12d9c8-e70f-4f03-8465-926cdc4645f5-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-xtgcn\" (UID: \"0a12d9c8-e70f-4f03-8465-926cdc4645f5\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xtgcn" Jul 21 01:43:13.330410 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:13.330415 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0a12d9c8-e70f-4f03-8465-926cdc4645f5-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-xtgcn\" (UID: \"0a12d9c8-e70f-4f03-8465-926cdc4645f5\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xtgcn" Jul 21 01:43:13.330709 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:13.330481 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv7hm\" (UniqueName: \"kubernetes.io/projected/0a12d9c8-e70f-4f03-8465-926cdc4645f5-kube-api-access-tv7hm\") pod \"kuadrant-console-plugin-66b5774985-xtgcn\" (UID: \"0a12d9c8-e70f-4f03-8465-926cdc4645f5\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xtgcn" Jul 21 01:43:13.331120 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:13.331087 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0a12d9c8-e70f-4f03-8465-926cdc4645f5-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-xtgcn\" (UID: \"0a12d9c8-e70f-4f03-8465-926cdc4645f5\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xtgcn" Jul 21 01:43:13.332873 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:13.332853 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0a12d9c8-e70f-4f03-8465-926cdc4645f5-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-xtgcn\" (UID: \"0a12d9c8-e70f-4f03-8465-926cdc4645f5\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xtgcn" Jul 21 01:43:13.338176 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:13.338147 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv7hm\" (UniqueName: \"kubernetes.io/projected/0a12d9c8-e70f-4f03-8465-926cdc4645f5-kube-api-access-tv7hm\") pod \"kuadrant-console-plugin-66b5774985-xtgcn\" (UID: \"0a12d9c8-e70f-4f03-8465-926cdc4645f5\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xtgcn" Jul 21 01:43:13.483422 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:13.483333 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xtgcn" Jul 21 01:43:13.620259 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:13.620232 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-xtgcn"] Jul 21 01:43:13.622876 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:43:13.622851 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0a12d9c8_e70f_4f03_8465_926cdc4645f5.slice/crio-e02b2069d2028ea4bb1cddab5baeee94ff707b234c6832f31bc2e24b238a591a WatchSource:0}: Error finding container e02b2069d2028ea4bb1cddab5baeee94ff707b234c6832f31bc2e24b238a591a: Status 404 returned error can't find the container with id e02b2069d2028ea4bb1cddab5baeee94ff707b234c6832f31bc2e24b238a591a Jul 21 01:43:14.018605 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:14.018575 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-kxb76" Jul 21 01:43:14.075566 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:14.075519 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-8fdf8bfcb-qdwmd"] Jul 21 01:43:14.082377 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:14.081088 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xtgcn" event={"ID":"0a12d9c8-e70f-4f03-8465-926cdc4645f5","Type":"ContainerStarted","Data":"e02b2069d2028ea4bb1cddab5baeee94ff707b234c6832f31bc2e24b238a591a"} Jul 21 01:43:14.082377 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:14.081207 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8fdf8bfcb-qdwmd" Jul 21 01:43:14.083048 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:14.083025 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8fdf8bfcb-qdwmd"] Jul 21 01:43:14.135897 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:14.135863 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01a8b910-97b0-4384-a18e-bf3188092b4b-trusted-ca-bundle\") pod \"console-8fdf8bfcb-qdwmd\" (UID: \"01a8b910-97b0-4384-a18e-bf3188092b4b\") " pod="openshift-console/console-8fdf8bfcb-qdwmd" Jul 21 01:43:14.136066 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:14.135909 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/01a8b910-97b0-4384-a18e-bf3188092b4b-console-config\") pod \"console-8fdf8bfcb-qdwmd\" (UID: \"01a8b910-97b0-4384-a18e-bf3188092b4b\") " pod="openshift-console/console-8fdf8bfcb-qdwmd" Jul 21 01:43:14.136066 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:14.135931 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/01a8b910-97b0-4384-a18e-bf3188092b4b-oauth-serving-cert\") pod \"console-8fdf8bfcb-qdwmd\" (UID: \"01a8b910-97b0-4384-a18e-bf3188092b4b\") " pod="openshift-console/console-8fdf8bfcb-qdwmd" Jul 21 01:43:14.136066 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:14.135980 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/01a8b910-97b0-4384-a18e-bf3188092b4b-service-ca\") pod \"console-8fdf8bfcb-qdwmd\" (UID: \"01a8b910-97b0-4384-a18e-bf3188092b4b\") " pod="openshift-console/console-8fdf8bfcb-qdwmd" Jul 21 01:43:14.136066 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:14.136005 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/01a8b910-97b0-4384-a18e-bf3188092b4b-console-serving-cert\") pod \"console-8fdf8bfcb-qdwmd\" (UID: \"01a8b910-97b0-4384-a18e-bf3188092b4b\") " pod="openshift-console/console-8fdf8bfcb-qdwmd" Jul 21 01:43:14.136066 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:14.136045 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7c5x\" (UniqueName: \"kubernetes.io/projected/01a8b910-97b0-4384-a18e-bf3188092b4b-kube-api-access-b7c5x\") pod \"console-8fdf8bfcb-qdwmd\" (UID: \"01a8b910-97b0-4384-a18e-bf3188092b4b\") " pod="openshift-console/console-8fdf8bfcb-qdwmd" Jul 21 01:43:14.136249 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:14.136104 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/01a8b910-97b0-4384-a18e-bf3188092b4b-console-oauth-config\") pod \"console-8fdf8bfcb-qdwmd\" (UID: \"01a8b910-97b0-4384-a18e-bf3188092b4b\") " pod="openshift-console/console-8fdf8bfcb-qdwmd" Jul 21 01:43:14.236655 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:14.236623 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/01a8b910-97b0-4384-a18e-bf3188092b4b-console-config\") pod \"console-8fdf8bfcb-qdwmd\" (UID: \"01a8b910-97b0-4384-a18e-bf3188092b4b\") " pod="openshift-console/console-8fdf8bfcb-qdwmd" Jul 21 01:43:14.236655 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:14.236660 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/01a8b910-97b0-4384-a18e-bf3188092b4b-oauth-serving-cert\") pod \"console-8fdf8bfcb-qdwmd\" (UID: \"01a8b910-97b0-4384-a18e-bf3188092b4b\") " pod="openshift-console/console-8fdf8bfcb-qdwmd" Jul 21 01:43:14.237180 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:14.236682 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/01a8b910-97b0-4384-a18e-bf3188092b4b-service-ca\") pod \"console-8fdf8bfcb-qdwmd\" (UID: \"01a8b910-97b0-4384-a18e-bf3188092b4b\") " pod="openshift-console/console-8fdf8bfcb-qdwmd" Jul 21 01:43:14.237180 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:14.236709 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/01a8b910-97b0-4384-a18e-bf3188092b4b-console-serving-cert\") pod \"console-8fdf8bfcb-qdwmd\" (UID: \"01a8b910-97b0-4384-a18e-bf3188092b4b\") " pod="openshift-console/console-8fdf8bfcb-qdwmd" Jul 21 01:43:14.237180 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:14.236736 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7c5x\" (UniqueName: \"kubernetes.io/projected/01a8b910-97b0-4384-a18e-bf3188092b4b-kube-api-access-b7c5x\") pod \"console-8fdf8bfcb-qdwmd\" (UID: \"01a8b910-97b0-4384-a18e-bf3188092b4b\") " pod="openshift-console/console-8fdf8bfcb-qdwmd" Jul 21 01:43:14.237180 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:14.236755 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/01a8b910-97b0-4384-a18e-bf3188092b4b-console-oauth-config\") pod \"console-8fdf8bfcb-qdwmd\" (UID: \"01a8b910-97b0-4384-a18e-bf3188092b4b\") " pod="openshift-console/console-8fdf8bfcb-qdwmd" Jul 21 01:43:14.237180 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:14.236793 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01a8b910-97b0-4384-a18e-bf3188092b4b-trusted-ca-bundle\") pod \"console-8fdf8bfcb-qdwmd\" (UID: \"01a8b910-97b0-4384-a18e-bf3188092b4b\") " pod="openshift-console/console-8fdf8bfcb-qdwmd" Jul 21 01:43:14.237515 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:14.237487 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/01a8b910-97b0-4384-a18e-bf3188092b4b-console-config\") pod \"console-8fdf8bfcb-qdwmd\" (UID: \"01a8b910-97b0-4384-a18e-bf3188092b4b\") " pod="openshift-console/console-8fdf8bfcb-qdwmd" Jul 21 01:43:14.237701 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:14.237679 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/01a8b910-97b0-4384-a18e-bf3188092b4b-service-ca\") pod \"console-8fdf8bfcb-qdwmd\" (UID: \"01a8b910-97b0-4384-a18e-bf3188092b4b\") " pod="openshift-console/console-8fdf8bfcb-qdwmd" Jul 21 01:43:14.237813 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:14.237711 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/01a8b910-97b0-4384-a18e-bf3188092b4b-oauth-serving-cert\") pod \"console-8fdf8bfcb-qdwmd\" (UID: \"01a8b910-97b0-4384-a18e-bf3188092b4b\") " pod="openshift-console/console-8fdf8bfcb-qdwmd" Jul 21 01:43:14.237977 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:14.237951 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01a8b910-97b0-4384-a18e-bf3188092b4b-trusted-ca-bundle\") pod \"console-8fdf8bfcb-qdwmd\" (UID: \"01a8b910-97b0-4384-a18e-bf3188092b4b\") " pod="openshift-console/console-8fdf8bfcb-qdwmd" Jul 21 01:43:14.239386 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:14.239361 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/01a8b910-97b0-4384-a18e-bf3188092b4b-console-serving-cert\") pod \"console-8fdf8bfcb-qdwmd\" (UID: \"01a8b910-97b0-4384-a18e-bf3188092b4b\") " pod="openshift-console/console-8fdf8bfcb-qdwmd" Jul 21 01:43:14.239507 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:14.239425 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/01a8b910-97b0-4384-a18e-bf3188092b4b-console-oauth-config\") pod \"console-8fdf8bfcb-qdwmd\" (UID: \"01a8b910-97b0-4384-a18e-bf3188092b4b\") " pod="openshift-console/console-8fdf8bfcb-qdwmd" Jul 21 01:43:14.245673 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:14.245652 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7c5x\" (UniqueName: \"kubernetes.io/projected/01a8b910-97b0-4384-a18e-bf3188092b4b-kube-api-access-b7c5x\") pod \"console-8fdf8bfcb-qdwmd\" (UID: \"01a8b910-97b0-4384-a18e-bf3188092b4b\") " pod="openshift-console/console-8fdf8bfcb-qdwmd" Jul 21 01:43:14.395304 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:14.395215 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8fdf8bfcb-qdwmd" Jul 21 01:43:14.571777 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:14.571740 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8fdf8bfcb-qdwmd"] Jul 21 01:43:14.576059 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:43:14.576028 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01a8b910_97b0_4384_a18e_bf3188092b4b.slice/crio-fac38d8797fb9a22e1e1fcd56a241f5f976f86648c15bb81077eff6251452b80 WatchSource:0}: Error finding container fac38d8797fb9a22e1e1fcd56a241f5f976f86648c15bb81077eff6251452b80: Status 404 returned error can't find the container with id fac38d8797fb9a22e1e1fcd56a241f5f976f86648c15bb81077eff6251452b80 Jul 21 01:43:15.086809 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:15.086772 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8fdf8bfcb-qdwmd" event={"ID":"01a8b910-97b0-4384-a18e-bf3188092b4b","Type":"ContainerStarted","Data":"cf0813bd607e84844f388b098e2ee799342ce1885812526a8ee321f8350de5eb"} Jul 21 01:43:15.087001 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:15.086825 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8fdf8bfcb-qdwmd" event={"ID":"01a8b910-97b0-4384-a18e-bf3188092b4b","Type":"ContainerStarted","Data":"fac38d8797fb9a22e1e1fcd56a241f5f976f86648c15bb81077eff6251452b80"} Jul 21 01:43:22.066065 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:22.066034 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-db4ck" Jul 21 01:43:22.084561 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:22.084512 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-8fdf8bfcb-qdwmd" podStartSLOduration=8.084496818 podStartE2EDuration="8.084496818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 01:43:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:43:15.105182756 +0000 UTC m=+418.692348734" watchObservedRunningTime="2026-07-21 01:43:22.084496818 +0000 UTC m=+425.671662774" Jul 21 01:43:23.131210 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:23.131172 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-nnnng"] Jul 21 01:43:23.141068 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:23.141025 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-nnnng" Jul 21 01:43:23.149703 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:23.149582 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-nnnng"] Jul 21 01:43:23.224124 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:23.224076 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e11d7ab7-2dcc-4357-aca4-af93f4a8531b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-nnnng\" (UID: \"e11d7ab7-2dcc-4357-aca4-af93f4a8531b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-nnnng" Jul 21 01:43:23.224124 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:23.224120 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfqp9\" (UniqueName: \"kubernetes.io/projected/e11d7ab7-2dcc-4357-aca4-af93f4a8531b-kube-api-access-gfqp9\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-nnnng\" (UID: \"e11d7ab7-2dcc-4357-aca4-af93f4a8531b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-nnnng" Jul 21 01:43:23.325388 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:23.325353 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e11d7ab7-2dcc-4357-aca4-af93f4a8531b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-nnnng\" (UID: \"e11d7ab7-2dcc-4357-aca4-af93f4a8531b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-nnnng" Jul 21 01:43:23.325580 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:23.325400 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfqp9\" (UniqueName: \"kubernetes.io/projected/e11d7ab7-2dcc-4357-aca4-af93f4a8531b-kube-api-access-gfqp9\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-nnnng\" (UID: \"e11d7ab7-2dcc-4357-aca4-af93f4a8531b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-nnnng" Jul 21 01:43:23.325824 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:23.325801 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e11d7ab7-2dcc-4357-aca4-af93f4a8531b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-nnnng\" (UID: \"e11d7ab7-2dcc-4357-aca4-af93f4a8531b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-nnnng" Jul 21 01:43:23.333852 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:23.333823 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfqp9\" (UniqueName: \"kubernetes.io/projected/e11d7ab7-2dcc-4357-aca4-af93f4a8531b-kube-api-access-gfqp9\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-nnnng\" (UID: \"e11d7ab7-2dcc-4357-aca4-af93f4a8531b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-nnnng" Jul 21 01:43:23.460359 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:23.460280 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-nnnng" Jul 21 01:43:23.986746 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:23.986708 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-nnnng"] Jul 21 01:43:24.001572 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:24.001491 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-nnnng"] Jul 21 01:43:24.005360 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:24.005317 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sqkjk"] Jul 21 01:43:24.022917 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:24.022687 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-db4ck"] Jul 21 01:43:24.022917 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:24.022846 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sqkjk" Jul 21 01:43:24.023193 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:24.022975 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-db4ck" podUID="42789c7f-7215-4f03-896f-67b73cbd82dd" containerName="manager" containerID="cri-o://a8c8eb75d4e4eee1e86b78adea372bca5953f2bb6215bdee39c3a106dc202dc1" gracePeriod=2 Jul 21 01:43:24.027698 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:24.027634 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sqkjk"] Jul 21 01:43:24.031987 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:24.030936 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-db4ck"] Jul 21 01:43:24.032845 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:43:24.032759 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-db4ck\" is forbidden: User \"system:node:ip-10-0-139-183.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-183.ec2.internal' and this object" podUID="42789c7f-7215-4f03-896f-67b73cbd82dd" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-db4ck" Jul 21 01:43:24.041814 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:24.041710 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mkv9c"] Jul 21 01:43:24.042668 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:24.042428 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42789c7f-7215-4f03-896f-67b73cbd82dd" containerName="manager" Jul 21 01:43:24.042668 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:24.042465 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="42789c7f-7215-4f03-896f-67b73cbd82dd" containerName="manager" Jul 21 01:43:24.042668 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:24.042570 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="42789c7f-7215-4f03-896f-67b73cbd82dd" containerName="manager" Jul 21 01:43:24.046547 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:24.046511 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gb22k"] Jul 21 01:43:24.047941 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:24.046997 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gb22k" podUID="3faa50b6-8d97-43a1-844d-91962b724e8f" containerName="manager" containerID="cri-o://bca458d627abd46c8520548563f212402c3aa297fe550a1d67e06964a10e9d8f" gracePeriod=2 Jul 21 01:43:24.047941 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:24.046805 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mkv9c" Jul 21 01:43:24.048677 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:24.048657 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gb22k"] Jul 21 01:43:24.053668 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:24.053610 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mkv9c"] Jul 21 01:43:24.060488 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:43:24.060432 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-db4ck\" is forbidden: User \"system:node:ip-10-0-139-183.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-183.ec2.internal' and this object" podUID="42789c7f-7215-4f03-896f-67b73cbd82dd" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-db4ck" Jul 21 01:43:24.063438 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:43:24.063397 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-gb22k\" is forbidden: User \"system:node:ip-10-0-139-183.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-183.ec2.internal' and this object" podUID="3faa50b6-8d97-43a1-844d-91962b724e8f" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gb22k" Jul 21 01:43:24.133241 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:24.133208 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88q8z\" (UniqueName: \"kubernetes.io/projected/dbdfd33d-b0c4-4993-8b64-adee4d6b9f7d-kube-api-access-88q8z\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-sqkjk\" (UID: \"dbdfd33d-b0c4-4993-8b64-adee4d6b9f7d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sqkjk" Jul 21 01:43:24.133630 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:24.133294 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dbdfd33d-b0c4-4993-8b64-adee4d6b9f7d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-sqkjk\" (UID: \"dbdfd33d-b0c4-4993-8b64-adee4d6b9f7d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sqkjk" Jul 21 01:43:24.133630 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:24.133334 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n88jx\" (UniqueName: \"kubernetes.io/projected/28a69e4b-001d-48b2-aec3-6765f7e179b6-kube-api-access-n88jx\") pod \"kuadrant-operator-controller-manager-8684c9ff49-mkv9c\" (UID: \"28a69e4b-001d-48b2-aec3-6765f7e179b6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mkv9c" Jul 21 01:43:24.133630 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:24.133489 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/28a69e4b-001d-48b2-aec3-6765f7e179b6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-mkv9c\" (UID: \"28a69e4b-001d-48b2-aec3-6765f7e179b6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mkv9c" Jul 21 01:43:24.234590 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:24.234546 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88q8z\" (UniqueName: \"kubernetes.io/projected/dbdfd33d-b0c4-4993-8b64-adee4d6b9f7d-kube-api-access-88q8z\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-sqkjk\" (UID: \"dbdfd33d-b0c4-4993-8b64-adee4d6b9f7d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sqkjk" Jul 21 01:43:24.234778 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:24.234634 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dbdfd33d-b0c4-4993-8b64-adee4d6b9f7d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-sqkjk\" (UID: \"dbdfd33d-b0c4-4993-8b64-adee4d6b9f7d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sqkjk" Jul 21 01:43:24.234778 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:24.234667 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n88jx\" (UniqueName: \"kubernetes.io/projected/28a69e4b-001d-48b2-aec3-6765f7e179b6-kube-api-access-n88jx\") pod \"kuadrant-operator-controller-manager-8684c9ff49-mkv9c\" (UID: \"28a69e4b-001d-48b2-aec3-6765f7e179b6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mkv9c" Jul 21 01:43:24.234778 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:24.234751 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/28a69e4b-001d-48b2-aec3-6765f7e179b6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-mkv9c\" (UID: \"28a69e4b-001d-48b2-aec3-6765f7e179b6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mkv9c" Jul 21 01:43:24.235103 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:24.235063 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dbdfd33d-b0c4-4993-8b64-adee4d6b9f7d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-sqkjk\" (UID: \"dbdfd33d-b0c4-4993-8b64-adee4d6b9f7d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sqkjk" Jul 21 01:43:24.235206 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:24.235165 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/28a69e4b-001d-48b2-aec3-6765f7e179b6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-mkv9c\" (UID: \"28a69e4b-001d-48b2-aec3-6765f7e179b6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mkv9c" Jul 21 01:43:24.249743 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:24.249710 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88q8z\" (UniqueName: \"kubernetes.io/projected/dbdfd33d-b0c4-4993-8b64-adee4d6b9f7d-kube-api-access-88q8z\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-sqkjk\" (UID: \"dbdfd33d-b0c4-4993-8b64-adee4d6b9f7d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sqkjk" Jul 21 01:43:24.249902 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:24.249820 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n88jx\" (UniqueName: \"kubernetes.io/projected/28a69e4b-001d-48b2-aec3-6765f7e179b6-kube-api-access-n88jx\") pod \"kuadrant-operator-controller-manager-8684c9ff49-mkv9c\" (UID: \"28a69e4b-001d-48b2-aec3-6765f7e179b6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mkv9c" Jul 21 01:43:24.348386 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:24.348344 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sqkjk" Jul 21 01:43:24.396067 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:24.396033 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-8fdf8bfcb-qdwmd" Jul 21 01:43:24.396259 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:24.396108 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8fdf8bfcb-qdwmd" Jul 21 01:43:24.402682 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:24.402650 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-8fdf8bfcb-qdwmd" Jul 21 01:43:24.404882 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:43:24.404844 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-db4ck\" is forbidden: User \"system:node:ip-10-0-139-183.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-183.ec2.internal' and this object" podUID="42789c7f-7215-4f03-896f-67b73cbd82dd" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-db4ck" Jul 21 01:43:24.406597 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:43:24.406558 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-gb22k\" is forbidden: User \"system:node:ip-10-0-139-183.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-183.ec2.internal' and this object" podUID="3faa50b6-8d97-43a1-844d-91962b724e8f" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gb22k" Jul 21 01:43:24.436402 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:24.436363 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mkv9c" Jul 21 01:43:25.154166 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:25.154138 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-8fdf8bfcb-qdwmd" Jul 21 01:43:25.204567 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:25.204532 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8dd66cfd6-9gbmt"] Jul 21 01:43:36.780222 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:36.780153 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-db4ck" Jul 21 01:43:36.782417 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:43:36.782357 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-db4ck\" is forbidden: User \"system:node:ip-10-0-139-183.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-183.ec2.internal' and this object" podUID="42789c7f-7215-4f03-896f-67b73cbd82dd" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-db4ck" Jul 21 01:43:36.784328 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:36.784305 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gb22k" Jul 21 01:43:36.788733 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:43:36.788632 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-db4ck\" is forbidden: User \"system:node:ip-10-0-139-183.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-183.ec2.internal' and this object" podUID="42789c7f-7215-4f03-896f-67b73cbd82dd" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-db4ck" Jul 21 01:43:36.790512 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:43:36.790486 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-gb22k\" is forbidden: User \"system:node:ip-10-0-139-183.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-183.ec2.internal' and this object" podUID="3faa50b6-8d97-43a1-844d-91962b724e8f" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gb22k" Jul 21 01:43:36.818220 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:36.818019 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mkv9c"] Jul 21 01:43:36.821233 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:43:36.821208 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28a69e4b_001d_48b2_aec3_6765f7e179b6.slice/crio-b6060a11eec06f1589c6e6e0cc531bd36eb5e749d063f567a78b55a503c3122b WatchSource:0}: Error finding container b6060a11eec06f1589c6e6e0cc531bd36eb5e749d063f567a78b55a503c3122b: Status 404 returned error can't find the container with id b6060a11eec06f1589c6e6e0cc531bd36eb5e749d063f567a78b55a503c3122b Jul 21 01:43:36.831254 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:36.831231 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sqkjk"] Jul 21 01:43:36.834187 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:43:36.834168 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbdfd33d_b0c4_4993_8b64_adee4d6b9f7d.slice/crio-4b4f932360de0b928e7b28a00e78418f310f3d0225c76d2ddff8967fc5643ebd WatchSource:0}: Error finding container 4b4f932360de0b928e7b28a00e78418f310f3d0225c76d2ddff8967fc5643ebd: Status 404 returned error can't find the container with id 4b4f932360de0b928e7b28a00e78418f310f3d0225c76d2ddff8967fc5643ebd Jul 21 01:43:36.874210 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:36.874182 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mr9sq\" (UniqueName: \"kubernetes.io/projected/42789c7f-7215-4f03-896f-67b73cbd82dd-kube-api-access-mr9sq\") pod \"42789c7f-7215-4f03-896f-67b73cbd82dd\" (UID: \"42789c7f-7215-4f03-896f-67b73cbd82dd\") " Jul 21 01:43:36.874319 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:36.874303 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/42789c7f-7215-4f03-896f-67b73cbd82dd-extensions-socket-volume\") pod \"42789c7f-7215-4f03-896f-67b73cbd82dd\" (UID: \"42789c7f-7215-4f03-896f-67b73cbd82dd\") " Jul 21 01:43:36.874390 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:36.874374 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wq2qt\" (UniqueName: \"kubernetes.io/projected/3faa50b6-8d97-43a1-844d-91962b724e8f-kube-api-access-wq2qt\") pod \"3faa50b6-8d97-43a1-844d-91962b724e8f\" (UID: \"3faa50b6-8d97-43a1-844d-91962b724e8f\") " Jul 21 01:43:36.874811 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:36.874770 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42789c7f-7215-4f03-896f-67b73cbd82dd-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "42789c7f-7215-4f03-896f-67b73cbd82dd" (UID: "42789c7f-7215-4f03-896f-67b73cbd82dd"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 01:43:36.876272 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:43:36.876145 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-gb22k\" is forbidden: User \"system:node:ip-10-0-139-183.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-183.ec2.internal' and this object" podUID="3faa50b6-8d97-43a1-844d-91962b724e8f" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gb22k" Jul 21 01:43:36.876882 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:36.876857 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42789c7f-7215-4f03-896f-67b73cbd82dd-kube-api-access-mr9sq" (OuterVolumeSpecName: "kube-api-access-mr9sq") pod "42789c7f-7215-4f03-896f-67b73cbd82dd" (UID: "42789c7f-7215-4f03-896f-67b73cbd82dd"). InnerVolumeSpecName "kube-api-access-mr9sq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 01:43:36.876944 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:36.876894 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3faa50b6-8d97-43a1-844d-91962b724e8f-kube-api-access-wq2qt" (OuterVolumeSpecName: "kube-api-access-wq2qt") pod "3faa50b6-8d97-43a1-844d-91962b724e8f" (UID: "3faa50b6-8d97-43a1-844d-91962b724e8f"). InnerVolumeSpecName "kube-api-access-wq2qt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 01:43:36.877716 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:43:36.877686 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-db4ck\" is forbidden: User \"system:node:ip-10-0-139-183.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-183.ec2.internal' and this object" podUID="42789c7f-7215-4f03-896f-67b73cbd82dd" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-db4ck" Jul 21 01:43:36.879414 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:36.879395 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3faa50b6-8d97-43a1-844d-91962b724e8f" path="/var/lib/kubelet/pods/3faa50b6-8d97-43a1-844d-91962b724e8f/volumes" Jul 21 01:43:36.879780 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:36.879756 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42789c7f-7215-4f03-896f-67b73cbd82dd" path="/var/lib/kubelet/pods/42789c7f-7215-4f03-896f-67b73cbd82dd/volumes" Jul 21 01:43:36.975946 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:36.975911 2580 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/42789c7f-7215-4f03-896f-67b73cbd82dd-extensions-socket-volume\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 21 01:43:36.975946 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:36.975948 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wq2qt\" (UniqueName: \"kubernetes.io/projected/3faa50b6-8d97-43a1-844d-91962b724e8f-kube-api-access-wq2qt\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 21 01:43:36.976139 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:36.975964 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mr9sq\" (UniqueName: \"kubernetes.io/projected/42789c7f-7215-4f03-896f-67b73cbd82dd-kube-api-access-mr9sq\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 21 01:43:37.246011 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:37.245426 2580 generic.go:356] "Generic (PLEG): container finished" podID="42789c7f-7215-4f03-896f-67b73cbd82dd" containerID="a8c8eb75d4e4eee1e86b78adea372bca5953f2bb6215bdee39c3a106dc202dc1" exitCode=0 Jul 21 01:43:37.246011 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:37.245568 2580 scope.go:117] "RemoveContainer" containerID="a8c8eb75d4e4eee1e86b78adea372bca5953f2bb6215bdee39c3a106dc202dc1" Jul 21 01:43:37.246011 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:37.245830 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-db4ck" Jul 21 01:43:37.247905 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:43:37.247867 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-db4ck\" is forbidden: User \"system:node:ip-10-0-139-183.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-183.ec2.internal' and this object" podUID="42789c7f-7215-4f03-896f-67b73cbd82dd" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-db4ck" Jul 21 01:43:37.249120 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:37.249091 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xtgcn" event={"ID":"0a12d9c8-e70f-4f03-8465-926cdc4645f5","Type":"ContainerStarted","Data":"e711d344ec436c3906d3de69dff374749b28a357b77a69a2c7bd5f5da7ed8638"} Jul 21 01:43:37.250902 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:37.250872 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mkv9c" event={"ID":"28a69e4b-001d-48b2-aec3-6765f7e179b6","Type":"ContainerStarted","Data":"ec17651fe18f3673b8995f0a0e70ab2395542d561308a237aebaf12589cf1109"} Jul 21 01:43:37.251006 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:37.250952 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mkv9c" event={"ID":"28a69e4b-001d-48b2-aec3-6765f7e179b6","Type":"ContainerStarted","Data":"b6060a11eec06f1589c6e6e0cc531bd36eb5e749d063f567a78b55a503c3122b"} Jul 21 01:43:37.251006 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:43:37.250871 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-db4ck\" is forbidden: User \"system:node:ip-10-0-139-183.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-183.ec2.internal' and this object" podUID="42789c7f-7215-4f03-896f-67b73cbd82dd" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-db4ck" Jul 21 01:43:37.252793 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:37.252770 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sqkjk" event={"ID":"dbdfd33d-b0c4-4993-8b64-adee4d6b9f7d","Type":"ContainerStarted","Data":"f82029b1de6f4b3f2c939bd58d73a10dce12787dfc761a227143587b6b42ade8"} Jul 21 01:43:37.252898 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:37.252813 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sqkjk" event={"ID":"dbdfd33d-b0c4-4993-8b64-adee4d6b9f7d","Type":"ContainerStarted","Data":"4b4f932360de0b928e7b28a00e78418f310f3d0225c76d2ddff8967fc5643ebd"} Jul 21 01:43:37.253069 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:37.253055 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sqkjk" Jul 21 01:43:37.254759 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:37.254738 2580 generic.go:356] "Generic (PLEG): container finished" podID="3faa50b6-8d97-43a1-844d-91962b724e8f" containerID="bca458d627abd46c8520548563f212402c3aa297fe550a1d67e06964a10e9d8f" exitCode=0 Jul 21 01:43:37.254866 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:37.254791 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gb22k" Jul 21 01:43:37.257211 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:37.257196 2580 scope.go:117] "RemoveContainer" containerID="a8c8eb75d4e4eee1e86b78adea372bca5953f2bb6215bdee39c3a106dc202dc1" Jul 21 01:43:37.257495 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:43:37.257474 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8c8eb75d4e4eee1e86b78adea372bca5953f2bb6215bdee39c3a106dc202dc1\": container with ID starting with a8c8eb75d4e4eee1e86b78adea372bca5953f2bb6215bdee39c3a106dc202dc1 not found: ID does not exist" containerID="a8c8eb75d4e4eee1e86b78adea372bca5953f2bb6215bdee39c3a106dc202dc1" Jul 21 01:43:37.257596 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:37.257580 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8c8eb75d4e4eee1e86b78adea372bca5953f2bb6215bdee39c3a106dc202dc1"} err="failed to get container status \"a8c8eb75d4e4eee1e86b78adea372bca5953f2bb6215bdee39c3a106dc202dc1\": rpc error: code = NotFound desc = could not find container \"a8c8eb75d4e4eee1e86b78adea372bca5953f2bb6215bdee39c3a106dc202dc1\": container with ID starting with a8c8eb75d4e4eee1e86b78adea372bca5953f2bb6215bdee39c3a106dc202dc1 not found: ID does not exist" Jul 21 01:43:37.257660 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:37.257653 2580 scope.go:117] "RemoveContainer" containerID="bca458d627abd46c8520548563f212402c3aa297fe550a1d67e06964a10e9d8f" Jul 21 01:43:37.265974 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:37.265953 2580 scope.go:117] "RemoveContainer" containerID="bca458d627abd46c8520548563f212402c3aa297fe550a1d67e06964a10e9d8f" Jul 21 01:43:37.266268 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:43:37.266243 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bca458d627abd46c8520548563f212402c3aa297fe550a1d67e06964a10e9d8f\": container with ID starting with bca458d627abd46c8520548563f212402c3aa297fe550a1d67e06964a10e9d8f not found: ID does not exist" containerID="bca458d627abd46c8520548563f212402c3aa297fe550a1d67e06964a10e9d8f" Jul 21 01:43:37.266315 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:37.266273 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bca458d627abd46c8520548563f212402c3aa297fe550a1d67e06964a10e9d8f"} err="failed to get container status \"bca458d627abd46c8520548563f212402c3aa297fe550a1d67e06964a10e9d8f\": rpc error: code = NotFound desc = could not find container \"bca458d627abd46c8520548563f212402c3aa297fe550a1d67e06964a10e9d8f\": container with ID starting with bca458d627abd46c8520548563f212402c3aa297fe550a1d67e06964a10e9d8f not found: ID does not exist" Jul 21 01:43:37.266524 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:37.266486 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xtgcn" podStartSLOduration=1.222785662 podStartE2EDuration="24.266445568s" totalImagesPullingTime="23.043659906s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 01:43:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:43:37.266058597 +0000 UTC m=+440.853224575" watchObservedRunningTime="2026-07-21 01:43:37.266445568 +0000 UTC m=+440.853611548" Jul 21 01:43:37.267690 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:43:37.267666 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-db4ck\" is forbidden: User \"system:node:ip-10-0-139-183.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-183.ec2.internal' and this object" podUID="42789c7f-7215-4f03-896f-67b73cbd82dd" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-db4ck" Jul 21 01:43:37.294421 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:37.294363 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sqkjk" podStartSLOduration=14.294342727 podStartE2EDuration="14.294342727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 01:43:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:43:37.293866394 +0000 UTC m=+440.881032374" watchObservedRunningTime="2026-07-21 01:43:37.294342727 +0000 UTC m=+440.881508708" Jul 21 01:43:37.295569 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:43:37.295531 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-db4ck\" is forbidden: User \"system:node:ip-10-0-139-183.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-183.ec2.internal' and this object" podUID="42789c7f-7215-4f03-896f-67b73cbd82dd" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-db4ck" Jul 21 01:43:37.297207 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:43:37.297183 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-gb22k\" is forbidden: User \"system:node:ip-10-0-139-183.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-183.ec2.internal' and this object" podUID="3faa50b6-8d97-43a1-844d-91962b724e8f" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gb22k" Jul 21 01:43:37.316022 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:37.315969 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mkv9c" podStartSLOduration=13.315951467 podStartE2EDuration="13.315951467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 01:43:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:43:37.313740053 +0000 UTC m=+440.900906033" watchObservedRunningTime="2026-07-21 01:43:37.315951467 +0000 UTC m=+440.903117444" Jul 21 01:43:38.261685 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:38.261655 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mkv9c" Jul 21 01:43:39.166794 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:43:39.166758 2580 log.go:32] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-6fb7879bc9-nnnng_kuadrant-system_e11d7ab7-2dcc-4357-aca4-af93f4a8531b_0(15cd255c91090419f8f70a4a38ae328e851d18d7bab1fd4454791c5470e99289): error adding pod kuadrant-system_kuadrant-operator-controller-manager-6fb7879bc9-nnnng to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}" Jul 21 01:43:39.166860 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:43:39.166826 2580 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-6fb7879bc9-nnnng_kuadrant-system_e11d7ab7-2dcc-4357-aca4-af93f4a8531b_0(15cd255c91090419f8f70a4a38ae328e851d18d7bab1fd4454791c5470e99289): error adding pod kuadrant-system_kuadrant-operator-controller-manager-6fb7879bc9-nnnng to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-nnnng" Jul 21 01:43:39.266132 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:39.266096 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-nnnng" Jul 21 01:43:39.268382 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:43:39.268357 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-nnnng\" is forbidden: User \"system:node:ip-10-0-139-183.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-183.ec2.internal' and this object" podUID="e11d7ab7-2dcc-4357-aca4-af93f4a8531b" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-nnnng" Jul 21 01:43:39.273121 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:39.273101 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-nnnng" Jul 21 01:43:39.274913 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:43:39.274891 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-nnnng\" is forbidden: User \"system:node:ip-10-0-139-183.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-183.ec2.internal' and this object" podUID="e11d7ab7-2dcc-4357-aca4-af93f4a8531b" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-nnnng" Jul 21 01:43:39.396613 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:39.396585 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e11d7ab7-2dcc-4357-aca4-af93f4a8531b-extensions-socket-volume\") pod \"e11d7ab7-2dcc-4357-aca4-af93f4a8531b\" (UID: \"e11d7ab7-2dcc-4357-aca4-af93f4a8531b\") " Jul 21 01:43:39.396785 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:39.396631 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gfqp9\" (UniqueName: \"kubernetes.io/projected/e11d7ab7-2dcc-4357-aca4-af93f4a8531b-kube-api-access-gfqp9\") pod \"e11d7ab7-2dcc-4357-aca4-af93f4a8531b\" (UID: \"e11d7ab7-2dcc-4357-aca4-af93f4a8531b\") " Jul 21 01:43:39.396869 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:39.396847 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e11d7ab7-2dcc-4357-aca4-af93f4a8531b-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "e11d7ab7-2dcc-4357-aca4-af93f4a8531b" (UID: "e11d7ab7-2dcc-4357-aca4-af93f4a8531b"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 01:43:39.397065 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:39.397037 2580 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e11d7ab7-2dcc-4357-aca4-af93f4a8531b-extensions-socket-volume\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 21 01:43:39.398820 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:39.398798 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e11d7ab7-2dcc-4357-aca4-af93f4a8531b-kube-api-access-gfqp9" (OuterVolumeSpecName: "kube-api-access-gfqp9") pod "e11d7ab7-2dcc-4357-aca4-af93f4a8531b" (UID: "e11d7ab7-2dcc-4357-aca4-af93f4a8531b"). InnerVolumeSpecName "kube-api-access-gfqp9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 01:43:39.497529 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:39.497492 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfqp9\" (UniqueName: \"kubernetes.io/projected/e11d7ab7-2dcc-4357-aca4-af93f4a8531b-kube-api-access-gfqp9\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 21 01:43:40.271381 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:40.271349 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-nnnng" Jul 21 01:43:40.273373 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:43:40.273342 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-nnnng\" is forbidden: User \"system:node:ip-10-0-139-183.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-183.ec2.internal' and this object" podUID="e11d7ab7-2dcc-4357-aca4-af93f4a8531b" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-nnnng" Jul 21 01:43:40.278760 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:43:40.278735 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-nnnng\" is forbidden: User \"system:node:ip-10-0-139-183.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-183.ec2.internal' and this object" podUID="e11d7ab7-2dcc-4357-aca4-af93f4a8531b" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-nnnng" Jul 21 01:43:40.877815 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:40.877784 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e11d7ab7-2dcc-4357-aca4-af93f4a8531b" path="/var/lib/kubelet/pods/e11d7ab7-2dcc-4357-aca4-af93f4a8531b/volumes" Jul 21 01:43:48.264025 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:48.263994 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sqkjk" Jul 21 01:43:48.314821 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:48.314790 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mkv9c"] Jul 21 01:43:48.315065 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:48.315020 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mkv9c" podUID="28a69e4b-001d-48b2-aec3-6765f7e179b6" containerName="manager" containerID="cri-o://ec17651fe18f3673b8995f0a0e70ab2395542d561308a237aebaf12589cf1109" gracePeriod=10 Jul 21 01:43:48.317075 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:48.317052 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mkv9c" Jul 21 01:43:48.539558 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:48.539471 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-smdz7"] Jul 21 01:43:48.540096 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:48.540079 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3faa50b6-8d97-43a1-844d-91962b724e8f" containerName="manager" Jul 21 01:43:48.540143 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:48.540101 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3faa50b6-8d97-43a1-844d-91962b724e8f" containerName="manager" Jul 21 01:43:48.540222 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:48.540211 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3faa50b6-8d97-43a1-844d-91962b724e8f" containerName="manager" Jul 21 01:43:48.543063 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:48.543041 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-smdz7" Jul 21 01:43:48.562790 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:48.562604 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-smdz7"] Jul 21 01:43:48.585624 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:48.585597 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mkv9c" Jul 21 01:43:48.683773 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:48.683743 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n88jx\" (UniqueName: \"kubernetes.io/projected/28a69e4b-001d-48b2-aec3-6765f7e179b6-kube-api-access-n88jx\") pod \"28a69e4b-001d-48b2-aec3-6765f7e179b6\" (UID: \"28a69e4b-001d-48b2-aec3-6765f7e179b6\") " Jul 21 01:43:48.683940 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:48.683815 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/28a69e4b-001d-48b2-aec3-6765f7e179b6-extensions-socket-volume\") pod \"28a69e4b-001d-48b2-aec3-6765f7e179b6\" (UID: \"28a69e4b-001d-48b2-aec3-6765f7e179b6\") " Jul 21 01:43:48.684082 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:48.684061 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e0ff7d70-51eb-4155-8c32-6d1173257ae5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-smdz7\" (UID: \"e0ff7d70-51eb-4155-8c32-6d1173257ae5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-smdz7" Jul 21 01:43:48.684145 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:48.684108 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmkgg\" (UniqueName: \"kubernetes.io/projected/e0ff7d70-51eb-4155-8c32-6d1173257ae5-kube-api-access-cmkgg\") pod \"kuadrant-operator-controller-manager-86878c769f-smdz7\" (UID: \"e0ff7d70-51eb-4155-8c32-6d1173257ae5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-smdz7" Jul 21 01:43:48.684194 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:48.684164 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28a69e4b-001d-48b2-aec3-6765f7e179b6-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "28a69e4b-001d-48b2-aec3-6765f7e179b6" (UID: "28a69e4b-001d-48b2-aec3-6765f7e179b6"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 01:43:48.684245 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:48.684230 2580 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/28a69e4b-001d-48b2-aec3-6765f7e179b6-extensions-socket-volume\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 21 01:43:48.685847 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:48.685828 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28a69e4b-001d-48b2-aec3-6765f7e179b6-kube-api-access-n88jx" (OuterVolumeSpecName: "kube-api-access-n88jx") pod "28a69e4b-001d-48b2-aec3-6765f7e179b6" (UID: "28a69e4b-001d-48b2-aec3-6765f7e179b6"). InnerVolumeSpecName "kube-api-access-n88jx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 01:43:48.784868 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:48.784828 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e0ff7d70-51eb-4155-8c32-6d1173257ae5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-smdz7\" (UID: \"e0ff7d70-51eb-4155-8c32-6d1173257ae5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-smdz7" Jul 21 01:43:48.784868 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:48.784871 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmkgg\" (UniqueName: \"kubernetes.io/projected/e0ff7d70-51eb-4155-8c32-6d1173257ae5-kube-api-access-cmkgg\") pod \"kuadrant-operator-controller-manager-86878c769f-smdz7\" (UID: \"e0ff7d70-51eb-4155-8c32-6d1173257ae5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-smdz7" Jul 21 01:43:48.785096 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:48.785000 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n88jx\" (UniqueName: \"kubernetes.io/projected/28a69e4b-001d-48b2-aec3-6765f7e179b6-kube-api-access-n88jx\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 21 01:43:48.785237 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:48.785217 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e0ff7d70-51eb-4155-8c32-6d1173257ae5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-smdz7\" (UID: \"e0ff7d70-51eb-4155-8c32-6d1173257ae5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-smdz7" Jul 21 01:43:48.793118 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:48.793052 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmkgg\" (UniqueName: \"kubernetes.io/projected/e0ff7d70-51eb-4155-8c32-6d1173257ae5-kube-api-access-cmkgg\") pod \"kuadrant-operator-controller-manager-86878c769f-smdz7\" (UID: \"e0ff7d70-51eb-4155-8c32-6d1173257ae5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-smdz7" Jul 21 01:43:48.854576 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:48.854532 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-smdz7" Jul 21 01:43:48.997859 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:48.997828 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-smdz7"] Jul 21 01:43:49.001490 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:43:49.001432 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0ff7d70_51eb_4155_8c32_6d1173257ae5.slice/crio-401bbb7d800f0257e24258980397733ede618875edca8777653fe5e66471d407 WatchSource:0}: Error finding container 401bbb7d800f0257e24258980397733ede618875edca8777653fe5e66471d407: Status 404 returned error can't find the container with id 401bbb7d800f0257e24258980397733ede618875edca8777653fe5e66471d407 Jul 21 01:43:49.320246 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:49.320158 2580 generic.go:356] "Generic (PLEG): container finished" podID="28a69e4b-001d-48b2-aec3-6765f7e179b6" containerID="ec17651fe18f3673b8995f0a0e70ab2395542d561308a237aebaf12589cf1109" exitCode=0 Jul 21 01:43:49.320246 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:49.320199 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mkv9c" Jul 21 01:43:49.320740 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:49.320248 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mkv9c" event={"ID":"28a69e4b-001d-48b2-aec3-6765f7e179b6","Type":"ContainerDied","Data":"ec17651fe18f3673b8995f0a0e70ab2395542d561308a237aebaf12589cf1109"} Jul 21 01:43:49.320740 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:49.320299 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mkv9c" event={"ID":"28a69e4b-001d-48b2-aec3-6765f7e179b6","Type":"ContainerDied","Data":"b6060a11eec06f1589c6e6e0cc531bd36eb5e749d063f567a78b55a503c3122b"} Jul 21 01:43:49.320740 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:49.320322 2580 scope.go:117] "RemoveContainer" containerID="ec17651fe18f3673b8995f0a0e70ab2395542d561308a237aebaf12589cf1109" Jul 21 01:43:49.322035 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:49.322010 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-smdz7" event={"ID":"e0ff7d70-51eb-4155-8c32-6d1173257ae5","Type":"ContainerStarted","Data":"31d37d063f5d8f397b419f63e09185c67c0b1711f71d36b11beb1c647db301f2"} Jul 21 01:43:49.322148 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:49.322058 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-smdz7" event={"ID":"e0ff7d70-51eb-4155-8c32-6d1173257ae5","Type":"ContainerStarted","Data":"401bbb7d800f0257e24258980397733ede618875edca8777653fe5e66471d407"} Jul 21 01:43:49.322299 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:49.322280 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-smdz7" Jul 21 01:43:49.339812 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:49.339779 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mkv9c"] Jul 21 01:43:49.340495 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:49.340370 2580 scope.go:117] "RemoveContainer" containerID="ec17651fe18f3673b8995f0a0e70ab2395542d561308a237aebaf12589cf1109" Jul 21 01:43:49.340769 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:43:49.340748 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec17651fe18f3673b8995f0a0e70ab2395542d561308a237aebaf12589cf1109\": container with ID starting with ec17651fe18f3673b8995f0a0e70ab2395542d561308a237aebaf12589cf1109 not found: ID does not exist" containerID="ec17651fe18f3673b8995f0a0e70ab2395542d561308a237aebaf12589cf1109" Jul 21 01:43:49.340838 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:49.340777 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec17651fe18f3673b8995f0a0e70ab2395542d561308a237aebaf12589cf1109"} err="failed to get container status \"ec17651fe18f3673b8995f0a0e70ab2395542d561308a237aebaf12589cf1109\": rpc error: code = NotFound desc = could not find container \"ec17651fe18f3673b8995f0a0e70ab2395542d561308a237aebaf12589cf1109\": container with ID starting with ec17651fe18f3673b8995f0a0e70ab2395542d561308a237aebaf12589cf1109 not found: ID does not exist" Jul 21 01:43:49.342509 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:49.342486 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mkv9c"] Jul 21 01:43:49.357599 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:49.357552 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-smdz7" podStartSLOduration=1.357534635 podStartE2EDuration="1.357534635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 01:43:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:43:49.355942898 +0000 UTC m=+452.943108876" watchObservedRunningTime="2026-07-21 01:43:49.357534635 +0000 UTC m=+452.944700612" Jul 21 01:43:50.233131 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:50.233089 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-8dd66cfd6-9gbmt" podUID="6796addb-d850-411b-80db-84e97535ca94" containerName="console" containerID="cri-o://dcc1038581097533a22b68e48cbacecf8edbb00fb4a4022add2e1be14aab3abe" gracePeriod=15 Jul 21 01:43:50.488794 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:50.488732 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8dd66cfd6-9gbmt" Jul 21 01:43:50.600919 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:50.600886 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bm848\" (UniqueName: \"kubernetes.io/projected/6796addb-d850-411b-80db-84e97535ca94-kube-api-access-bm848\") pod \"6796addb-d850-411b-80db-84e97535ca94\" (UID: \"6796addb-d850-411b-80db-84e97535ca94\") " Jul 21 01:43:50.600919 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:50.600925 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6796addb-d850-411b-80db-84e97535ca94-trusted-ca-bundle\") pod \"6796addb-d850-411b-80db-84e97535ca94\" (UID: \"6796addb-d850-411b-80db-84e97535ca94\") " Jul 21 01:43:50.601140 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:50.600963 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6796addb-d850-411b-80db-84e97535ca94-console-oauth-config\") pod \"6796addb-d850-411b-80db-84e97535ca94\" (UID: \"6796addb-d850-411b-80db-84e97535ca94\") " Jul 21 01:43:50.601140 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:50.601029 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6796addb-d850-411b-80db-84e97535ca94-oauth-serving-cert\") pod \"6796addb-d850-411b-80db-84e97535ca94\" (UID: \"6796addb-d850-411b-80db-84e97535ca94\") " Jul 21 01:43:50.601140 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:50.601054 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6796addb-d850-411b-80db-84e97535ca94-console-config\") pod \"6796addb-d850-411b-80db-84e97535ca94\" (UID: \"6796addb-d850-411b-80db-84e97535ca94\") " Jul 21 01:43:50.601140 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:50.601080 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6796addb-d850-411b-80db-84e97535ca94-console-serving-cert\") pod \"6796addb-d850-411b-80db-84e97535ca94\" (UID: \"6796addb-d850-411b-80db-84e97535ca94\") " Jul 21 01:43:50.601140 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:50.601125 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6796addb-d850-411b-80db-84e97535ca94-service-ca\") pod \"6796addb-d850-411b-80db-84e97535ca94\" (UID: \"6796addb-d850-411b-80db-84e97535ca94\") " Jul 21 01:43:50.601478 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:50.601426 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6796addb-d850-411b-80db-84e97535ca94-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "6796addb-d850-411b-80db-84e97535ca94" (UID: "6796addb-d850-411b-80db-84e97535ca94"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 01:43:50.601560 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:50.601476 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6796addb-d850-411b-80db-84e97535ca94-console-config" (OuterVolumeSpecName: "console-config") pod "6796addb-d850-411b-80db-84e97535ca94" (UID: "6796addb-d850-411b-80db-84e97535ca94"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 01:43:50.601560 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:50.601507 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6796addb-d850-411b-80db-84e97535ca94-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "6796addb-d850-411b-80db-84e97535ca94" (UID: "6796addb-d850-411b-80db-84e97535ca94"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 01:43:50.601751 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:50.601728 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6796addb-d850-411b-80db-84e97535ca94-service-ca" (OuterVolumeSpecName: "service-ca") pod "6796addb-d850-411b-80db-84e97535ca94" (UID: "6796addb-d850-411b-80db-84e97535ca94"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 01:43:50.603133 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:50.603107 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6796addb-d850-411b-80db-84e97535ca94-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "6796addb-d850-411b-80db-84e97535ca94" (UID: "6796addb-d850-411b-80db-84e97535ca94"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 01:43:50.603133 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:50.603123 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6796addb-d850-411b-80db-84e97535ca94-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "6796addb-d850-411b-80db-84e97535ca94" (UID: "6796addb-d850-411b-80db-84e97535ca94"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 01:43:50.603288 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:50.603228 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6796addb-d850-411b-80db-84e97535ca94-kube-api-access-bm848" (OuterVolumeSpecName: "kube-api-access-bm848") pod "6796addb-d850-411b-80db-84e97535ca94" (UID: "6796addb-d850-411b-80db-84e97535ca94"). InnerVolumeSpecName "kube-api-access-bm848". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 01:43:50.702552 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:50.702519 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bm848\" (UniqueName: \"kubernetes.io/projected/6796addb-d850-411b-80db-84e97535ca94-kube-api-access-bm848\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 21 01:43:50.702552 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:50.702544 2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6796addb-d850-411b-80db-84e97535ca94-trusted-ca-bundle\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 21 01:43:50.702552 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:50.702553 2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6796addb-d850-411b-80db-84e97535ca94-console-oauth-config\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 21 01:43:50.703002 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:50.702562 2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6796addb-d850-411b-80db-84e97535ca94-oauth-serving-cert\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 21 01:43:50.703002 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:50.702571 2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6796addb-d850-411b-80db-84e97535ca94-console-config\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 21 01:43:50.703002 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:50.702579 2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6796addb-d850-411b-80db-84e97535ca94-console-serving-cert\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 21 01:43:50.703002 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:50.702587 2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6796addb-d850-411b-80db-84e97535ca94-service-ca\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 21 01:43:50.877199 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:50.877161 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28a69e4b-001d-48b2-aec3-6765f7e179b6" path="/var/lib/kubelet/pods/28a69e4b-001d-48b2-aec3-6765f7e179b6/volumes" Jul 21 01:43:51.338797 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:51.338767 2580 generic.go:356] "Generic (PLEG): container finished" podID="6796addb-d850-411b-80db-84e97535ca94" containerID="dcc1038581097533a22b68e48cbacecf8edbb00fb4a4022add2e1be14aab3abe" exitCode=0 Jul 21 01:43:51.338965 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:51.338821 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8dd66cfd6-9gbmt" Jul 21 01:43:51.338965 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:51.338852 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8dd66cfd6-9gbmt" event={"ID":"6796addb-d850-411b-80db-84e97535ca94","Type":"ContainerDied","Data":"dcc1038581097533a22b68e48cbacecf8edbb00fb4a4022add2e1be14aab3abe"} Jul 21 01:43:51.338965 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:51.338910 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8dd66cfd6-9gbmt" event={"ID":"6796addb-d850-411b-80db-84e97535ca94","Type":"ContainerDied","Data":"99fbdf7efa16cbd518295c0d38574ff4da06be558d70c09d03c26cd54257bcca"} Jul 21 01:43:51.338965 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:51.338932 2580 scope.go:117] "RemoveContainer" containerID="dcc1038581097533a22b68e48cbacecf8edbb00fb4a4022add2e1be14aab3abe" Jul 21 01:43:51.347910 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:51.347894 2580 scope.go:117] "RemoveContainer" containerID="dcc1038581097533a22b68e48cbacecf8edbb00fb4a4022add2e1be14aab3abe" Jul 21 01:43:51.348156 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:43:51.348139 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcc1038581097533a22b68e48cbacecf8edbb00fb4a4022add2e1be14aab3abe\": container with ID starting with dcc1038581097533a22b68e48cbacecf8edbb00fb4a4022add2e1be14aab3abe not found: ID does not exist" containerID="dcc1038581097533a22b68e48cbacecf8edbb00fb4a4022add2e1be14aab3abe" Jul 21 01:43:51.348204 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:51.348163 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcc1038581097533a22b68e48cbacecf8edbb00fb4a4022add2e1be14aab3abe"} err="failed to get container status \"dcc1038581097533a22b68e48cbacecf8edbb00fb4a4022add2e1be14aab3abe\": rpc error: code = NotFound desc = could not find container \"dcc1038581097533a22b68e48cbacecf8edbb00fb4a4022add2e1be14aab3abe\": container with ID starting with dcc1038581097533a22b68e48cbacecf8edbb00fb4a4022add2e1be14aab3abe not found: ID does not exist" Jul 21 01:43:51.356736 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:51.356708 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8dd66cfd6-9gbmt"] Jul 21 01:43:51.358701 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:51.358679 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-8dd66cfd6-9gbmt"] Jul 21 01:43:52.877671 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:43:52.877634 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6796addb-d850-411b-80db-84e97535ca94" path="/var/lib/kubelet/pods/6796addb-d850-411b-80db-84e97535ca94/volumes" Jul 21 01:44:00.331397 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:00.331365 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-smdz7" Jul 21 01:44:00.381003 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:00.380974 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sqkjk"] Jul 21 01:44:00.381204 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:00.381178 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sqkjk" podUID="dbdfd33d-b0c4-4993-8b64-adee4d6b9f7d" containerName="manager" containerID="cri-o://f82029b1de6f4b3f2c939bd58d73a10dce12787dfc761a227143587b6b42ade8" gracePeriod=10 Jul 21 01:44:01.185159 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:01.185135 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sqkjk" Jul 21 01:44:01.296028 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:01.295916 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dbdfd33d-b0c4-4993-8b64-adee4d6b9f7d-extensions-socket-volume\") pod \"dbdfd33d-b0c4-4993-8b64-adee4d6b9f7d\" (UID: \"dbdfd33d-b0c4-4993-8b64-adee4d6b9f7d\") " Jul 21 01:44:01.296028 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:01.295982 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-88q8z\" (UniqueName: \"kubernetes.io/projected/dbdfd33d-b0c4-4993-8b64-adee4d6b9f7d-kube-api-access-88q8z\") pod \"dbdfd33d-b0c4-4993-8b64-adee4d6b9f7d\" (UID: \"dbdfd33d-b0c4-4993-8b64-adee4d6b9f7d\") " Jul 21 01:44:01.296333 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:01.296310 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbdfd33d-b0c4-4993-8b64-adee4d6b9f7d-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "dbdfd33d-b0c4-4993-8b64-adee4d6b9f7d" (UID: "dbdfd33d-b0c4-4993-8b64-adee4d6b9f7d"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 01:44:01.298097 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:01.298074 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbdfd33d-b0c4-4993-8b64-adee4d6b9f7d-kube-api-access-88q8z" (OuterVolumeSpecName: "kube-api-access-88q8z") pod "dbdfd33d-b0c4-4993-8b64-adee4d6b9f7d" (UID: "dbdfd33d-b0c4-4993-8b64-adee4d6b9f7d"). InnerVolumeSpecName "kube-api-access-88q8z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 01:44:01.393650 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:01.393616 2580 generic.go:356] "Generic (PLEG): container finished" podID="dbdfd33d-b0c4-4993-8b64-adee4d6b9f7d" containerID="f82029b1de6f4b3f2c939bd58d73a10dce12787dfc761a227143587b6b42ade8" exitCode=0 Jul 21 01:44:01.394082 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:01.393663 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sqkjk" Jul 21 01:44:01.394082 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:01.393690 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sqkjk" event={"ID":"dbdfd33d-b0c4-4993-8b64-adee4d6b9f7d","Type":"ContainerDied","Data":"f82029b1de6f4b3f2c939bd58d73a10dce12787dfc761a227143587b6b42ade8"} Jul 21 01:44:01.394082 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:01.393732 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sqkjk" event={"ID":"dbdfd33d-b0c4-4993-8b64-adee4d6b9f7d","Type":"ContainerDied","Data":"4b4f932360de0b928e7b28a00e78418f310f3d0225c76d2ddff8967fc5643ebd"} Jul 21 01:44:01.394082 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:01.393753 2580 scope.go:117] "RemoveContainer" containerID="f82029b1de6f4b3f2c939bd58d73a10dce12787dfc761a227143587b6b42ade8" Jul 21 01:44:01.396865 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:01.396842 2580 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dbdfd33d-b0c4-4993-8b64-adee4d6b9f7d-extensions-socket-volume\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 21 01:44:01.396865 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:01.396864 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88q8z\" (UniqueName: \"kubernetes.io/projected/dbdfd33d-b0c4-4993-8b64-adee4d6b9f7d-kube-api-access-88q8z\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 21 01:44:01.403279 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:01.403262 2580 scope.go:117] "RemoveContainer" containerID="f82029b1de6f4b3f2c939bd58d73a10dce12787dfc761a227143587b6b42ade8" Jul 21 01:44:01.403587 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:44:01.403569 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f82029b1de6f4b3f2c939bd58d73a10dce12787dfc761a227143587b6b42ade8\": container with ID starting with f82029b1de6f4b3f2c939bd58d73a10dce12787dfc761a227143587b6b42ade8 not found: ID does not exist" containerID="f82029b1de6f4b3f2c939bd58d73a10dce12787dfc761a227143587b6b42ade8" Jul 21 01:44:01.403646 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:01.403595 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f82029b1de6f4b3f2c939bd58d73a10dce12787dfc761a227143587b6b42ade8"} err="failed to get container status \"f82029b1de6f4b3f2c939bd58d73a10dce12787dfc761a227143587b6b42ade8\": rpc error: code = NotFound desc = could not find container \"f82029b1de6f4b3f2c939bd58d73a10dce12787dfc761a227143587b6b42ade8\": container with ID starting with f82029b1de6f4b3f2c939bd58d73a10dce12787dfc761a227143587b6b42ade8 not found: ID does not exist" Jul 21 01:44:01.413803 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:01.413774 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sqkjk"] Jul 21 01:44:01.420273 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:01.420249 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sqkjk"] Jul 21 01:44:02.876912 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:02.876877 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbdfd33d-b0c4-4993-8b64-adee4d6b9f7d" path="/var/lib/kubelet/pods/dbdfd33d-b0c4-4993-8b64-adee4d6b9f7d/volumes" Jul 21 01:44:04.715282 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:04.715245 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-5zsqr"] Jul 21 01:44:04.716314 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:04.715970 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28a69e4b-001d-48b2-aec3-6765f7e179b6" containerName="manager" Jul 21 01:44:04.716314 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:04.715992 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="28a69e4b-001d-48b2-aec3-6765f7e179b6" containerName="manager" Jul 21 01:44:04.716314 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:04.716016 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbdfd33d-b0c4-4993-8b64-adee4d6b9f7d" containerName="manager" Jul 21 01:44:04.716314 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:04.716024 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbdfd33d-b0c4-4993-8b64-adee4d6b9f7d" containerName="manager" Jul 21 01:44:04.716314 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:04.716043 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6796addb-d850-411b-80db-84e97535ca94" containerName="console" Jul 21 01:44:04.716314 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:04.716052 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6796addb-d850-411b-80db-84e97535ca94" containerName="console" Jul 21 01:44:04.716314 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:04.716168 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbdfd33d-b0c4-4993-8b64-adee4d6b9f7d" containerName="manager" Jul 21 01:44:04.716314 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:04.716184 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6796addb-d850-411b-80db-84e97535ca94" containerName="console" Jul 21 01:44:04.716314 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:04.716193 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="28a69e4b-001d-48b2-aec3-6765f7e179b6" containerName="manager" Jul 21 01:44:04.720958 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:04.720937 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-5zsqr" Jul 21 01:44:04.723142 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:04.723119 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 21 01:44:04.723251 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:04.723119 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-zlsb5\"" Jul 21 01:44:04.735710 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:04.735679 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-5zsqr"] Jul 21 01:44:04.830801 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:04.830762 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3cd4166-77e6-4b24-ae7d-b0d823eba80d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-5zsqr\" (UID: \"b3cd4166-77e6-4b24-ae7d-b0d823eba80d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-5zsqr" Jul 21 01:44:04.830801 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:04.830807 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3cd4166-77e6-4b24-ae7d-b0d823eba80d-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-5zsqr\" (UID: \"b3cd4166-77e6-4b24-ae7d-b0d823eba80d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-5zsqr" Jul 21 01:44:04.831014 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:04.830875 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3cd4166-77e6-4b24-ae7d-b0d823eba80d-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-5zsqr\" (UID: \"b3cd4166-77e6-4b24-ae7d-b0d823eba80d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-5zsqr" Jul 21 01:44:04.831014 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:04.830912 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3cd4166-77e6-4b24-ae7d-b0d823eba80d-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-5zsqr\" (UID: \"b3cd4166-77e6-4b24-ae7d-b0d823eba80d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-5zsqr" Jul 21 01:44:04.831014 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:04.830955 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3cd4166-77e6-4b24-ae7d-b0d823eba80d-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-5zsqr\" (UID: \"b3cd4166-77e6-4b24-ae7d-b0d823eba80d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-5zsqr" Jul 21 01:44:04.831014 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:04.831009 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3cd4166-77e6-4b24-ae7d-b0d823eba80d-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-5zsqr\" (UID: \"b3cd4166-77e6-4b24-ae7d-b0d823eba80d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-5zsqr" Jul 21 01:44:04.831185 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:04.831033 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8tmz\" (UniqueName: \"kubernetes.io/projected/b3cd4166-77e6-4b24-ae7d-b0d823eba80d-kube-api-access-f8tmz\") pod \"maas-default-gateway-openshift-default-5896d4dd49-5zsqr\" (UID: \"b3cd4166-77e6-4b24-ae7d-b0d823eba80d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-5zsqr" Jul 21 01:44:04.831185 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:04.831086 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3cd4166-77e6-4b24-ae7d-b0d823eba80d-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-5zsqr\" (UID: \"b3cd4166-77e6-4b24-ae7d-b0d823eba80d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-5zsqr" Jul 21 01:44:04.831185 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:04.831110 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3cd4166-77e6-4b24-ae7d-b0d823eba80d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-5zsqr\" (UID: \"b3cd4166-77e6-4b24-ae7d-b0d823eba80d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-5zsqr" Jul 21 01:44:04.931905 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:04.931859 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3cd4166-77e6-4b24-ae7d-b0d823eba80d-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-5zsqr\" (UID: \"b3cd4166-77e6-4b24-ae7d-b0d823eba80d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-5zsqr" Jul 21 01:44:04.932113 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:04.931918 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3cd4166-77e6-4b24-ae7d-b0d823eba80d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-5zsqr\" (UID: \"b3cd4166-77e6-4b24-ae7d-b0d823eba80d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-5zsqr" Jul 21 01:44:04.932113 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:04.931972 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3cd4166-77e6-4b24-ae7d-b0d823eba80d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-5zsqr\" (UID: \"b3cd4166-77e6-4b24-ae7d-b0d823eba80d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-5zsqr" Jul 21 01:44:04.932113 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:04.932008 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3cd4166-77e6-4b24-ae7d-b0d823eba80d-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-5zsqr\" (UID: \"b3cd4166-77e6-4b24-ae7d-b0d823eba80d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-5zsqr" Jul 21 01:44:04.932113 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:04.932037 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3cd4166-77e6-4b24-ae7d-b0d823eba80d-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-5zsqr\" (UID: \"b3cd4166-77e6-4b24-ae7d-b0d823eba80d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-5zsqr" Jul 21 01:44:04.932113 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:04.932065 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3cd4166-77e6-4b24-ae7d-b0d823eba80d-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-5zsqr\" (UID: \"b3cd4166-77e6-4b24-ae7d-b0d823eba80d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-5zsqr" Jul 21 01:44:04.932113 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:04.932098 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3cd4166-77e6-4b24-ae7d-b0d823eba80d-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-5zsqr\" (UID: \"b3cd4166-77e6-4b24-ae7d-b0d823eba80d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-5zsqr" Jul 21 01:44:04.932431 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:04.932171 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3cd4166-77e6-4b24-ae7d-b0d823eba80d-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-5zsqr\" (UID: \"b3cd4166-77e6-4b24-ae7d-b0d823eba80d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-5zsqr" Jul 21 01:44:04.932431 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:04.932203 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8tmz\" (UniqueName: \"kubernetes.io/projected/b3cd4166-77e6-4b24-ae7d-b0d823eba80d-kube-api-access-f8tmz\") pod \"maas-default-gateway-openshift-default-5896d4dd49-5zsqr\" (UID: \"b3cd4166-77e6-4b24-ae7d-b0d823eba80d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-5zsqr" Jul 21 01:44:04.932431 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:04.932405 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3cd4166-77e6-4b24-ae7d-b0d823eba80d-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-5zsqr\" (UID: \"b3cd4166-77e6-4b24-ae7d-b0d823eba80d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-5zsqr" Jul 21 01:44:04.932645 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:04.932487 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3cd4166-77e6-4b24-ae7d-b0d823eba80d-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-5zsqr\" (UID: \"b3cd4166-77e6-4b24-ae7d-b0d823eba80d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-5zsqr" Jul 21 01:44:04.932756 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:04.932733 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3cd4166-77e6-4b24-ae7d-b0d823eba80d-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-5zsqr\" (UID: \"b3cd4166-77e6-4b24-ae7d-b0d823eba80d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-5zsqr" Jul 21 01:44:04.932818 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:04.932749 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3cd4166-77e6-4b24-ae7d-b0d823eba80d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-5zsqr\" (UID: \"b3cd4166-77e6-4b24-ae7d-b0d823eba80d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-5zsqr" Jul 21 01:44:04.932818 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:04.932755 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3cd4166-77e6-4b24-ae7d-b0d823eba80d-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-5zsqr\" (UID: \"b3cd4166-77e6-4b24-ae7d-b0d823eba80d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-5zsqr" Jul 21 01:44:04.934224 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:04.934203 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3cd4166-77e6-4b24-ae7d-b0d823eba80d-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-5zsqr\" (UID: \"b3cd4166-77e6-4b24-ae7d-b0d823eba80d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-5zsqr" Jul 21 01:44:04.934499 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:04.934482 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3cd4166-77e6-4b24-ae7d-b0d823eba80d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-5zsqr\" (UID: \"b3cd4166-77e6-4b24-ae7d-b0d823eba80d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-5zsqr" Jul 21 01:44:04.939216 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:04.939187 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3cd4166-77e6-4b24-ae7d-b0d823eba80d-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-5zsqr\" (UID: \"b3cd4166-77e6-4b24-ae7d-b0d823eba80d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-5zsqr" Jul 21 01:44:04.939381 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:04.939364 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8tmz\" (UniqueName: \"kubernetes.io/projected/b3cd4166-77e6-4b24-ae7d-b0d823eba80d-kube-api-access-f8tmz\") pod \"maas-default-gateway-openshift-default-5896d4dd49-5zsqr\" (UID: \"b3cd4166-77e6-4b24-ae7d-b0d823eba80d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-5zsqr" Jul 21 01:44:05.031385 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:05.031346 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-5zsqr" Jul 21 01:44:05.172058 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:05.172036 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-5zsqr"] Jul 21 01:44:05.174075 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:44:05.174044 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3cd4166_77e6_4b24_ae7d_b0d823eba80d.slice/crio-8dda87f35973548ecaa9bf3a6f4f1c7d4898b685463d6c1ebf6147b1c709cc8a WatchSource:0}: Error finding container 8dda87f35973548ecaa9bf3a6f4f1c7d4898b685463d6c1ebf6147b1c709cc8a: Status 404 returned error can't find the container with id 8dda87f35973548ecaa9bf3a6f4f1c7d4898b685463d6c1ebf6147b1c709cc8a Jul 21 01:44:05.416617 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:05.416526 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-5zsqr" event={"ID":"b3cd4166-77e6-4b24-ae7d-b0d823eba80d","Type":"ContainerStarted","Data":"8dda87f35973548ecaa9bf3a6f4f1c7d4898b685463d6c1ebf6147b1c709cc8a"} Jul 21 01:44:09.037387 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:09.037347 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 01:44:09.037761 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:09.037442 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 01:44:09.037761 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:09.037497 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 01:44:09.440827 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:09.440734 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-5zsqr" event={"ID":"b3cd4166-77e6-4b24-ae7d-b0d823eba80d","Type":"ContainerStarted","Data":"2e11cc31c26b5a1f52bec1315573d958c510446ba2ef5842e3b4bbaa2c2819db"} Jul 21 01:44:09.460645 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:09.460594 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-5zsqr" podStartSLOduration=1.5996240830000001 podStartE2EDuration="5.460577293s" totalImagesPullingTime="3.86095321s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 01:44:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:44:09.458915673 +0000 UTC m=+473.046081679" watchObservedRunningTime="2026-07-21 01:44:09.460577293 +0000 UTC m=+473.047743270" Jul 21 01:44:10.032477 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:10.032412 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-5zsqr" Jul 21 01:44:10.037142 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:10.037116 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-5zsqr" Jul 21 01:44:10.446660 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:10.446585 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-5zsqr" Jul 21 01:44:10.447504 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:10.447476 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-5zsqr" Jul 21 01:44:20.886591 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:20.886557 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-pmgsx"] Jul 21 01:44:20.891634 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:20.891614 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-pmgsx" Jul 21 01:44:20.893817 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:20.893799 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 21 01:44:20.898505 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:20.898485 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-pmgsx"] Jul 21 01:44:20.980120 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:20.980087 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg6pt\" (UniqueName: \"kubernetes.io/projected/8708d9af-3d4e-46cc-9e94-6295dc26aab8-kube-api-access-xg6pt\") pod \"limitador-limitador-64c94fd64f-pmgsx\" (UID: \"8708d9af-3d4e-46cc-9e94-6295dc26aab8\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-pmgsx" Jul 21 01:44:20.980519 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:20.980442 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8708d9af-3d4e-46cc-9e94-6295dc26aab8-config-file\") pod \"limitador-limitador-64c94fd64f-pmgsx\" (UID: \"8708d9af-3d4e-46cc-9e94-6295dc26aab8\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-pmgsx" Jul 21 01:44:20.991994 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:20.991958 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-pmgsx"] Jul 21 01:44:21.082058 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:21.082016 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg6pt\" (UniqueName: \"kubernetes.io/projected/8708d9af-3d4e-46cc-9e94-6295dc26aab8-kube-api-access-xg6pt\") pod \"limitador-limitador-64c94fd64f-pmgsx\" (UID: \"8708d9af-3d4e-46cc-9e94-6295dc26aab8\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-pmgsx" Jul 21 01:44:21.082248 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:21.082088 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8708d9af-3d4e-46cc-9e94-6295dc26aab8-config-file\") pod \"limitador-limitador-64c94fd64f-pmgsx\" (UID: \"8708d9af-3d4e-46cc-9e94-6295dc26aab8\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-pmgsx" Jul 21 01:44:21.082852 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:21.082830 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8708d9af-3d4e-46cc-9e94-6295dc26aab8-config-file\") pod \"limitador-limitador-64c94fd64f-pmgsx\" (UID: \"8708d9af-3d4e-46cc-9e94-6295dc26aab8\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-pmgsx" Jul 21 01:44:21.090616 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:21.090587 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg6pt\" (UniqueName: \"kubernetes.io/projected/8708d9af-3d4e-46cc-9e94-6295dc26aab8-kube-api-access-xg6pt\") pod \"limitador-limitador-64c94fd64f-pmgsx\" (UID: \"8708d9af-3d4e-46cc-9e94-6295dc26aab8\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-pmgsx" Jul 21 01:44:21.203728 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:21.203636 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-pmgsx" Jul 21 01:44:21.281517 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:21.281479 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 01:44:21.285373 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:21.285351 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-4glr6" Jul 21 01:44:21.290995 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:21.290751 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 01:44:21.318842 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:21.318746 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 01:44:21.345368 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:21.345338 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-pmgsx"] Jul 21 01:44:21.348485 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:44:21.348424 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8708d9af_3d4e_46cc_9e94_6295dc26aab8.slice/crio-1b9e98319105148a7879b681ee2217d93d3a50db24e62adebd5a6c40a200f13c WatchSource:0}: Error finding container 1b9e98319105148a7879b681ee2217d93d3a50db24e62adebd5a6c40a200f13c: Status 404 returned error can't find the container with id 1b9e98319105148a7879b681ee2217d93d3a50db24e62adebd5a6c40a200f13c Jul 21 01:44:21.385062 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:21.385032 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/81f38466-10f7-408f-ad65-44bdbb3b9e91-config-file\") pod \"limitador-limitador-54d6d988d5-4glr6\" (UID: \"81f38466-10f7-408f-ad65-44bdbb3b9e91\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-4glr6" Jul 21 01:44:21.385217 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:21.385070 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bd8rx\" (UniqueName: \"kubernetes.io/projected/81f38466-10f7-408f-ad65-44bdbb3b9e91-kube-api-access-bd8rx\") pod \"limitador-limitador-54d6d988d5-4glr6\" (UID: \"81f38466-10f7-408f-ad65-44bdbb3b9e91\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-4glr6" Jul 21 01:44:21.486491 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:21.486394 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/81f38466-10f7-408f-ad65-44bdbb3b9e91-config-file\") pod \"limitador-limitador-54d6d988d5-4glr6\" (UID: \"81f38466-10f7-408f-ad65-44bdbb3b9e91\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-4glr6" Jul 21 01:44:21.486491 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:21.486426 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd8rx\" (UniqueName: \"kubernetes.io/projected/81f38466-10f7-408f-ad65-44bdbb3b9e91-kube-api-access-bd8rx\") pod \"limitador-limitador-54d6d988d5-4glr6\" (UID: \"81f38466-10f7-408f-ad65-44bdbb3b9e91\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-4glr6" Jul 21 01:44:21.487034 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:21.487013 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/81f38466-10f7-408f-ad65-44bdbb3b9e91-config-file\") pod \"limitador-limitador-54d6d988d5-4glr6\" (UID: \"81f38466-10f7-408f-ad65-44bdbb3b9e91\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-4glr6" Jul 21 01:44:21.494163 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:21.494141 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bd8rx\" (UniqueName: \"kubernetes.io/projected/81f38466-10f7-408f-ad65-44bdbb3b9e91-kube-api-access-bd8rx\") pod \"limitador-limitador-54d6d988d5-4glr6\" (UID: \"81f38466-10f7-408f-ad65-44bdbb3b9e91\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-4glr6" Jul 21 01:44:21.507847 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:21.507819 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-pmgsx" event={"ID":"8708d9af-3d4e-46cc-9e94-6295dc26aab8","Type":"ContainerStarted","Data":"1b9e98319105148a7879b681ee2217d93d3a50db24e62adebd5a6c40a200f13c"} Jul 21 01:44:21.600054 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:21.600027 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-4glr6" Jul 21 01:44:21.741518 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:21.741496 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 01:44:21.744164 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:44:21.744135 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81f38466_10f7_408f_ad65_44bdbb3b9e91.slice/crio-1ab160293bd55346ddb7e702660b6f7d687fc185992d243e8ac3e8277ea5df75 WatchSource:0}: Error finding container 1ab160293bd55346ddb7e702660b6f7d687fc185992d243e8ac3e8277ea5df75: Status 404 returned error can't find the container with id 1ab160293bd55346ddb7e702660b6f7d687fc185992d243e8ac3e8277ea5df75 Jul 21 01:44:22.529566 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:22.529488 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-4glr6" event={"ID":"81f38466-10f7-408f-ad65-44bdbb3b9e91","Type":"ContainerStarted","Data":"1ab160293bd55346ddb7e702660b6f7d687fc185992d243e8ac3e8277ea5df75"} Jul 21 01:44:25.548810 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:25.548771 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-pmgsx" event={"ID":"8708d9af-3d4e-46cc-9e94-6295dc26aab8","Type":"ContainerStarted","Data":"e2955b0d3f1189a970d940ad206cdb53a3252b763816c2603bd8c209949699a1"} Jul 21 01:44:25.549289 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:25.548837 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-pmgsx" Jul 21 01:44:25.550438 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:25.550414 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-4glr6" event={"ID":"81f38466-10f7-408f-ad65-44bdbb3b9e91","Type":"ContainerStarted","Data":"c08261e30a03484ddb679feef62b3a38f6e9f6c4ec33a7c5c563fc49a3095e2c"} Jul 21 01:44:25.550564 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:25.550482 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-4glr6" Jul 21 01:44:25.564356 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:25.564306 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-pmgsx" podStartSLOduration=2.25444252 podStartE2EDuration="5.564291162s" totalImagesPullingTime="3.309848642s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 01:44:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:44:25.563441481 +0000 UTC m=+489.150607459" watchObservedRunningTime="2026-07-21 01:44:25.564291162 +0000 UTC m=+489.151457141" Jul 21 01:44:25.585027 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:25.584974 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-4glr6" podStartSLOduration=1.664111742 podStartE2EDuration="4.584957564s" totalImagesPullingTime="2.920845822s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 01:44:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:44:25.581708529 +0000 UTC m=+489.168874520" watchObservedRunningTime="2026-07-21 01:44:25.584957564 +0000 UTC m=+489.172123543" Jul 21 01:44:36.556053 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:36.556023 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-pmgsx" Jul 21 01:44:36.556538 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:36.556179 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-4glr6" Jul 21 01:44:36.611162 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:36.611128 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-pmgsx"] Jul 21 01:44:36.611426 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:36.611290 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-pmgsx" podUID="8708d9af-3d4e-46cc-9e94-6295dc26aab8" containerName="limitador" containerID="cri-o://e2955b0d3f1189a970d940ad206cdb53a3252b763816c2603bd8c209949699a1" gracePeriod=30 Jul 21 01:44:37.169847 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:37.169784 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-pmgsx" Jul 21 01:44:37.237368 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:37.237340 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xg6pt\" (UniqueName: \"kubernetes.io/projected/8708d9af-3d4e-46cc-9e94-6295dc26aab8-kube-api-access-xg6pt\") pod \"8708d9af-3d4e-46cc-9e94-6295dc26aab8\" (UID: \"8708d9af-3d4e-46cc-9e94-6295dc26aab8\") " Jul 21 01:44:37.239509 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:37.239466 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8708d9af-3d4e-46cc-9e94-6295dc26aab8-kube-api-access-xg6pt" (OuterVolumeSpecName: "kube-api-access-xg6pt") pod "8708d9af-3d4e-46cc-9e94-6295dc26aab8" (UID: "8708d9af-3d4e-46cc-9e94-6295dc26aab8"). InnerVolumeSpecName "kube-api-access-xg6pt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 01:44:37.337836 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:37.337801 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8708d9af-3d4e-46cc-9e94-6295dc26aab8-config-file\") pod \"8708d9af-3d4e-46cc-9e94-6295dc26aab8\" (UID: \"8708d9af-3d4e-46cc-9e94-6295dc26aab8\") " Jul 21 01:44:37.338008 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:37.337974 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xg6pt\" (UniqueName: \"kubernetes.io/projected/8708d9af-3d4e-46cc-9e94-6295dc26aab8-kube-api-access-xg6pt\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 21 01:44:37.338150 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:37.338127 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8708d9af-3d4e-46cc-9e94-6295dc26aab8-config-file" (OuterVolumeSpecName: "config-file") pod "8708d9af-3d4e-46cc-9e94-6295dc26aab8" (UID: "8708d9af-3d4e-46cc-9e94-6295dc26aab8"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 01:44:37.438739 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:37.438649 2580 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8708d9af-3d4e-46cc-9e94-6295dc26aab8-config-file\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 21 01:44:37.613544 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:37.613505 2580 generic.go:356] "Generic (PLEG): container finished" podID="8708d9af-3d4e-46cc-9e94-6295dc26aab8" containerID="e2955b0d3f1189a970d940ad206cdb53a3252b763816c2603bd8c209949699a1" exitCode=0 Jul 21 01:44:37.613947 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:37.613558 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-pmgsx" Jul 21 01:44:37.613947 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:37.613596 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-pmgsx" event={"ID":"8708d9af-3d4e-46cc-9e94-6295dc26aab8","Type":"ContainerDied","Data":"e2955b0d3f1189a970d940ad206cdb53a3252b763816c2603bd8c209949699a1"} Jul 21 01:44:37.613947 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:37.613677 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-pmgsx" event={"ID":"8708d9af-3d4e-46cc-9e94-6295dc26aab8","Type":"ContainerDied","Data":"1b9e98319105148a7879b681ee2217d93d3a50db24e62adebd5a6c40a200f13c"} Jul 21 01:44:37.613947 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:37.613700 2580 scope.go:117] "RemoveContainer" containerID="e2955b0d3f1189a970d940ad206cdb53a3252b763816c2603bd8c209949699a1" Jul 21 01:44:37.623151 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:37.623131 2580 scope.go:117] "RemoveContainer" containerID="e2955b0d3f1189a970d940ad206cdb53a3252b763816c2603bd8c209949699a1" Jul 21 01:44:37.623430 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:44:37.623413 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2955b0d3f1189a970d940ad206cdb53a3252b763816c2603bd8c209949699a1\": container with ID starting with e2955b0d3f1189a970d940ad206cdb53a3252b763816c2603bd8c209949699a1 not found: ID does not exist" containerID="e2955b0d3f1189a970d940ad206cdb53a3252b763816c2603bd8c209949699a1" Jul 21 01:44:37.623518 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:37.623439 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2955b0d3f1189a970d940ad206cdb53a3252b763816c2603bd8c209949699a1"} err="failed to get container status \"e2955b0d3f1189a970d940ad206cdb53a3252b763816c2603bd8c209949699a1\": rpc error: code = NotFound desc = could not find container \"e2955b0d3f1189a970d940ad206cdb53a3252b763816c2603bd8c209949699a1\": container with ID starting with e2955b0d3f1189a970d940ad206cdb53a3252b763816c2603bd8c209949699a1 not found: ID does not exist" Jul 21 01:44:37.634463 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:37.634426 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-pmgsx"] Jul 21 01:44:37.638674 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:37.638645 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-pmgsx"] Jul 21 01:44:38.877396 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:44:38.877363 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8708d9af-3d4e-46cc-9e94-6295dc26aab8" path="/var/lib/kubelet/pods/8708d9af-3d4e-46cc-9e94-6295dc26aab8/volumes" Jul 21 01:45:02.213732 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:02.213693 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-78679d4b88-wrb6f"] Jul 21 01:45:02.214166 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:02.214146 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8708d9af-3d4e-46cc-9e94-6295dc26aab8" containerName="limitador" Jul 21 01:45:02.214166 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:02.214158 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8708d9af-3d4e-46cc-9e94-6295dc26aab8" containerName="limitador" Jul 21 01:45:02.214242 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:02.214231 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8708d9af-3d4e-46cc-9e94-6295dc26aab8" containerName="limitador" Jul 21 01:45:02.224583 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:02.224548 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78679d4b88-wrb6f" Jul 21 01:45:02.228071 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:02.227769 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 21 01:45:02.228071 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:02.227947 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 21 01:45:02.228071 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:02.227947 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-2mqz7\"" Jul 21 01:45:02.228353 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:02.228308 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-78679d4b88-wrb6f"] Jul 21 01:45:02.364330 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:02.364292 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e528c65d-2fc9-442a-8c63-07c4beafcd9e-webhook-cert\") pod \"maas-controller-78679d4b88-wrb6f\" (UID: \"e528c65d-2fc9-442a-8c63-07c4beafcd9e\") " pod="opendatahub/maas-controller-78679d4b88-wrb6f" Jul 21 01:45:02.364330 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:02.364342 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9699\" (UniqueName: \"kubernetes.io/projected/e528c65d-2fc9-442a-8c63-07c4beafcd9e-kube-api-access-w9699\") pod \"maas-controller-78679d4b88-wrb6f\" (UID: \"e528c65d-2fc9-442a-8c63-07c4beafcd9e\") " pod="opendatahub/maas-controller-78679d4b88-wrb6f" Jul 21 01:45:02.396723 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:02.396685 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5cd8fcb89-xrcw8"] Jul 21 01:45:02.400126 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:02.400099 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5cd8fcb89-xrcw8" Jul 21 01:45:02.415162 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:02.415124 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5cd8fcb89-xrcw8"] Jul 21 01:45:02.465843 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:02.465739 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e485a5c0-b377-44b7-a923-c07a8aa5f928-webhook-cert\") pod \"maas-controller-5cd8fcb89-xrcw8\" (UID: \"e485a5c0-b377-44b7-a923-c07a8aa5f928\") " pod="opendatahub/maas-controller-5cd8fcb89-xrcw8" Jul 21 01:45:02.466027 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:02.465886 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e528c65d-2fc9-442a-8c63-07c4beafcd9e-webhook-cert\") pod \"maas-controller-78679d4b88-wrb6f\" (UID: \"e528c65d-2fc9-442a-8c63-07c4beafcd9e\") " pod="opendatahub/maas-controller-78679d4b88-wrb6f" Jul 21 01:45:02.466027 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:02.465922 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqhnf\" (UniqueName: \"kubernetes.io/projected/e485a5c0-b377-44b7-a923-c07a8aa5f928-kube-api-access-kqhnf\") pod \"maas-controller-5cd8fcb89-xrcw8\" (UID: \"e485a5c0-b377-44b7-a923-c07a8aa5f928\") " pod="opendatahub/maas-controller-5cd8fcb89-xrcw8" Jul 21 01:45:02.466027 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:02.465969 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9699\" (UniqueName: \"kubernetes.io/projected/e528c65d-2fc9-442a-8c63-07c4beafcd9e-kube-api-access-w9699\") pod \"maas-controller-78679d4b88-wrb6f\" (UID: \"e528c65d-2fc9-442a-8c63-07c4beafcd9e\") " pod="opendatahub/maas-controller-78679d4b88-wrb6f" Jul 21 01:45:02.468385 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:02.468361 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e528c65d-2fc9-442a-8c63-07c4beafcd9e-webhook-cert\") pod \"maas-controller-78679d4b88-wrb6f\" (UID: \"e528c65d-2fc9-442a-8c63-07c4beafcd9e\") " pod="opendatahub/maas-controller-78679d4b88-wrb6f" Jul 21 01:45:02.473868 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:02.473844 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9699\" (UniqueName: \"kubernetes.io/projected/e528c65d-2fc9-442a-8c63-07c4beafcd9e-kube-api-access-w9699\") pod \"maas-controller-78679d4b88-wrb6f\" (UID: \"e528c65d-2fc9-442a-8c63-07c4beafcd9e\") " pod="opendatahub/maas-controller-78679d4b88-wrb6f" Jul 21 01:45:02.538165 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:02.538126 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78679d4b88-wrb6f" Jul 21 01:45:02.567261 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:02.567227 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqhnf\" (UniqueName: \"kubernetes.io/projected/e485a5c0-b377-44b7-a923-c07a8aa5f928-kube-api-access-kqhnf\") pod \"maas-controller-5cd8fcb89-xrcw8\" (UID: \"e485a5c0-b377-44b7-a923-c07a8aa5f928\") " pod="opendatahub/maas-controller-5cd8fcb89-xrcw8" Jul 21 01:45:02.567432 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:02.567276 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e485a5c0-b377-44b7-a923-c07a8aa5f928-webhook-cert\") pod \"maas-controller-5cd8fcb89-xrcw8\" (UID: \"e485a5c0-b377-44b7-a923-c07a8aa5f928\") " pod="opendatahub/maas-controller-5cd8fcb89-xrcw8" Jul 21 01:45:02.569792 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:02.569766 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e485a5c0-b377-44b7-a923-c07a8aa5f928-webhook-cert\") pod \"maas-controller-5cd8fcb89-xrcw8\" (UID: \"e485a5c0-b377-44b7-a923-c07a8aa5f928\") " pod="opendatahub/maas-controller-5cd8fcb89-xrcw8" Jul 21 01:45:02.578134 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:02.578109 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqhnf\" (UniqueName: \"kubernetes.io/projected/e485a5c0-b377-44b7-a923-c07a8aa5f928-kube-api-access-kqhnf\") pod \"maas-controller-5cd8fcb89-xrcw8\" (UID: \"e485a5c0-b377-44b7-a923-c07a8aa5f928\") " pod="opendatahub/maas-controller-5cd8fcb89-xrcw8" Jul 21 01:45:02.679103 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:02.679069 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-78679d4b88-wrb6f"] Jul 21 01:45:02.681377 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:45:02.681350 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode528c65d_2fc9_442a_8c63_07c4beafcd9e.slice/crio-175590e7225ae43cf3a602bf5640b2f9358ef28807aa0e921fc779ae0a9c46c3 WatchSource:0}: Error finding container 175590e7225ae43cf3a602bf5640b2f9358ef28807aa0e921fc779ae0a9c46c3: Status 404 returned error can't find the container with id 175590e7225ae43cf3a602bf5640b2f9358ef28807aa0e921fc779ae0a9c46c3 Jul 21 01:45:02.710746 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:02.710717 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5cd8fcb89-xrcw8" Jul 21 01:45:02.751773 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:02.751734 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78679d4b88-wrb6f" event={"ID":"e528c65d-2fc9-442a-8c63-07c4beafcd9e","Type":"ContainerStarted","Data":"175590e7225ae43cf3a602bf5640b2f9358ef28807aa0e921fc779ae0a9c46c3"} Jul 21 01:45:02.859200 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:02.859166 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5cd8fcb89-xrcw8"] Jul 21 01:45:02.860943 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:45:02.860906 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode485a5c0_b377_44b7_a923_c07a8aa5f928.slice/crio-9423245c65f6478a30389fb77adb717bc47bf3f8ec9f58c6acc2a762f9a13033 WatchSource:0}: Error finding container 9423245c65f6478a30389fb77adb717bc47bf3f8ec9f58c6acc2a762f9a13033: Status 404 returned error can't find the container with id 9423245c65f6478a30389fb77adb717bc47bf3f8ec9f58c6acc2a762f9a13033 Jul 21 01:45:03.762804 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:03.762768 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5cd8fcb89-xrcw8" event={"ID":"e485a5c0-b377-44b7-a923-c07a8aa5f928","Type":"ContainerStarted","Data":"9423245c65f6478a30389fb77adb717bc47bf3f8ec9f58c6acc2a762f9a13033"} Jul 21 01:45:05.777028 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:05.776951 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78679d4b88-wrb6f" event={"ID":"e528c65d-2fc9-442a-8c63-07c4beafcd9e","Type":"ContainerStarted","Data":"164b080f502b12561992205296db73fe585aff3cc85fb3d7fa462cff74700bae"} Jul 21 01:45:05.777553 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:05.777124 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-78679d4b88-wrb6f" Jul 21 01:45:05.778667 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:05.778642 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5cd8fcb89-xrcw8" event={"ID":"e485a5c0-b377-44b7-a923-c07a8aa5f928","Type":"ContainerStarted","Data":"9eeb7ce58b66e263fd439f44156479561fc595c0c4c6a2a6e0fd7a7fe441c64d"} Jul 21 01:45:05.778783 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:05.778691 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5cd8fcb89-xrcw8" Jul 21 01:45:05.793015 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:05.792955 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-78679d4b88-wrb6f" podStartSLOduration=1.006952053 podStartE2EDuration="3.792933734s" totalImagesPullingTime="2.785981681s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 01:45:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:45:05.792305795 +0000 UTC m=+529.379471791" watchObservedRunningTime="2026-07-21 01:45:05.792933734 +0000 UTC m=+529.380099715" Jul 21 01:45:05.813472 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:05.811062 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5cd8fcb89-xrcw8" podStartSLOduration=1.211151389 podStartE2EDuration="3.811034391s" totalImagesPullingTime="2.599883002s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 01:45:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:45:05.808321912 +0000 UTC m=+529.395487890" watchObservedRunningTime="2026-07-21 01:45:05.811034391 +0000 UTC m=+529.398200370" Jul 21 01:45:16.786519 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:16.786424 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5cd8fcb89-xrcw8" Jul 21 01:45:16.786895 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:16.786622 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-78679d4b88-wrb6f" Jul 21 01:45:16.857929 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:16.857893 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-78679d4b88-wrb6f"] Jul 21 01:45:16.860821 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:16.860773 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-78679d4b88-wrb6f" podUID="e528c65d-2fc9-442a-8c63-07c4beafcd9e" containerName="manager" containerID="cri-o://164b080f502b12561992205296db73fe585aff3cc85fb3d7fa462cff74700bae" gracePeriod=30 Jul 21 01:45:17.118676 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:17.118645 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78679d4b88-wrb6f" Jul 21 01:45:17.213757 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:17.213717 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e528c65d-2fc9-442a-8c63-07c4beafcd9e-webhook-cert\") pod \"e528c65d-2fc9-442a-8c63-07c4beafcd9e\" (UID: \"e528c65d-2fc9-442a-8c63-07c4beafcd9e\") " Jul 21 01:45:17.213757 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:17.213760 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w9699\" (UniqueName: \"kubernetes.io/projected/e528c65d-2fc9-442a-8c63-07c4beafcd9e-kube-api-access-w9699\") pod \"e528c65d-2fc9-442a-8c63-07c4beafcd9e\" (UID: \"e528c65d-2fc9-442a-8c63-07c4beafcd9e\") " Jul 21 01:45:17.216014 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:17.215987 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e528c65d-2fc9-442a-8c63-07c4beafcd9e-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "e528c65d-2fc9-442a-8c63-07c4beafcd9e" (UID: "e528c65d-2fc9-442a-8c63-07c4beafcd9e"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 01:45:17.216133 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:17.216073 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e528c65d-2fc9-442a-8c63-07c4beafcd9e-kube-api-access-w9699" (OuterVolumeSpecName: "kube-api-access-w9699") pod "e528c65d-2fc9-442a-8c63-07c4beafcd9e" (UID: "e528c65d-2fc9-442a-8c63-07c4beafcd9e"). InnerVolumeSpecName "kube-api-access-w9699". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 01:45:17.314984 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:17.314952 2580 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e528c65d-2fc9-442a-8c63-07c4beafcd9e-webhook-cert\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 21 01:45:17.314984 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:17.314982 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9699\" (UniqueName: \"kubernetes.io/projected/e528c65d-2fc9-442a-8c63-07c4beafcd9e-kube-api-access-w9699\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 21 01:45:17.849776 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:17.849741 2580 generic.go:356] "Generic (PLEG): container finished" podID="e528c65d-2fc9-442a-8c63-07c4beafcd9e" containerID="164b080f502b12561992205296db73fe585aff3cc85fb3d7fa462cff74700bae" exitCode=0 Jul 21 01:45:17.850187 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:17.849784 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78679d4b88-wrb6f" Jul 21 01:45:17.850187 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:17.849798 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78679d4b88-wrb6f" event={"ID":"e528c65d-2fc9-442a-8c63-07c4beafcd9e","Type":"ContainerDied","Data":"164b080f502b12561992205296db73fe585aff3cc85fb3d7fa462cff74700bae"} Jul 21 01:45:17.850187 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:17.849856 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78679d4b88-wrb6f" event={"ID":"e528c65d-2fc9-442a-8c63-07c4beafcd9e","Type":"ContainerDied","Data":"175590e7225ae43cf3a602bf5640b2f9358ef28807aa0e921fc779ae0a9c46c3"} Jul 21 01:45:17.850187 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:17.849886 2580 scope.go:117] "RemoveContainer" containerID="164b080f502b12561992205296db73fe585aff3cc85fb3d7fa462cff74700bae" Jul 21 01:45:17.859404 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:17.859386 2580 scope.go:117] "RemoveContainer" containerID="164b080f502b12561992205296db73fe585aff3cc85fb3d7fa462cff74700bae" Jul 21 01:45:17.859726 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:45:17.859706 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"164b080f502b12561992205296db73fe585aff3cc85fb3d7fa462cff74700bae\": container with ID starting with 164b080f502b12561992205296db73fe585aff3cc85fb3d7fa462cff74700bae not found: ID does not exist" containerID="164b080f502b12561992205296db73fe585aff3cc85fb3d7fa462cff74700bae" Jul 21 01:45:17.859783 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:17.859736 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"164b080f502b12561992205296db73fe585aff3cc85fb3d7fa462cff74700bae"} err="failed to get container status \"164b080f502b12561992205296db73fe585aff3cc85fb3d7fa462cff74700bae\": rpc error: code = NotFound desc = could not find container \"164b080f502b12561992205296db73fe585aff3cc85fb3d7fa462cff74700bae\": container with ID starting with 164b080f502b12561992205296db73fe585aff3cc85fb3d7fa462cff74700bae not found: ID does not exist" Jul 21 01:45:17.869606 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:17.869581 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-78679d4b88-wrb6f"] Jul 21 01:45:17.873030 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:17.873006 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-78679d4b88-wrb6f"] Jul 21 01:45:18.877067 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:18.877035 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e528c65d-2fc9-442a-8c63-07c4beafcd9e" path="/var/lib/kubelet/pods/e528c65d-2fc9-442a-8c63-07c4beafcd9e/volumes" Jul 21 01:45:38.469883 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:38.469853 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-frgrf"] Jul 21 01:45:38.470314 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:38.470262 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e528c65d-2fc9-442a-8c63-07c4beafcd9e" containerName="manager" Jul 21 01:45:38.470314 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:38.470273 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e528c65d-2fc9-442a-8c63-07c4beafcd9e" containerName="manager" Jul 21 01:45:38.470393 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:38.470345 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e528c65d-2fc9-442a-8c63-07c4beafcd9e" containerName="manager" Jul 21 01:45:38.478490 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:38.478467 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-frgrf" Jul 21 01:45:38.481085 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:38.481061 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-2c6d9\"" Jul 21 01:45:38.481975 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:38.481954 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 21 01:45:38.483213 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:38.483192 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-frgrf"] Jul 21 01:45:38.503523 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:38.503490 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-bxgxv"] Jul 21 01:45:38.507536 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:38.507517 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-bxgxv" Jul 21 01:45:38.523484 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:38.523427 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-bxgxv"] Jul 21 01:45:38.610931 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:38.610896 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9775328a-6f07-4ded-b3a0-1484ed0969b6-plugins-config-volume\") pod \"payload-processing-5bf47cd988-bxgxv\" (UID: \"9775328a-6f07-4ded-b3a0-1484ed0969b6\") " pod="openshift-ingress/payload-processing-5bf47cd988-bxgxv" Jul 21 01:45:38.611098 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:38.610945 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d04fe9b-af81-49cd-91c3-ed777f12282a-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-frgrf\" (UID: \"7d04fe9b-af81-49cd-91c3-ed777f12282a\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-frgrf" Jul 21 01:45:38.611098 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:38.611061 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2l8kg\" (UniqueName: \"kubernetes.io/projected/7d04fe9b-af81-49cd-91c3-ed777f12282a-kube-api-access-2l8kg\") pod \"payload-pre-processing-7677b774bb-frgrf\" (UID: \"7d04fe9b-af81-49cd-91c3-ed777f12282a\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-frgrf" Jul 21 01:45:38.611187 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:38.611123 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdll8\" (UniqueName: \"kubernetes.io/projected/9775328a-6f07-4ded-b3a0-1484ed0969b6-kube-api-access-mdll8\") pod \"payload-processing-5bf47cd988-bxgxv\" (UID: \"9775328a-6f07-4ded-b3a0-1484ed0969b6\") " pod="openshift-ingress/payload-processing-5bf47cd988-bxgxv" Jul 21 01:45:38.712550 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:38.712517 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9775328a-6f07-4ded-b3a0-1484ed0969b6-plugins-config-volume\") pod \"payload-processing-5bf47cd988-bxgxv\" (UID: \"9775328a-6f07-4ded-b3a0-1484ed0969b6\") " pod="openshift-ingress/payload-processing-5bf47cd988-bxgxv" Jul 21 01:45:38.712766 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:38.712562 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d04fe9b-af81-49cd-91c3-ed777f12282a-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-frgrf\" (UID: \"7d04fe9b-af81-49cd-91c3-ed777f12282a\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-frgrf" Jul 21 01:45:38.712766 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:38.712631 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l8kg\" (UniqueName: \"kubernetes.io/projected/7d04fe9b-af81-49cd-91c3-ed777f12282a-kube-api-access-2l8kg\") pod \"payload-pre-processing-7677b774bb-frgrf\" (UID: \"7d04fe9b-af81-49cd-91c3-ed777f12282a\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-frgrf" Jul 21 01:45:38.712766 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:38.712669 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdll8\" (UniqueName: \"kubernetes.io/projected/9775328a-6f07-4ded-b3a0-1484ed0969b6-kube-api-access-mdll8\") pod \"payload-processing-5bf47cd988-bxgxv\" (UID: \"9775328a-6f07-4ded-b3a0-1484ed0969b6\") " pod="openshift-ingress/payload-processing-5bf47cd988-bxgxv" Jul 21 01:45:38.713315 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:38.713290 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9775328a-6f07-4ded-b3a0-1484ed0969b6-plugins-config-volume\") pod \"payload-processing-5bf47cd988-bxgxv\" (UID: \"9775328a-6f07-4ded-b3a0-1484ed0969b6\") " pod="openshift-ingress/payload-processing-5bf47cd988-bxgxv" Jul 21 01:45:38.713398 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:38.713292 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d04fe9b-af81-49cd-91c3-ed777f12282a-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-frgrf\" (UID: \"7d04fe9b-af81-49cd-91c3-ed777f12282a\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-frgrf" Jul 21 01:45:38.721431 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:38.721360 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdll8\" (UniqueName: \"kubernetes.io/projected/9775328a-6f07-4ded-b3a0-1484ed0969b6-kube-api-access-mdll8\") pod \"payload-processing-5bf47cd988-bxgxv\" (UID: \"9775328a-6f07-4ded-b3a0-1484ed0969b6\") " pod="openshift-ingress/payload-processing-5bf47cd988-bxgxv" Jul 21 01:45:38.721587 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:38.721463 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l8kg\" (UniqueName: \"kubernetes.io/projected/7d04fe9b-af81-49cd-91c3-ed777f12282a-kube-api-access-2l8kg\") pod \"payload-pre-processing-7677b774bb-frgrf\" (UID: \"7d04fe9b-af81-49cd-91c3-ed777f12282a\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-frgrf" Jul 21 01:45:38.791600 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:38.791566 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-frgrf" Jul 21 01:45:38.818407 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:38.818377 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-bxgxv" Jul 21 01:45:38.827382 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:38.827352 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 01:45:39.018148 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:39.017746 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-bxgxv"] Jul 21 01:45:39.179511 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:39.179481 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-frgrf"] Jul 21 01:45:39.181703 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:45:39.181674 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d04fe9b_af81_49cd_91c3_ed777f12282a.slice/crio-cc2c3212b08548c09d88c3a253aa78a737df016b7573e629117872c95881abbe WatchSource:0}: Error finding container cc2c3212b08548c09d88c3a253aa78a737df016b7573e629117872c95881abbe: Status 404 returned error can't find the container with id cc2c3212b08548c09d88c3a253aa78a737df016b7573e629117872c95881abbe Jul 21 01:45:39.979430 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:39.979339 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-bxgxv" event={"ID":"9775328a-6f07-4ded-b3a0-1484ed0969b6","Type":"ContainerStarted","Data":"2c5639edaa25aead04399ab9b9453723a9c1a6d8fbcc20d2a9b25689f5631f17"} Jul 21 01:45:39.981985 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:39.981868 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-frgrf" event={"ID":"7d04fe9b-af81-49cd-91c3-ed777f12282a","Type":"ContainerStarted","Data":"cc2c3212b08548c09d88c3a253aa78a737df016b7573e629117872c95881abbe"} Jul 21 01:45:41.996117 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:41.996082 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-bxgxv" event={"ID":"9775328a-6f07-4ded-b3a0-1484ed0969b6","Type":"ContainerStarted","Data":"17124745b483a38c602ddb62c516062a2bc3cb246d322c0cb389dd4a71e876bf"} Jul 21 01:45:41.996576 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:41.996187 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5bf47cd988-bxgxv" Jul 21 01:45:41.997957 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:41.997931 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-frgrf" event={"ID":"7d04fe9b-af81-49cd-91c3-ed777f12282a","Type":"ContainerStarted","Data":"6e95b3efb7f0710ebe3ca35c833efd52fea994e93470307dff4a087fa964ff29"} Jul 21 01:45:41.998071 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:41.998024 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-frgrf" Jul 21 01:45:42.011629 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:42.011577 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5bf47cd988-bxgxv" podStartSLOduration=1.6948087520000001 podStartE2EDuration="4.011560324s" totalImagesPullingTime="2.316751572s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 01:45:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:45:42.00966685 +0000 UTC m=+565.596832844" watchObservedRunningTime="2026-07-21 01:45:42.011560324 +0000 UTC m=+565.598726301" Jul 21 01:45:42.025057 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:42.025010 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7677b774bb-frgrf" podStartSLOduration=1.844768441 podStartE2EDuration="4.024991983s" totalImagesPullingTime="2.180223542s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 01:45:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:45:42.023295659 +0000 UTC m=+565.610461657" watchObservedRunningTime="2026-07-21 01:45:42.024991983 +0000 UTC m=+565.612157961" Jul 21 01:45:47.027221 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:47.027187 2580 generic.go:356] "Generic (PLEG): container finished" podID="b3cd4166-77e6-4b24-ae7d-b0d823eba80d" containerID="2e11cc31c26b5a1f52bec1315573d958c510446ba2ef5842e3b4bbaa2c2819db" exitCode=0 Jul 21 01:45:47.027767 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:47.027248 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-5zsqr" event={"ID":"b3cd4166-77e6-4b24-ae7d-b0d823eba80d","Type":"ContainerDied","Data":"2e11cc31c26b5a1f52bec1315573d958c510446ba2ef5842e3b4bbaa2c2819db"} Jul 21 01:45:47.027767 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:47.027649 2580 scope.go:117] "RemoveContainer" containerID="2e11cc31c26b5a1f52bec1315573d958c510446ba2ef5842e3b4bbaa2c2819db" Jul 21 01:45:47.028631 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:47.028603 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 01:45:47.028692 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:47.028669 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 01:45:47.028728 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:47.028703 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 01:45:48.034626 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:48.034591 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-5zsqr" event={"ID":"b3cd4166-77e6-4b24-ae7d-b0d823eba80d","Type":"ContainerStarted","Data":"915a486674a4161ceac49dd2c0e4c3f79f706516427b71e4a832a71d430f4e3c"} Jul 21 01:45:49.031827 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:49.031787 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-5zsqr" Jul 21 01:45:49.032047 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:49.032023 2580 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-5zsqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Jul 21 01:45:49.032114 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:49.032094 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-5zsqr" podUID="b3cd4166-77e6-4b24-ae7d-b0d823eba80d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Jul 21 01:45:50.031836 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:50.031801 2580 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-5zsqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Jul 21 01:45:50.032211 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:50.031852 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-5zsqr" podUID="b3cd4166-77e6-4b24-ae7d-b0d823eba80d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Jul 21 01:45:51.032333 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:51.032298 2580 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-5zsqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Jul 21 01:45:51.032765 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:51.032371 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-5zsqr" podUID="b3cd4166-77e6-4b24-ae7d-b0d823eba80d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Jul 21 01:45:52.035744 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:52.035715 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-5zsqr" Jul 21 01:45:52.036205 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:52.036154 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-5zsqr" Jul 21 01:45:52.036772 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:52.036752 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-5zsqr" Jul 21 01:45:53.005407 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:53.005375 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5bf47cd988-bxgxv" Jul 21 01:45:53.005744 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:53.005727 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-frgrf" Jul 21 01:45:55.681264 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:55.681234 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z7r2c"] Jul 21 01:45:55.684303 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:55.684287 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z7r2c" Jul 21 01:45:55.687598 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:55.687568 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 21 01:45:55.687598 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:55.687591 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 21 01:45:55.687789 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:55.687590 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-fqk5w\"" Jul 21 01:45:55.687789 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:55.687597 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 21 01:45:55.694611 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:55.694592 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z7r2c"] Jul 21 01:45:55.701146 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:55.701118 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/098b3243-a8a5-4ee6-8e82-2e7b27ac6671-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z7r2c\" (UID: \"098b3243-a8a5-4ee6-8e82-2e7b27ac6671\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z7r2c" Jul 21 01:45:55.701262 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:55.701182 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2d4k\" (UniqueName: \"kubernetes.io/projected/098b3243-a8a5-4ee6-8e82-2e7b27ac6671-kube-api-access-c2d4k\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z7r2c\" (UID: \"098b3243-a8a5-4ee6-8e82-2e7b27ac6671\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z7r2c" Jul 21 01:45:55.701262 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:55.701203 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/098b3243-a8a5-4ee6-8e82-2e7b27ac6671-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z7r2c\" (UID: \"098b3243-a8a5-4ee6-8e82-2e7b27ac6671\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z7r2c" Jul 21 01:45:55.701262 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:55.701235 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/098b3243-a8a5-4ee6-8e82-2e7b27ac6671-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z7r2c\" (UID: \"098b3243-a8a5-4ee6-8e82-2e7b27ac6671\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z7r2c" Jul 21 01:45:55.701262 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:55.701262 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/098b3243-a8a5-4ee6-8e82-2e7b27ac6671-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z7r2c\" (UID: \"098b3243-a8a5-4ee6-8e82-2e7b27ac6671\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z7r2c" Jul 21 01:45:55.701548 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:55.701302 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/098b3243-a8a5-4ee6-8e82-2e7b27ac6671-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z7r2c\" (UID: \"098b3243-a8a5-4ee6-8e82-2e7b27ac6671\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z7r2c" Jul 21 01:45:55.801807 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:55.801775 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/098b3243-a8a5-4ee6-8e82-2e7b27ac6671-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z7r2c\" (UID: \"098b3243-a8a5-4ee6-8e82-2e7b27ac6671\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z7r2c" Jul 21 01:45:55.801991 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:55.801832 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/098b3243-a8a5-4ee6-8e82-2e7b27ac6671-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z7r2c\" (UID: \"098b3243-a8a5-4ee6-8e82-2e7b27ac6671\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z7r2c" Jul 21 01:45:55.801991 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:55.801857 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/098b3243-a8a5-4ee6-8e82-2e7b27ac6671-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z7r2c\" (UID: \"098b3243-a8a5-4ee6-8e82-2e7b27ac6671\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z7r2c" Jul 21 01:45:55.801991 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:55.801872 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/098b3243-a8a5-4ee6-8e82-2e7b27ac6671-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z7r2c\" (UID: \"098b3243-a8a5-4ee6-8e82-2e7b27ac6671\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z7r2c" Jul 21 01:45:55.801991 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:55.801892 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/098b3243-a8a5-4ee6-8e82-2e7b27ac6671-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z7r2c\" (UID: \"098b3243-a8a5-4ee6-8e82-2e7b27ac6671\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z7r2c" Jul 21 01:45:55.801991 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:55.801957 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2d4k\" (UniqueName: \"kubernetes.io/projected/098b3243-a8a5-4ee6-8e82-2e7b27ac6671-kube-api-access-c2d4k\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z7r2c\" (UID: \"098b3243-a8a5-4ee6-8e82-2e7b27ac6671\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z7r2c" Jul 21 01:45:55.802321 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:55.802299 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/098b3243-a8a5-4ee6-8e82-2e7b27ac6671-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z7r2c\" (UID: \"098b3243-a8a5-4ee6-8e82-2e7b27ac6671\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z7r2c" Jul 21 01:45:55.802398 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:55.802343 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/098b3243-a8a5-4ee6-8e82-2e7b27ac6671-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z7r2c\" (UID: \"098b3243-a8a5-4ee6-8e82-2e7b27ac6671\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z7r2c" Jul 21 01:45:55.802483 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:55.802394 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/098b3243-a8a5-4ee6-8e82-2e7b27ac6671-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z7r2c\" (UID: \"098b3243-a8a5-4ee6-8e82-2e7b27ac6671\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z7r2c" Jul 21 01:45:55.804135 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:55.804109 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/098b3243-a8a5-4ee6-8e82-2e7b27ac6671-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z7r2c\" (UID: \"098b3243-a8a5-4ee6-8e82-2e7b27ac6671\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z7r2c" Jul 21 01:45:55.804403 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:55.804385 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/098b3243-a8a5-4ee6-8e82-2e7b27ac6671-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z7r2c\" (UID: \"098b3243-a8a5-4ee6-8e82-2e7b27ac6671\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z7r2c" Jul 21 01:45:55.809185 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:55.809166 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2d4k\" (UniqueName: \"kubernetes.io/projected/098b3243-a8a5-4ee6-8e82-2e7b27ac6671-kube-api-access-c2d4k\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z7r2c\" (UID: \"098b3243-a8a5-4ee6-8e82-2e7b27ac6671\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z7r2c" Jul 21 01:45:55.995220 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:55.995122 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z7r2c" Jul 21 01:45:56.146382 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:56.146208 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z7r2c"] Jul 21 01:45:56.149463 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:45:56.149416 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod098b3243_a8a5_4ee6_8e82_2e7b27ac6671.slice/crio-803a730717508f2bb8ee11e488e3bdb09aeca84933ea0aceda359e6d368dc979 WatchSource:0}: Error finding container 803a730717508f2bb8ee11e488e3bdb09aeca84933ea0aceda359e6d368dc979: Status 404 returned error can't find the container with id 803a730717508f2bb8ee11e488e3bdb09aeca84933ea0aceda359e6d368dc979 Jul 21 01:45:56.213143 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:56.213103 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 01:45:57.092597 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:57.092556 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z7r2c" event={"ID":"098b3243-a8a5-4ee6-8e82-2e7b27ac6671","Type":"ContainerStarted","Data":"803a730717508f2bb8ee11e488e3bdb09aeca84933ea0aceda359e6d368dc979"} Jul 21 01:45:58.099838 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:58.099802 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z7r2c" event={"ID":"098b3243-a8a5-4ee6-8e82-2e7b27ac6671","Type":"ContainerStarted","Data":"ea54b032f188e245f480d5dc1a9f597865cbb63496d9bd6097997334bc89e622"} Jul 21 01:45:58.117568 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:45:58.117513 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z7r2c" podStartSLOduration=1.7761286059999999 podStartE2EDuration="3.117493375s" totalImagesPullingTime="1.341364769s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 01:45:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:45:58.114668861 +0000 UTC m=+581.701834857" watchObservedRunningTime="2026-07-21 01:45:58.117493375 +0000 UTC m=+581.704659349" Jul 21 01:46:05.995595 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:46:05.995561 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z7r2c" Jul 21 01:46:05.996092 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:46:05.995648 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z7r2c" Jul 21 01:46:06.008435 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:46:06.008406 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z7r2c" Jul 21 01:46:06.161521 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:46:06.161487 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z7r2c" Jul 21 01:46:15.295679 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:46:15.295644 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 01:46:16.793951 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:46:16.793920 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-9ctn2_1e930d67-d711-419c-af02-86b43027a95d/cluster-monitoring-operator/0.log" Jul 21 01:46:16.796042 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:46:16.796023 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-9ctn2_1e930d67-d711-419c-af02-86b43027a95d/cluster-monitoring-operator/0.log" Jul 21 01:46:16.798950 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:46:16.798918 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-4mtp7_9c380009-9bf5-4b73-803a-5a1ea9c1abf0/console-operator/1.log" Jul 21 01:46:16.800947 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:46:16.800926 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-4mtp7_9c380009-9bf5-4b73-803a-5a1ea9c1abf0/console-operator/1.log" Jul 21 01:46:31.400039 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:46:31.400001 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 01:46:39.495206 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:46:39.495119 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 01:46:45.295605 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:46:45.295541 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-qvwzz"] Jul 21 01:46:45.299538 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:46:45.299520 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qvwzz" Jul 21 01:46:45.301660 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:46:45.301638 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 21 01:46:45.308581 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:46:45.308558 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-qvwzz"] Jul 21 01:46:45.466020 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:46:45.465978 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c02cf82d-8313-4edb-9f25-e7a93dabecba-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qvwzz\" (UID: \"c02cf82d-8313-4edb-9f25-e7a93dabecba\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qvwzz" Jul 21 01:46:45.466020 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:46:45.466026 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8f8ls\" (UniqueName: \"kubernetes.io/projected/c02cf82d-8313-4edb-9f25-e7a93dabecba-kube-api-access-8f8ls\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qvwzz\" (UID: \"c02cf82d-8313-4edb-9f25-e7a93dabecba\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qvwzz" Jul 21 01:46:45.466292 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:46:45.466093 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c02cf82d-8313-4edb-9f25-e7a93dabecba-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qvwzz\" (UID: \"c02cf82d-8313-4edb-9f25-e7a93dabecba\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qvwzz" Jul 21 01:46:45.466292 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:46:45.466265 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c02cf82d-8313-4edb-9f25-e7a93dabecba-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qvwzz\" (UID: \"c02cf82d-8313-4edb-9f25-e7a93dabecba\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qvwzz" Jul 21 01:46:45.466398 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:46:45.466319 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c02cf82d-8313-4edb-9f25-e7a93dabecba-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qvwzz\" (UID: \"c02cf82d-8313-4edb-9f25-e7a93dabecba\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qvwzz" Jul 21 01:46:45.466398 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:46:45.466376 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c02cf82d-8313-4edb-9f25-e7a93dabecba-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qvwzz\" (UID: \"c02cf82d-8313-4edb-9f25-e7a93dabecba\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qvwzz" Jul 21 01:46:45.567399 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:46:45.567312 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c02cf82d-8313-4edb-9f25-e7a93dabecba-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qvwzz\" (UID: \"c02cf82d-8313-4edb-9f25-e7a93dabecba\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qvwzz" Jul 21 01:46:45.567399 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:46:45.567358 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c02cf82d-8313-4edb-9f25-e7a93dabecba-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qvwzz\" (UID: \"c02cf82d-8313-4edb-9f25-e7a93dabecba\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qvwzz" Jul 21 01:46:45.567399 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:46:45.567376 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c02cf82d-8313-4edb-9f25-e7a93dabecba-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qvwzz\" (UID: \"c02cf82d-8313-4edb-9f25-e7a93dabecba\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qvwzz" Jul 21 01:46:45.567720 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:46:45.567414 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c02cf82d-8313-4edb-9f25-e7a93dabecba-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qvwzz\" (UID: \"c02cf82d-8313-4edb-9f25-e7a93dabecba\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qvwzz" Jul 21 01:46:45.567720 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:46:45.567470 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f8ls\" (UniqueName: \"kubernetes.io/projected/c02cf82d-8313-4edb-9f25-e7a93dabecba-kube-api-access-8f8ls\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qvwzz\" (UID: \"c02cf82d-8313-4edb-9f25-e7a93dabecba\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qvwzz" Jul 21 01:46:45.567720 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:46:45.567500 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c02cf82d-8313-4edb-9f25-e7a93dabecba-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qvwzz\" (UID: \"c02cf82d-8313-4edb-9f25-e7a93dabecba\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qvwzz" Jul 21 01:46:45.567885 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:46:45.567815 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c02cf82d-8313-4edb-9f25-e7a93dabecba-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qvwzz\" (UID: \"c02cf82d-8313-4edb-9f25-e7a93dabecba\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qvwzz" Jul 21 01:46:45.567885 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:46:45.567844 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c02cf82d-8313-4edb-9f25-e7a93dabecba-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qvwzz\" (UID: \"c02cf82d-8313-4edb-9f25-e7a93dabecba\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qvwzz" Jul 21 01:46:45.567967 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:46:45.567948 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c02cf82d-8313-4edb-9f25-e7a93dabecba-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qvwzz\" (UID: \"c02cf82d-8313-4edb-9f25-e7a93dabecba\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qvwzz" Jul 21 01:46:45.569740 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:46:45.569711 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c02cf82d-8313-4edb-9f25-e7a93dabecba-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qvwzz\" (UID: \"c02cf82d-8313-4edb-9f25-e7a93dabecba\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qvwzz" Jul 21 01:46:45.569946 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:46:45.569927 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c02cf82d-8313-4edb-9f25-e7a93dabecba-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qvwzz\" (UID: \"c02cf82d-8313-4edb-9f25-e7a93dabecba\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qvwzz" Jul 21 01:46:45.575004 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:46:45.574983 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f8ls\" (UniqueName: \"kubernetes.io/projected/c02cf82d-8313-4edb-9f25-e7a93dabecba-kube-api-access-8f8ls\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qvwzz\" (UID: \"c02cf82d-8313-4edb-9f25-e7a93dabecba\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qvwzz" Jul 21 01:46:45.609568 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:46:45.609532 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qvwzz" Jul 21 01:46:45.754316 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:46:45.754288 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-qvwzz"] Jul 21 01:46:45.756285 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:46:45.756258 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc02cf82d_8313_4edb_9f25_e7a93dabecba.slice/crio-6e96ac25c21cd7bca86a13800f284ed6ddc1780d0eabb024932944cc922bb3f6 WatchSource:0}: Error finding container 6e96ac25c21cd7bca86a13800f284ed6ddc1780d0eabb024932944cc922bb3f6: Status 404 returned error can't find the container with id 6e96ac25c21cd7bca86a13800f284ed6ddc1780d0eabb024932944cc922bb3f6 Jul 21 01:46:45.757650 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:46:45.757630 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 01:46:46.370869 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:46:46.370829 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qvwzz" event={"ID":"c02cf82d-8313-4edb-9f25-e7a93dabecba","Type":"ContainerStarted","Data":"c703e9faf9e3d7982516c8f15c1c1956c2603c7f8d6a440eb74a41150a6e39a6"} Jul 21 01:46:46.371267 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:46:46.370880 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qvwzz" event={"ID":"c02cf82d-8313-4edb-9f25-e7a93dabecba","Type":"ContainerStarted","Data":"6e96ac25c21cd7bca86a13800f284ed6ddc1780d0eabb024932944cc922bb3f6"} Jul 21 01:46:46.387371 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:46:46.387320 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qvwzz" podStartSLOduration=1.185631286 podStartE2EDuration="1.387302725s" totalImagesPullingTime="201.671439ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 01:46:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:46:46.386569762 +0000 UTC m=+629.973735740" watchObservedRunningTime="2026-07-21 01:46:46.387302725 +0000 UTC m=+629.974468703" Jul 21 01:46:48.792469 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:46:48.792411 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 01:46:55.609718 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:46:55.609678 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qvwzz" Jul 21 01:46:55.610173 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:46:55.609737 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qvwzz" Jul 21 01:46:55.623576 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:46:55.623543 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qvwzz" Jul 21 01:46:56.438198 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:46:56.438172 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qvwzz" Jul 21 01:48:21.243373 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:48:21.243296 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 01:48:31.304946 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:48:31.304908 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 01:48:38.783845 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:48:38.783802 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5cd8fcb89-xrcw8"] Jul 21 01:48:38.784309 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:48:38.784062 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5cd8fcb89-xrcw8" podUID="e485a5c0-b377-44b7-a923-c07a8aa5f928" containerName="manager" containerID="cri-o://9eeb7ce58b66e263fd439f44156479561fc595c0c4c6a2a6e0fd7a7fe441c64d" gracePeriod=30 Jul 21 01:48:38.995696 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:48:38.995660 2580 generic.go:356] "Generic (PLEG): container finished" podID="e485a5c0-b377-44b7-a923-c07a8aa5f928" containerID="9eeb7ce58b66e263fd439f44156479561fc595c0c4c6a2a6e0fd7a7fe441c64d" exitCode=0 Jul 21 01:48:38.995868 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:48:38.995729 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5cd8fcb89-xrcw8" event={"ID":"e485a5c0-b377-44b7-a923-c07a8aa5f928","Type":"ContainerDied","Data":"9eeb7ce58b66e263fd439f44156479561fc595c0c4c6a2a6e0fd7a7fe441c64d"} Jul 21 01:48:39.048431 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:48:39.048354 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5cd8fcb89-xrcw8" Jul 21 01:48:39.153625 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:48:39.153579 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e485a5c0-b377-44b7-a923-c07a8aa5f928-webhook-cert\") pod \"e485a5c0-b377-44b7-a923-c07a8aa5f928\" (UID: \"e485a5c0-b377-44b7-a923-c07a8aa5f928\") " Jul 21 01:48:39.153802 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:48:39.153692 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kqhnf\" (UniqueName: \"kubernetes.io/projected/e485a5c0-b377-44b7-a923-c07a8aa5f928-kube-api-access-kqhnf\") pod \"e485a5c0-b377-44b7-a923-c07a8aa5f928\" (UID: \"e485a5c0-b377-44b7-a923-c07a8aa5f928\") " Jul 21 01:48:39.155770 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:48:39.155740 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e485a5c0-b377-44b7-a923-c07a8aa5f928-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "e485a5c0-b377-44b7-a923-c07a8aa5f928" (UID: "e485a5c0-b377-44b7-a923-c07a8aa5f928"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 01:48:39.155845 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:48:39.155804 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e485a5c0-b377-44b7-a923-c07a8aa5f928-kube-api-access-kqhnf" (OuterVolumeSpecName: "kube-api-access-kqhnf") pod "e485a5c0-b377-44b7-a923-c07a8aa5f928" (UID: "e485a5c0-b377-44b7-a923-c07a8aa5f928"). InnerVolumeSpecName "kube-api-access-kqhnf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 01:48:39.254922 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:48:39.254879 2580 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e485a5c0-b377-44b7-a923-c07a8aa5f928-webhook-cert\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 21 01:48:39.254922 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:48:39.254911 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqhnf\" (UniqueName: \"kubernetes.io/projected/e485a5c0-b377-44b7-a923-c07a8aa5f928-kube-api-access-kqhnf\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 21 01:48:40.002347 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:48:40.002320 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5cd8fcb89-xrcw8" Jul 21 01:48:40.002847 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:48:40.002318 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5cd8fcb89-xrcw8" event={"ID":"e485a5c0-b377-44b7-a923-c07a8aa5f928","Type":"ContainerDied","Data":"9423245c65f6478a30389fb77adb717bc47bf3f8ec9f58c6acc2a762f9a13033"} Jul 21 01:48:40.002847 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:48:40.002445 2580 scope.go:117] "RemoveContainer" containerID="9eeb7ce58b66e263fd439f44156479561fc595c0c4c6a2a6e0fd7a7fe441c64d" Jul 21 01:48:40.024770 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:48:40.024742 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5cd8fcb89-xrcw8"] Jul 21 01:48:40.026790 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:48:40.026767 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5cd8fcb89-xrcw8"] Jul 21 01:48:40.877961 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:48:40.877919 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e485a5c0-b377-44b7-a923-c07a8aa5f928" path="/var/lib/kubelet/pods/e485a5c0-b377-44b7-a923-c07a8aa5f928/volumes" Jul 21 01:48:41.896348 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:48:41.896313 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 01:49:40.865927 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:49:40.865844 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5cd8fcb89-v57hj"] Jul 21 01:49:40.866356 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:49:40.866340 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e485a5c0-b377-44b7-a923-c07a8aa5f928" containerName="manager" Jul 21 01:49:40.866405 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:49:40.866358 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e485a5c0-b377-44b7-a923-c07a8aa5f928" containerName="manager" Jul 21 01:49:40.866442 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:49:40.866417 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e485a5c0-b377-44b7-a923-c07a8aa5f928" containerName="manager" Jul 21 01:49:40.870843 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:49:40.870816 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5cd8fcb89-v57hj" Jul 21 01:49:40.874469 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:49:40.874433 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 21 01:49:40.874606 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:49:40.874433 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-2mqz7\"" Jul 21 01:49:40.874606 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:49:40.874506 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 21 01:49:40.880394 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:49:40.880368 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5cd8fcb89-v57hj"] Jul 21 01:49:40.975953 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:49:40.975912 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhx8b\" (UniqueName: \"kubernetes.io/projected/5b203646-5e44-412b-9c0b-3088af4b8e90-kube-api-access-xhx8b\") pod \"maas-controller-5cd8fcb89-v57hj\" (UID: \"5b203646-5e44-412b-9c0b-3088af4b8e90\") " pod="opendatahub/maas-controller-5cd8fcb89-v57hj" Jul 21 01:49:40.976163 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:49:40.975985 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5b203646-5e44-412b-9c0b-3088af4b8e90-webhook-cert\") pod \"maas-controller-5cd8fcb89-v57hj\" (UID: \"5b203646-5e44-412b-9c0b-3088af4b8e90\") " pod="opendatahub/maas-controller-5cd8fcb89-v57hj" Jul 21 01:49:41.076798 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:49:41.076761 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhx8b\" (UniqueName: \"kubernetes.io/projected/5b203646-5e44-412b-9c0b-3088af4b8e90-kube-api-access-xhx8b\") pod \"maas-controller-5cd8fcb89-v57hj\" (UID: \"5b203646-5e44-412b-9c0b-3088af4b8e90\") " pod="opendatahub/maas-controller-5cd8fcb89-v57hj" Jul 21 01:49:41.076981 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:49:41.076822 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5b203646-5e44-412b-9c0b-3088af4b8e90-webhook-cert\") pod \"maas-controller-5cd8fcb89-v57hj\" (UID: \"5b203646-5e44-412b-9c0b-3088af4b8e90\") " pod="opendatahub/maas-controller-5cd8fcb89-v57hj" Jul 21 01:49:41.079417 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:49:41.079396 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5b203646-5e44-412b-9c0b-3088af4b8e90-webhook-cert\") pod \"maas-controller-5cd8fcb89-v57hj\" (UID: \"5b203646-5e44-412b-9c0b-3088af4b8e90\") " pod="opendatahub/maas-controller-5cd8fcb89-v57hj" Jul 21 01:49:41.085492 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:49:41.085472 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhx8b\" (UniqueName: \"kubernetes.io/projected/5b203646-5e44-412b-9c0b-3088af4b8e90-kube-api-access-xhx8b\") pod \"maas-controller-5cd8fcb89-v57hj\" (UID: \"5b203646-5e44-412b-9c0b-3088af4b8e90\") " pod="opendatahub/maas-controller-5cd8fcb89-v57hj" Jul 21 01:49:41.184393 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:49:41.184309 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5cd8fcb89-v57hj" Jul 21 01:49:41.346971 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:49:41.346938 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5cd8fcb89-v57hj"] Jul 21 01:49:41.348664 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:49:41.348636 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b203646_5e44_412b_9c0b_3088af4b8e90.slice/crio-4902776818fb4aaa04e21ffb37f799b75ac39d381d0216e748c496e78e342685 WatchSource:0}: Error finding container 4902776818fb4aaa04e21ffb37f799b75ac39d381d0216e748c496e78e342685: Status 404 returned error can't find the container with id 4902776818fb4aaa04e21ffb37f799b75ac39d381d0216e748c496e78e342685 Jul 21 01:49:42.346028 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:49:42.345994 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5cd8fcb89-v57hj" event={"ID":"5b203646-5e44-412b-9c0b-3088af4b8e90","Type":"ContainerStarted","Data":"ada862326014f735431558d2237938b15ffdff20fefaf6487b9e4e2b8073eb69"} Jul 21 01:49:42.346487 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:49:42.346044 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5cd8fcb89-v57hj" event={"ID":"5b203646-5e44-412b-9c0b-3088af4b8e90","Type":"ContainerStarted","Data":"4902776818fb4aaa04e21ffb37f799b75ac39d381d0216e748c496e78e342685"} Jul 21 01:49:42.346487 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:49:42.346144 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5cd8fcb89-v57hj" Jul 21 01:49:42.365972 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:49:42.365920 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5cd8fcb89-v57hj" podStartSLOduration=1.904212518 podStartE2EDuration="2.365903354s" totalImagesPullingTime="461.690836ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 01:49:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:49:42.363602512 +0000 UTC m=+805.950768489" watchObservedRunningTime="2026-07-21 01:49:42.365903354 +0000 UTC m=+805.953069333" Jul 21 01:49:53.353835 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:49:53.353806 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5cd8fcb89-v57hj" Jul 21 01:49:57.991977 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:49:57.991930 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5cd8fcb89-v57hj"] Jul 21 01:49:57.992517 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:49:57.992186 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5cd8fcb89-v57hj" podUID="5b203646-5e44-412b-9c0b-3088af4b8e90" containerName="manager" containerID="cri-o://ada862326014f735431558d2237938b15ffdff20fefaf6487b9e4e2b8073eb69" gracePeriod=30 Jul 21 01:49:58.258018 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:49:58.257991 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5cd8fcb89-v57hj" Jul 21 01:49:58.347707 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:49:58.347669 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xhx8b\" (UniqueName: \"kubernetes.io/projected/5b203646-5e44-412b-9c0b-3088af4b8e90-kube-api-access-xhx8b\") pod \"5b203646-5e44-412b-9c0b-3088af4b8e90\" (UID: \"5b203646-5e44-412b-9c0b-3088af4b8e90\") " Jul 21 01:49:58.347855 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:49:58.347799 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5b203646-5e44-412b-9c0b-3088af4b8e90-webhook-cert\") pod \"5b203646-5e44-412b-9c0b-3088af4b8e90\" (UID: \"5b203646-5e44-412b-9c0b-3088af4b8e90\") " Jul 21 01:49:58.349827 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:49:58.349793 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b203646-5e44-412b-9c0b-3088af4b8e90-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "5b203646-5e44-412b-9c0b-3088af4b8e90" (UID: "5b203646-5e44-412b-9c0b-3088af4b8e90"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 01:49:58.349969 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:49:58.349829 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b203646-5e44-412b-9c0b-3088af4b8e90-kube-api-access-xhx8b" (OuterVolumeSpecName: "kube-api-access-xhx8b") pod "5b203646-5e44-412b-9c0b-3088af4b8e90" (UID: "5b203646-5e44-412b-9c0b-3088af4b8e90"). InnerVolumeSpecName "kube-api-access-xhx8b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 01:49:58.435800 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:49:58.435766 2580 generic.go:356] "Generic (PLEG): container finished" podID="5b203646-5e44-412b-9c0b-3088af4b8e90" containerID="ada862326014f735431558d2237938b15ffdff20fefaf6487b9e4e2b8073eb69" exitCode=0 Jul 21 01:49:58.435992 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:49:58.435821 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5cd8fcb89-v57hj" Jul 21 01:49:58.435992 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:49:58.435849 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5cd8fcb89-v57hj" event={"ID":"5b203646-5e44-412b-9c0b-3088af4b8e90","Type":"ContainerDied","Data":"ada862326014f735431558d2237938b15ffdff20fefaf6487b9e4e2b8073eb69"} Jul 21 01:49:58.435992 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:49:58.435903 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5cd8fcb89-v57hj" event={"ID":"5b203646-5e44-412b-9c0b-3088af4b8e90","Type":"ContainerDied","Data":"4902776818fb4aaa04e21ffb37f799b75ac39d381d0216e748c496e78e342685"} Jul 21 01:49:58.435992 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:49:58.435926 2580 scope.go:117] "RemoveContainer" containerID="ada862326014f735431558d2237938b15ffdff20fefaf6487b9e4e2b8073eb69" Jul 21 01:49:58.445893 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:49:58.445865 2580 scope.go:117] "RemoveContainer" containerID="ada862326014f735431558d2237938b15ffdff20fefaf6487b9e4e2b8073eb69" Jul 21 01:49:58.446205 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:49:58.446147 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ada862326014f735431558d2237938b15ffdff20fefaf6487b9e4e2b8073eb69\": container with ID starting with ada862326014f735431558d2237938b15ffdff20fefaf6487b9e4e2b8073eb69 not found: ID does not exist" containerID="ada862326014f735431558d2237938b15ffdff20fefaf6487b9e4e2b8073eb69" Jul 21 01:49:58.446255 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:49:58.446193 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ada862326014f735431558d2237938b15ffdff20fefaf6487b9e4e2b8073eb69"} err="failed to get container status \"ada862326014f735431558d2237938b15ffdff20fefaf6487b9e4e2b8073eb69\": rpc error: code = NotFound desc = could not find container \"ada862326014f735431558d2237938b15ffdff20fefaf6487b9e4e2b8073eb69\": container with ID starting with ada862326014f735431558d2237938b15ffdff20fefaf6487b9e4e2b8073eb69 not found: ID does not exist" Jul 21 01:49:58.449342 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:49:58.449321 2580 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5b203646-5e44-412b-9c0b-3088af4b8e90-webhook-cert\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 21 01:49:58.449342 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:49:58.449343 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhx8b\" (UniqueName: \"kubernetes.io/projected/5b203646-5e44-412b-9c0b-3088af4b8e90-kube-api-access-xhx8b\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 21 01:49:58.460065 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:49:58.460039 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5cd8fcb89-v57hj"] Jul 21 01:49:58.465440 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:49:58.465416 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5cd8fcb89-v57hj"] Jul 21 01:49:58.877871 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:49:58.877840 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b203646-5e44-412b-9c0b-3088af4b8e90" path="/var/lib/kubelet/pods/5b203646-5e44-412b-9c0b-3088af4b8e90/volumes" Jul 21 01:49:59.280081 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:49:59.280047 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5cd8fcb89-nf9zb"] Jul 21 01:49:59.280564 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:49:59.280548 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b203646-5e44-412b-9c0b-3088af4b8e90" containerName="manager" Jul 21 01:49:59.280622 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:49:59.280566 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b203646-5e44-412b-9c0b-3088af4b8e90" containerName="manager" Jul 21 01:49:59.280658 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:49:59.280651 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b203646-5e44-412b-9c0b-3088af4b8e90" containerName="manager" Jul 21 01:49:59.285172 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:49:59.285152 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5cd8fcb89-nf9zb" Jul 21 01:49:59.288759 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:49:59.288736 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-2mqz7\"" Jul 21 01:49:59.288892 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:49:59.288765 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 21 01:49:59.288892 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:49:59.288799 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 21 01:49:59.300155 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:49:59.300130 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5cd8fcb89-nf9zb"] Jul 21 01:49:59.458307 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:49:59.458273 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqhhr\" (UniqueName: \"kubernetes.io/projected/3c2ad5c2-e85c-49c3-8e07-adfec292999c-kube-api-access-gqhhr\") pod \"maas-controller-5cd8fcb89-nf9zb\" (UID: \"3c2ad5c2-e85c-49c3-8e07-adfec292999c\") " pod="opendatahub/maas-controller-5cd8fcb89-nf9zb" Jul 21 01:49:59.458524 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:49:59.458329 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3c2ad5c2-e85c-49c3-8e07-adfec292999c-webhook-cert\") pod \"maas-controller-5cd8fcb89-nf9zb\" (UID: \"3c2ad5c2-e85c-49c3-8e07-adfec292999c\") " pod="opendatahub/maas-controller-5cd8fcb89-nf9zb" Jul 21 01:49:59.558981 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:49:59.558875 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqhhr\" (UniqueName: \"kubernetes.io/projected/3c2ad5c2-e85c-49c3-8e07-adfec292999c-kube-api-access-gqhhr\") pod \"maas-controller-5cd8fcb89-nf9zb\" (UID: \"3c2ad5c2-e85c-49c3-8e07-adfec292999c\") " pod="opendatahub/maas-controller-5cd8fcb89-nf9zb" Jul 21 01:49:59.558981 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:49:59.558955 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3c2ad5c2-e85c-49c3-8e07-adfec292999c-webhook-cert\") pod \"maas-controller-5cd8fcb89-nf9zb\" (UID: \"3c2ad5c2-e85c-49c3-8e07-adfec292999c\") " pod="opendatahub/maas-controller-5cd8fcb89-nf9zb" Jul 21 01:49:59.561482 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:49:59.561434 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3c2ad5c2-e85c-49c3-8e07-adfec292999c-webhook-cert\") pod \"maas-controller-5cd8fcb89-nf9zb\" (UID: \"3c2ad5c2-e85c-49c3-8e07-adfec292999c\") " pod="opendatahub/maas-controller-5cd8fcb89-nf9zb" Jul 21 01:49:59.567313 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:49:59.567284 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqhhr\" (UniqueName: \"kubernetes.io/projected/3c2ad5c2-e85c-49c3-8e07-adfec292999c-kube-api-access-gqhhr\") pod \"maas-controller-5cd8fcb89-nf9zb\" (UID: \"3c2ad5c2-e85c-49c3-8e07-adfec292999c\") " pod="opendatahub/maas-controller-5cd8fcb89-nf9zb" Jul 21 01:49:59.595191 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:49:59.595154 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5cd8fcb89-nf9zb" Jul 21 01:49:59.749670 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:49:59.749639 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5cd8fcb89-nf9zb"] Jul 21 01:49:59.752608 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:49:59.752579 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c2ad5c2_e85c_49c3_8e07_adfec292999c.slice/crio-3ef45ca1692ff09811e9cd93cfe4bd7a9af96e992fafa22c392b4b61f8f63df5 WatchSource:0}: Error finding container 3ef45ca1692ff09811e9cd93cfe4bd7a9af96e992fafa22c392b4b61f8f63df5: Status 404 returned error can't find the container with id 3ef45ca1692ff09811e9cd93cfe4bd7a9af96e992fafa22c392b4b61f8f63df5 Jul 21 01:50:00.450989 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:50:00.450896 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5cd8fcb89-nf9zb" event={"ID":"3c2ad5c2-e85c-49c3-8e07-adfec292999c","Type":"ContainerStarted","Data":"f0f14f4842bcf28234ab17b433bcbd190e3aea4d25cc3961a567aa922a5bd6f4"} Jul 21 01:50:00.450989 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:50:00.450945 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5cd8fcb89-nf9zb" event={"ID":"3c2ad5c2-e85c-49c3-8e07-adfec292999c","Type":"ContainerStarted","Data":"3ef45ca1692ff09811e9cd93cfe4bd7a9af96e992fafa22c392b4b61f8f63df5"} Jul 21 01:50:00.450989 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:50:00.450977 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5cd8fcb89-nf9zb" Jul 21 01:50:00.469650 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:50:00.469602 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5cd8fcb89-nf9zb" podStartSLOduration=1.139026119 podStartE2EDuration="1.469583233s" totalImagesPullingTime="330.557114ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 01:49:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:50:00.467664387 +0000 UTC m=+824.054830366" watchObservedRunningTime="2026-07-21 01:50:00.469583233 +0000 UTC m=+824.056749267" Jul 21 01:50:11.458788 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:50:11.458759 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5cd8fcb89-nf9zb" Jul 21 01:50:23.998992 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:50:23.998957 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 01:50:57.492849 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:50:57.492808 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 01:51:06.502138 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:51:06.502100 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 01:51:16.493233 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:51:16.493148 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 01:51:16.832935 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:51:16.832903 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-9ctn2_1e930d67-d711-419c-af02-86b43027a95d/cluster-monitoring-operator/0.log" Jul 21 01:51:16.833455 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:51:16.833432 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-9ctn2_1e930d67-d711-419c-af02-86b43027a95d/cluster-monitoring-operator/0.log" Jul 21 01:51:16.838434 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:51:16.838410 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-4mtp7_9c380009-9bf5-4b73-803a-5a1ea9c1abf0/console-operator/1.log" Jul 21 01:51:16.838956 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:51:16.838931 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-4mtp7_9c380009-9bf5-4b73-803a-5a1ea9c1abf0/console-operator/1.log" Jul 21 01:51:24.092015 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:51:24.091975 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 01:51:34.595523 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:51:34.595491 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 01:51:43.395265 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:51:43.395226 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 01:51:55.897678 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:51:55.897640 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 01:52:06.190467 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:52:06.190412 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 01:52:28.091685 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:52:28.091649 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 01:52:39.796362 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:52:39.796282 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 01:53:03.399071 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:53:03.399031 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 01:53:13.695512 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:53:13.695463 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 01:53:28.898972 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:53:28.898939 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 01:53:40.001548 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:53:40.001500 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 01:53:49.604148 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:53:49.604109 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 01:53:59.195430 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:53:59.195394 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 01:54:08.693233 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:54:08.693149 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 01:54:34.462813 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:54:34.462778 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-smdz7"] Jul 21 01:54:34.463372 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:54:34.462996 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-smdz7" podUID="e0ff7d70-51eb-4155-8c32-6d1173257ae5" containerName="manager" containerID="cri-o://31d37d063f5d8f397b419f63e09185c67c0b1711f71d36b11beb1c647db301f2" gracePeriod=10 Jul 21 01:54:37.826381 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:54:37.826355 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-smdz7" Jul 21 01:54:37.977346 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:54:37.977256 2580 generic.go:356] "Generic (PLEG): container finished" podID="e0ff7d70-51eb-4155-8c32-6d1173257ae5" containerID="31d37d063f5d8f397b419f63e09185c67c0b1711f71d36b11beb1c647db301f2" exitCode=0 Jul 21 01:54:37.977346 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:54:37.977303 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-smdz7" Jul 21 01:54:37.977346 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:54:37.977321 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-smdz7" event={"ID":"e0ff7d70-51eb-4155-8c32-6d1173257ae5","Type":"ContainerDied","Data":"31d37d063f5d8f397b419f63e09185c67c0b1711f71d36b11beb1c647db301f2"} Jul 21 01:54:37.977684 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:54:37.977376 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-smdz7" event={"ID":"e0ff7d70-51eb-4155-8c32-6d1173257ae5","Type":"ContainerDied","Data":"401bbb7d800f0257e24258980397733ede618875edca8777653fe5e66471d407"} Jul 21 01:54:37.977684 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:54:37.977404 2580 scope.go:117] "RemoveContainer" containerID="31d37d063f5d8f397b419f63e09185c67c0b1711f71d36b11beb1c647db301f2" Jul 21 01:54:37.982897 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:54:37.982874 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cmkgg\" (UniqueName: \"kubernetes.io/projected/e0ff7d70-51eb-4155-8c32-6d1173257ae5-kube-api-access-cmkgg\") pod \"e0ff7d70-51eb-4155-8c32-6d1173257ae5\" (UID: \"e0ff7d70-51eb-4155-8c32-6d1173257ae5\") " Jul 21 01:54:37.983047 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:54:37.983030 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e0ff7d70-51eb-4155-8c32-6d1173257ae5-extensions-socket-volume\") pod \"e0ff7d70-51eb-4155-8c32-6d1173257ae5\" (UID: \"e0ff7d70-51eb-4155-8c32-6d1173257ae5\") " Jul 21 01:54:37.983399 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:54:37.983369 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0ff7d70-51eb-4155-8c32-6d1173257ae5-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "e0ff7d70-51eb-4155-8c32-6d1173257ae5" (UID: "e0ff7d70-51eb-4155-8c32-6d1173257ae5"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 01:54:37.984907 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:54:37.984886 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0ff7d70-51eb-4155-8c32-6d1173257ae5-kube-api-access-cmkgg" (OuterVolumeSpecName: "kube-api-access-cmkgg") pod "e0ff7d70-51eb-4155-8c32-6d1173257ae5" (UID: "e0ff7d70-51eb-4155-8c32-6d1173257ae5"). InnerVolumeSpecName "kube-api-access-cmkgg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 01:54:37.988543 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:54:37.988526 2580 scope.go:117] "RemoveContainer" containerID="31d37d063f5d8f397b419f63e09185c67c0b1711f71d36b11beb1c647db301f2" Jul 21 01:54:37.988829 ip-10-0-139-183 kubenswrapper[2580]: E0721 01:54:37.988808 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31d37d063f5d8f397b419f63e09185c67c0b1711f71d36b11beb1c647db301f2\": container with ID starting with 31d37d063f5d8f397b419f63e09185c67c0b1711f71d36b11beb1c647db301f2 not found: ID does not exist" containerID="31d37d063f5d8f397b419f63e09185c67c0b1711f71d36b11beb1c647db301f2" Jul 21 01:54:37.988907 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:54:37.988837 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31d37d063f5d8f397b419f63e09185c67c0b1711f71d36b11beb1c647db301f2"} err="failed to get container status \"31d37d063f5d8f397b419f63e09185c67c0b1711f71d36b11beb1c647db301f2\": rpc error: code = NotFound desc = could not find container \"31d37d063f5d8f397b419f63e09185c67c0b1711f71d36b11beb1c647db301f2\": container with ID starting with 31d37d063f5d8f397b419f63e09185c67c0b1711f71d36b11beb1c647db301f2 not found: ID does not exist" Jul 21 01:54:38.084612 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:54:38.084582 2580 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e0ff7d70-51eb-4155-8c32-6d1173257ae5-extensions-socket-volume\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 21 01:54:38.084612 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:54:38.084607 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmkgg\" (UniqueName: \"kubernetes.io/projected/e0ff7d70-51eb-4155-8c32-6d1173257ae5-kube-api-access-cmkgg\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 21 01:54:38.299942 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:54:38.299912 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-smdz7"] Jul 21 01:54:38.303960 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:54:38.303935 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-smdz7"] Jul 21 01:54:38.877232 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:54:38.877198 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0ff7d70-51eb-4155-8c32-6d1173257ae5" path="/var/lib/kubelet/pods/e0ff7d70-51eb-4155-8c32-6d1173257ae5/volumes" Jul 21 01:55:35.643829 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:55:35.643786 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8j6sk"] Jul 21 01:55:35.644445 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:55:35.644424 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0ff7d70-51eb-4155-8c32-6d1173257ae5" containerName="manager" Jul 21 01:55:35.644563 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:55:35.644463 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0ff7d70-51eb-4155-8c32-6d1173257ae5" containerName="manager" Jul 21 01:55:35.644632 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:55:35.644618 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0ff7d70-51eb-4155-8c32-6d1173257ae5" containerName="manager" Jul 21 01:55:35.649034 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:55:35.649013 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8j6sk" Jul 21 01:55:35.651651 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:55:35.651629 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-l692r\"" Jul 21 01:55:35.660906 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:55:35.660885 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8j6sk"] Jul 21 01:55:35.728058 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:55:35.728026 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lv6k\" (UniqueName: \"kubernetes.io/projected/0b10c41e-3302-47f3-8b6b-4057c89a130e-kube-api-access-7lv6k\") pod \"kuadrant-operator-controller-manager-86878c769f-8j6sk\" (UID: \"0b10c41e-3302-47f3-8b6b-4057c89a130e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8j6sk" Jul 21 01:55:35.728227 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:55:35.728064 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0b10c41e-3302-47f3-8b6b-4057c89a130e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-8j6sk\" (UID: \"0b10c41e-3302-47f3-8b6b-4057c89a130e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8j6sk" Jul 21 01:55:35.828857 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:55:35.828817 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lv6k\" (UniqueName: \"kubernetes.io/projected/0b10c41e-3302-47f3-8b6b-4057c89a130e-kube-api-access-7lv6k\") pod \"kuadrant-operator-controller-manager-86878c769f-8j6sk\" (UID: \"0b10c41e-3302-47f3-8b6b-4057c89a130e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8j6sk" Jul 21 01:55:35.828857 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:55:35.828860 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0b10c41e-3302-47f3-8b6b-4057c89a130e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-8j6sk\" (UID: \"0b10c41e-3302-47f3-8b6b-4057c89a130e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8j6sk" Jul 21 01:55:35.829244 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:55:35.829226 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0b10c41e-3302-47f3-8b6b-4057c89a130e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-8j6sk\" (UID: \"0b10c41e-3302-47f3-8b6b-4057c89a130e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8j6sk" Jul 21 01:55:35.837204 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:55:35.837174 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lv6k\" (UniqueName: \"kubernetes.io/projected/0b10c41e-3302-47f3-8b6b-4057c89a130e-kube-api-access-7lv6k\") pod \"kuadrant-operator-controller-manager-86878c769f-8j6sk\" (UID: \"0b10c41e-3302-47f3-8b6b-4057c89a130e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8j6sk" Jul 21 01:55:35.959943 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:55:35.959844 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8j6sk" Jul 21 01:55:36.119558 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:55:36.119508 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8j6sk"] Jul 21 01:55:36.120776 ip-10-0-139-183 kubenswrapper[2580]: W0721 01:55:36.120751 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b10c41e_3302_47f3_8b6b_4057c89a130e.slice/crio-778982b61fdb74ce0328cfe725d40c15db051b36f1a4e5b169bcd15ed4907665 WatchSource:0}: Error finding container 778982b61fdb74ce0328cfe725d40c15db051b36f1a4e5b169bcd15ed4907665: Status 404 returned error can't find the container with id 778982b61fdb74ce0328cfe725d40c15db051b36f1a4e5b169bcd15ed4907665 Jul 21 01:55:36.285732 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:55:36.285694 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8j6sk" event={"ID":"0b10c41e-3302-47f3-8b6b-4057c89a130e","Type":"ContainerStarted","Data":"8e0c595f6bbc5a189a2ec79f76554cc1e7d85e466499728690bb4679d2a0342a"} Jul 21 01:55:36.285921 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:55:36.285757 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8j6sk" event={"ID":"0b10c41e-3302-47f3-8b6b-4057c89a130e","Type":"ContainerStarted","Data":"778982b61fdb74ce0328cfe725d40c15db051b36f1a4e5b169bcd15ed4907665"} Jul 21 01:55:36.285921 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:55:36.285784 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8j6sk" Jul 21 01:55:36.308998 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:55:36.308945 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8j6sk" podStartSLOduration=1.30893028 podStartE2EDuration="1.30893028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 01:55:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:55:36.306513724 +0000 UTC m=+1159.893679722" watchObservedRunningTime="2026-07-21 01:55:36.30893028 +0000 UTC m=+1159.896096257" Jul 21 01:55:47.294429 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:55:47.294344 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8j6sk" Jul 21 01:55:59.423227 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:55:59.423182 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 01:56:11.196896 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:56:11.196860 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 01:56:16.869253 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:56:16.869222 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-9ctn2_1e930d67-d711-419c-af02-86b43027a95d/cluster-monitoring-operator/0.log" Jul 21 01:56:16.871001 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:56:16.870968 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-9ctn2_1e930d67-d711-419c-af02-86b43027a95d/cluster-monitoring-operator/0.log" Jul 21 01:56:16.875085 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:56:16.875056 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-4mtp7_9c380009-9bf5-4b73-803a-5a1ea9c1abf0/console-operator/1.log" Jul 21 01:56:16.877194 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:56:16.877173 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-4mtp7_9c380009-9bf5-4b73-803a-5a1ea9c1abf0/console-operator/1.log" Jul 21 01:56:19.704824 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:56:19.704785 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 01:56:32.099023 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:56:32.098987 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 01:56:42.796732 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:56:42.796696 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 01:56:54.099563 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:56:54.099522 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 01:57:03.398370 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:57:03.398331 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 01:57:13.396114 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:57:13.396022 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 01:57:24.295885 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:57:24.295839 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 01:57:33.739415 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:57:33.739375 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 01:57:44.296541 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:57:44.296501 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 01:58:08.997948 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:58:08.997912 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 01:58:16.397405 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:58:16.397366 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 01:58:34.095189 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:58:34.095151 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 01:58:51.902985 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:58:51.902955 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 01:59:17.198151 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:59:17.198117 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 01:59:32.395678 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:59:32.395641 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 01:59:39.892155 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:59:39.892115 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 01:59:56.105514 ip-10-0-139-183 kubenswrapper[2580]: I0721 01:59:56.105464 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 02:00:05.893774 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:00:05.893732 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 02:00:15.204670 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:00:15.204578 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 02:00:23.391859 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:00:23.391825 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 02:00:34.290358 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:00:34.290317 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 02:00:54.095382 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:00:54.095344 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 02:01:01.597772 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:01:01.597737 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 02:01:10.820828 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:01:10.820784 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 02:01:16.907742 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:01:16.907714 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-9ctn2_1e930d67-d711-419c-af02-86b43027a95d/cluster-monitoring-operator/0.log" Jul 21 02:01:16.911542 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:01:16.911515 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-9ctn2_1e930d67-d711-419c-af02-86b43027a95d/cluster-monitoring-operator/0.log" Jul 21 02:01:16.913137 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:01:16.913112 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-4mtp7_9c380009-9bf5-4b73-803a-5a1ea9c1abf0/console-operator/1.log" Jul 21 02:01:16.916848 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:01:16.916829 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-4mtp7_9c380009-9bf5-4b73-803a-5a1ea9c1abf0/console-operator/1.log" Jul 21 02:01:20.309608 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:01:20.309575 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 02:01:30.109355 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:01:30.109314 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 02:01:42.896275 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:01:42.896237 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 02:01:52.293892 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:01:52.293861 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 02:02:04.996725 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:02:04.996689 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 02:02:17.195398 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:02:17.195364 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 02:02:27.098151 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:02:27.098114 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 02:02:40.896798 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:02:40.896764 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 02:02:50.697231 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:02:50.697201 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 02:03:00.157666 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:03:00.157628 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 02:03:11.193664 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:03:11.193578 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 02:03:29.596205 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:03:29.596156 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-604509c9-lkx2j"] Jul 21 02:03:29.599601 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:03:29.599576 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-604509c9-lkx2j" Jul 21 02:03:29.602093 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:03:29.602064 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-x582z\"" Jul 21 02:03:29.602235 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:03:29.602107 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 21 02:03:29.607142 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:03:29.607112 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-604509c9-lkx2j"] Jul 21 02:03:29.768099 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:03:29.768054 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca5bc022-0ecd-43d5-9a26-0c89457efa1f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-604509c9-lkx2j\" (UID: \"ca5bc022-0ecd-43d5-9a26-0c89457efa1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-604509c9-lkx2j" Jul 21 02:03:29.869370 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:03:29.869274 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca5bc022-0ecd-43d5-9a26-0c89457efa1f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-604509c9-lkx2j\" (UID: \"ca5bc022-0ecd-43d5-9a26-0c89457efa1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-604509c9-lkx2j" Jul 21 02:03:29.869949 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:03:29.869926 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca5bc022-0ecd-43d5-9a26-0c89457efa1f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-604509c9-lkx2j\" (UID: \"ca5bc022-0ecd-43d5-9a26-0c89457efa1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-604509c9-lkx2j" Jul 21 02:03:29.912996 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:03:29.912964 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-604509c9-lkx2j" Jul 21 02:03:30.057086 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:03:30.057062 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-604509c9-lkx2j"] Jul 21 02:03:30.059831 ip-10-0-139-183 kubenswrapper[2580]: W0721 02:03:30.059796 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca5bc022_0ecd_43d5_9a26_0c89457efa1f.slice/crio-c0131842d2ee4c5698e480624525555fa792c63ae1bc9e562b67f25463e2ccb9 WatchSource:0}: Error finding container c0131842d2ee4c5698e480624525555fa792c63ae1bc9e562b67f25463e2ccb9: Status 404 returned error can't find the container with id c0131842d2ee4c5698e480624525555fa792c63ae1bc9e562b67f25463e2ccb9 Jul 21 02:03:30.062005 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:03:30.061985 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 02:03:30.908552 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:03:30.908510 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-604509c9-lkx2j" event={"ID":"ca5bc022-0ecd-43d5-9a26-0c89457efa1f","Type":"ContainerStarted","Data":"c0131842d2ee4c5698e480624525555fa792c63ae1bc9e562b67f25463e2ccb9"} Jul 21 02:03:33.927619 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:03:33.927580 2580 generic.go:356] "Generic (PLEG): container finished" podID="ca5bc022-0ecd-43d5-9a26-0c89457efa1f" containerID="a276c1332a69d0d66d989352ab0c312079572226128fcc60130be76d8cc51394" exitCode=0 Jul 21 02:03:33.927995 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:03:33.927661 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-604509c9-lkx2j" event={"ID":"ca5bc022-0ecd-43d5-9a26-0c89457efa1f","Type":"ContainerDied","Data":"a276c1332a69d0d66d989352ab0c312079572226128fcc60130be76d8cc51394"} Jul 21 02:03:35.077825 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:03:35.077797 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-604509c9-lkx2j" Jul 21 02:03:35.217312 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:03:35.217235 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca5bc022-0ecd-43d5-9a26-0c89457efa1f-maas-api-service-ca\") pod \"ca5bc022-0ecd-43d5-9a26-0c89457efa1f\" (UID: \"ca5bc022-0ecd-43d5-9a26-0c89457efa1f\") " Jul 21 02:03:35.217592 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:03:35.217572 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca5bc022-0ecd-43d5-9a26-0c89457efa1f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ca5bc022-0ecd-43d5-9a26-0c89457efa1f" (UID: "ca5bc022-0ecd-43d5-9a26-0c89457efa1f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 02:03:35.318948 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:03:35.318916 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca5bc022-0ecd-43d5-9a26-0c89457efa1f-maas-api-service-ca\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 21 02:03:35.938967 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:03:35.938934 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-604509c9-lkx2j" Jul 21 02:03:35.939144 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:03:35.938960 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-604509c9-lkx2j" event={"ID":"ca5bc022-0ecd-43d5-9a26-0c89457efa1f","Type":"ContainerDied","Data":"c0131842d2ee4c5698e480624525555fa792c63ae1bc9e562b67f25463e2ccb9"} Jul 21 02:03:35.939144 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:03:35.939013 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c0131842d2ee4c5698e480624525555fa792c63ae1bc9e562b67f25463e2ccb9" Jul 21 02:03:43.292736 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:03:43.292702 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 02:03:44.686830 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:03:44.686789 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-604509c9-lkx2j"] Jul 21 02:03:44.691090 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:03:44.691061 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-604509c9-lkx2j"] Jul 21 02:03:44.877114 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:03:44.877080 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca5bc022-0ecd-43d5-9a26-0c89457efa1f" path="/var/lib/kubelet/pods/ca5bc022-0ecd-43d5-9a26-0c89457efa1f/volumes" Jul 21 02:03:48.432681 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:03:48.432638 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-63c3423d-776588c476-pf8f5"] Jul 21 02:03:48.433316 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:03:48.433296 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca5bc022-0ecd-43d5-9a26-0c89457efa1f" containerName="revoke-keys" Jul 21 02:03:48.433404 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:03:48.433319 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca5bc022-0ecd-43d5-9a26-0c89457efa1f" containerName="revoke-keys" Jul 21 02:03:48.433483 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:03:48.433435 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca5bc022-0ecd-43d5-9a26-0c89457efa1f" containerName="revoke-keys" Jul 21 02:03:48.436712 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:03:48.436687 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-63c3423d-776588c476-pf8f5" Jul 21 02:03:48.439203 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:03:48.439180 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 21 02:03:48.439414 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:03:48.439398 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-63c3423d-serving-cert\"" Jul 21 02:03:48.439657 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:03:48.439641 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-z67q6\"" Jul 21 02:03:48.440029 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:03:48.439983 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 21 02:03:48.449142 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:03:48.449122 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-63c3423d-776588c476-pf8f5"] Jul 21 02:03:48.542079 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:03:48.542044 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7b86ce0-537e-4f87-bdf1-678fb786799e-openshift-service-ca\") pod \"maas-api-e2e-migrate-63c3423d-776588c476-pf8f5\" (UID: \"a7b86ce0-537e-4f87-bdf1-678fb786799e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-63c3423d-776588c476-pf8f5" Jul 21 02:03:48.542599 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:03:48.542118 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7b86ce0-537e-4f87-bdf1-678fb786799e-maas-api-tls\") pod \"maas-api-e2e-migrate-63c3423d-776588c476-pf8f5\" (UID: \"a7b86ce0-537e-4f87-bdf1-678fb786799e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-63c3423d-776588c476-pf8f5" Jul 21 02:03:48.542599 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:03:48.542141 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25q56\" (UniqueName: \"kubernetes.io/projected/a7b86ce0-537e-4f87-bdf1-678fb786799e-kube-api-access-25q56\") pod \"maas-api-e2e-migrate-63c3423d-776588c476-pf8f5\" (UID: \"a7b86ce0-537e-4f87-bdf1-678fb786799e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-63c3423d-776588c476-pf8f5" Jul 21 02:03:48.642724 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:03:48.642681 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7b86ce0-537e-4f87-bdf1-678fb786799e-maas-api-tls\") pod \"maas-api-e2e-migrate-63c3423d-776588c476-pf8f5\" (UID: \"a7b86ce0-537e-4f87-bdf1-678fb786799e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-63c3423d-776588c476-pf8f5" Jul 21 02:03:48.642724 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:03:48.642737 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25q56\" (UniqueName: \"kubernetes.io/projected/a7b86ce0-537e-4f87-bdf1-678fb786799e-kube-api-access-25q56\") pod \"maas-api-e2e-migrate-63c3423d-776588c476-pf8f5\" (UID: \"a7b86ce0-537e-4f87-bdf1-678fb786799e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-63c3423d-776588c476-pf8f5" Jul 21 02:03:48.642995 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:03:48.642828 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7b86ce0-537e-4f87-bdf1-678fb786799e-openshift-service-ca\") pod \"maas-api-e2e-migrate-63c3423d-776588c476-pf8f5\" (UID: \"a7b86ce0-537e-4f87-bdf1-678fb786799e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-63c3423d-776588c476-pf8f5" Jul 21 02:03:48.642995 ip-10-0-139-183 kubenswrapper[2580]: E0721 02:03:48.642849 2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-63c3423d-serving-cert: secret "maas-api-e2e-migrate-63c3423d-serving-cert" not found Jul 21 02:03:48.642995 ip-10-0-139-183 kubenswrapper[2580]: E0721 02:03:48.642914 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7b86ce0-537e-4f87-bdf1-678fb786799e-maas-api-tls podName:a7b86ce0-537e-4f87-bdf1-678fb786799e nodeName:}" failed. No retries permitted until 2026-07-21 02:03:49.142893712 +0000 UTC m=+1652.730059667 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a7b86ce0-537e-4f87-bdf1-678fb786799e-maas-api-tls") pod "maas-api-e2e-migrate-63c3423d-776588c476-pf8f5" (UID: "a7b86ce0-537e-4f87-bdf1-678fb786799e") : secret "maas-api-e2e-migrate-63c3423d-serving-cert" not found Jul 21 02:03:48.643550 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:03:48.643521 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7b86ce0-537e-4f87-bdf1-678fb786799e-openshift-service-ca\") pod \"maas-api-e2e-migrate-63c3423d-776588c476-pf8f5\" (UID: \"a7b86ce0-537e-4f87-bdf1-678fb786799e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-63c3423d-776588c476-pf8f5" Jul 21 02:03:48.650686 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:03:48.650644 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25q56\" (UniqueName: \"kubernetes.io/projected/a7b86ce0-537e-4f87-bdf1-678fb786799e-kube-api-access-25q56\") pod \"maas-api-e2e-migrate-63c3423d-776588c476-pf8f5\" (UID: \"a7b86ce0-537e-4f87-bdf1-678fb786799e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-63c3423d-776588c476-pf8f5" Jul 21 02:03:49.146855 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:03:49.146822 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7b86ce0-537e-4f87-bdf1-678fb786799e-maas-api-tls\") pod \"maas-api-e2e-migrate-63c3423d-776588c476-pf8f5\" (UID: \"a7b86ce0-537e-4f87-bdf1-678fb786799e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-63c3423d-776588c476-pf8f5" Jul 21 02:03:49.149173 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:03:49.149146 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7b86ce0-537e-4f87-bdf1-678fb786799e-maas-api-tls\") pod \"maas-api-e2e-migrate-63c3423d-776588c476-pf8f5\" (UID: \"a7b86ce0-537e-4f87-bdf1-678fb786799e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-63c3423d-776588c476-pf8f5" Jul 21 02:03:49.350190 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:03:49.350155 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-63c3423d-776588c476-pf8f5" Jul 21 02:03:49.496951 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:03:49.496921 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-63c3423d-776588c476-pf8f5"] Jul 21 02:03:49.498866 ip-10-0-139-183 kubenswrapper[2580]: W0721 02:03:49.498839 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7b86ce0_537e_4f87_bdf1_678fb786799e.slice/crio-bc9f8ea454529641d26c4af1706987ead5338300c66ea77358054fc9017e80a8 WatchSource:0}: Error finding container bc9f8ea454529641d26c4af1706987ead5338300c66ea77358054fc9017e80a8: Status 404 returned error can't find the container with id bc9f8ea454529641d26c4af1706987ead5338300c66ea77358054fc9017e80a8 Jul 21 02:03:50.017240 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:03:50.017209 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-63c3423d-776588c476-pf8f5" event={"ID":"a7b86ce0-537e-4f87-bdf1-678fb786799e","Type":"ContainerStarted","Data":"bc9f8ea454529641d26c4af1706987ead5338300c66ea77358054fc9017e80a8"} Jul 21 02:03:51.024188 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:03:51.024149 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-63c3423d-776588c476-pf8f5" event={"ID":"a7b86ce0-537e-4f87-bdf1-678fb786799e","Type":"ContainerStarted","Data":"cea11fc11e90310f5aa87741a46c89ebb2f29a40264a136bd9074b3e023a618c"} Jul 21 02:03:51.024625 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:03:51.024260 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-63c3423d-776588c476-pf8f5" Jul 21 02:03:51.043395 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:03:51.043344 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-63c3423d-776588c476-pf8f5" podStartSLOduration=1.645143504 podStartE2EDuration="3.043325769s" totalImagesPullingTime="1.398182265s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 02:03:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 02:03:51.0408212 +0000 UTC m=+1654.627987177" watchObservedRunningTime="2026-07-21 02:03:51.043325769 +0000 UTC m=+1654.630491766" Jul 21 02:03:54.696512 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:03:54.696476 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 02:03:57.034372 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:03:57.034346 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-63c3423d-776588c476-pf8f5" Jul 21 02:04:23.810062 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:04:23.810017 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-63c3423d-776588c476-pf8f5"] Jul 21 02:04:23.810616 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:04:23.810380 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-63c3423d-776588c476-pf8f5" podUID="a7b86ce0-537e-4f87-bdf1-678fb786799e" containerName="maas-api" containerID="cri-o://cea11fc11e90310f5aa87741a46c89ebb2f29a40264a136bd9074b3e023a618c" gracePeriod=30 Jul 21 02:04:24.067841 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:04:24.067774 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-63c3423d-776588c476-pf8f5" Jul 21 02:04:24.193769 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:04:24.193731 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7b86ce0-537e-4f87-bdf1-678fb786799e-maas-api-tls\") pod \"a7b86ce0-537e-4f87-bdf1-678fb786799e\" (UID: \"a7b86ce0-537e-4f87-bdf1-678fb786799e\") " Jul 21 02:04:24.193769 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:04:24.193776 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-25q56\" (UniqueName: \"kubernetes.io/projected/a7b86ce0-537e-4f87-bdf1-678fb786799e-kube-api-access-25q56\") pod \"a7b86ce0-537e-4f87-bdf1-678fb786799e\" (UID: \"a7b86ce0-537e-4f87-bdf1-678fb786799e\") " Jul 21 02:04:24.194044 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:04:24.193812 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7b86ce0-537e-4f87-bdf1-678fb786799e-openshift-service-ca\") pod \"a7b86ce0-537e-4f87-bdf1-678fb786799e\" (UID: \"a7b86ce0-537e-4f87-bdf1-678fb786799e\") " Jul 21 02:04:24.194248 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:04:24.194221 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7b86ce0-537e-4f87-bdf1-678fb786799e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a7b86ce0-537e-4f87-bdf1-678fb786799e" (UID: "a7b86ce0-537e-4f87-bdf1-678fb786799e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 02:04:24.196142 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:04:24.196100 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7b86ce0-537e-4f87-bdf1-678fb786799e-kube-api-access-25q56" (OuterVolumeSpecName: "kube-api-access-25q56") pod "a7b86ce0-537e-4f87-bdf1-678fb786799e" (UID: "a7b86ce0-537e-4f87-bdf1-678fb786799e"). InnerVolumeSpecName "kube-api-access-25q56". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 02:04:24.196212 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:04:24.196170 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7b86ce0-537e-4f87-bdf1-678fb786799e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a7b86ce0-537e-4f87-bdf1-678fb786799e" (UID: "a7b86ce0-537e-4f87-bdf1-678fb786799e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 02:04:24.209813 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:04:24.209774 2580 generic.go:356] "Generic (PLEG): container finished" podID="a7b86ce0-537e-4f87-bdf1-678fb786799e" containerID="cea11fc11e90310f5aa87741a46c89ebb2f29a40264a136bd9074b3e023a618c" exitCode=0 Jul 21 02:04:24.209978 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:04:24.209816 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-63c3423d-776588c476-pf8f5" Jul 21 02:04:24.209978 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:04:24.209857 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-63c3423d-776588c476-pf8f5" event={"ID":"a7b86ce0-537e-4f87-bdf1-678fb786799e","Type":"ContainerDied","Data":"cea11fc11e90310f5aa87741a46c89ebb2f29a40264a136bd9074b3e023a618c"} Jul 21 02:04:24.209978 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:04:24.209915 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-63c3423d-776588c476-pf8f5" event={"ID":"a7b86ce0-537e-4f87-bdf1-678fb786799e","Type":"ContainerDied","Data":"bc9f8ea454529641d26c4af1706987ead5338300c66ea77358054fc9017e80a8"} Jul 21 02:04:24.209978 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:04:24.209949 2580 scope.go:117] "RemoveContainer" containerID="cea11fc11e90310f5aa87741a46c89ebb2f29a40264a136bd9074b3e023a618c" Jul 21 02:04:24.220152 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:04:24.220134 2580 scope.go:117] "RemoveContainer" containerID="cea11fc11e90310f5aa87741a46c89ebb2f29a40264a136bd9074b3e023a618c" Jul 21 02:04:24.220431 ip-10-0-139-183 kubenswrapper[2580]: E0721 02:04:24.220409 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cea11fc11e90310f5aa87741a46c89ebb2f29a40264a136bd9074b3e023a618c\": container with ID starting with cea11fc11e90310f5aa87741a46c89ebb2f29a40264a136bd9074b3e023a618c not found: ID does not exist" containerID="cea11fc11e90310f5aa87741a46c89ebb2f29a40264a136bd9074b3e023a618c" Jul 21 02:04:24.220521 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:04:24.220441 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cea11fc11e90310f5aa87741a46c89ebb2f29a40264a136bd9074b3e023a618c"} err="failed to get container status \"cea11fc11e90310f5aa87741a46c89ebb2f29a40264a136bd9074b3e023a618c\": rpc error: code = NotFound desc = could not find container \"cea11fc11e90310f5aa87741a46c89ebb2f29a40264a136bd9074b3e023a618c\": container with ID starting with cea11fc11e90310f5aa87741a46c89ebb2f29a40264a136bd9074b3e023a618c not found: ID does not exist" Jul 21 02:04:24.231957 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:04:24.231931 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-63c3423d-776588c476-pf8f5"] Jul 21 02:04:24.235842 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:04:24.235811 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-63c3423d-776588c476-pf8f5"] Jul 21 02:04:24.295195 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:04:24.295160 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7b86ce0-537e-4f87-bdf1-678fb786799e-maas-api-tls\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 21 02:04:24.295195 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:04:24.295188 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25q56\" (UniqueName: \"kubernetes.io/projected/a7b86ce0-537e-4f87-bdf1-678fb786799e-kube-api-access-25q56\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 21 02:04:24.295195 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:04:24.295198 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7b86ce0-537e-4f87-bdf1-678fb786799e-openshift-service-ca\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 21 02:04:24.878008 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:04:24.877975 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7b86ce0-537e-4f87-bdf1-678fb786799e" path="/var/lib/kubelet/pods/a7b86ce0-537e-4f87-bdf1-678fb786799e/volumes" Jul 21 02:04:26.600231 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:04:26.600200 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 02:04:38.294848 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:04:38.294812 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 02:05:11.209499 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:05:11.209444 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 02:05:22.094438 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:05:22.094400 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 02:05:54.990809 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:05:54.990774 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 02:06:16.944056 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:06:16.944019 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-9ctn2_1e930d67-d711-419c-af02-86b43027a95d/cluster-monitoring-operator/0.log" Jul 21 02:06:16.949254 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:06:16.949226 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-9ctn2_1e930d67-d711-419c-af02-86b43027a95d/cluster-monitoring-operator/0.log" Jul 21 02:06:16.949440 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:06:16.949404 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-4mtp7_9c380009-9bf5-4b73-803a-5a1ea9c1abf0/console-operator/1.log" Jul 21 02:06:16.954574 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:06:16.954548 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-4mtp7_9c380009-9bf5-4b73-803a-5a1ea9c1abf0/console-operator/1.log" Jul 21 02:07:27.866103 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:07:27.866062 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 02:07:37.519827 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:07:37.519793 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 02:07:45.304436 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:07:45.304357 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1e560e69-69d489f879-qgztb"] Jul 21 02:07:45.304928 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:07:45.304905 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7b86ce0-537e-4f87-bdf1-678fb786799e" containerName="maas-api" Jul 21 02:07:45.304928 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:07:45.304926 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7b86ce0-537e-4f87-bdf1-678fb786799e" containerName="maas-api" Jul 21 02:07:45.305102 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:07:45.305041 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7b86ce0-537e-4f87-bdf1-678fb786799e" containerName="maas-api" Jul 21 02:07:45.308652 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:07:45.308634 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1e560e69-69d489f879-qgztb" Jul 21 02:07:45.311326 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:07:45.311282 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-1e560e69-serving-cert\"" Jul 21 02:07:45.311584 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:07:45.311548 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 21 02:07:45.312204 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:07:45.312188 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 21 02:07:45.312283 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:07:45.312207 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-z67q6\"" Jul 21 02:07:45.322777 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:07:45.322751 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1e560e69-69d489f879-qgztb"] Jul 21 02:07:45.370766 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:07:45.370735 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e46d54df-2c33-4dcd-a817-30c71bc008c9-maas-api-tls\") pod \"maas-api-e2e-mt-1e560e69-69d489f879-qgztb\" (UID: \"e46d54df-2c33-4dcd-a817-30c71bc008c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1e560e69-69d489f879-qgztb" Jul 21 02:07:45.370974 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:07:45.370774 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqnbk\" (UniqueName: \"kubernetes.io/projected/e46d54df-2c33-4dcd-a817-30c71bc008c9-kube-api-access-hqnbk\") pod \"maas-api-e2e-mt-1e560e69-69d489f879-qgztb\" (UID: \"e46d54df-2c33-4dcd-a817-30c71bc008c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1e560e69-69d489f879-qgztb" Jul 21 02:07:45.370974 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:07:45.370889 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e46d54df-2c33-4dcd-a817-30c71bc008c9-openshift-service-ca\") pod \"maas-api-e2e-mt-1e560e69-69d489f879-qgztb\" (UID: \"e46d54df-2c33-4dcd-a817-30c71bc008c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1e560e69-69d489f879-qgztb" Jul 21 02:07:45.471386 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:07:45.471346 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e46d54df-2c33-4dcd-a817-30c71bc008c9-maas-api-tls\") pod \"maas-api-e2e-mt-1e560e69-69d489f879-qgztb\" (UID: \"e46d54df-2c33-4dcd-a817-30c71bc008c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1e560e69-69d489f879-qgztb" Jul 21 02:07:45.471597 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:07:45.471393 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqnbk\" (UniqueName: \"kubernetes.io/projected/e46d54df-2c33-4dcd-a817-30c71bc008c9-kube-api-access-hqnbk\") pod \"maas-api-e2e-mt-1e560e69-69d489f879-qgztb\" (UID: \"e46d54df-2c33-4dcd-a817-30c71bc008c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1e560e69-69d489f879-qgztb" Jul 21 02:07:45.471597 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:07:45.471500 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e46d54df-2c33-4dcd-a817-30c71bc008c9-openshift-service-ca\") pod \"maas-api-e2e-mt-1e560e69-69d489f879-qgztb\" (UID: \"e46d54df-2c33-4dcd-a817-30c71bc008c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1e560e69-69d489f879-qgztb" Jul 21 02:07:45.472163 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:07:45.472142 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e46d54df-2c33-4dcd-a817-30c71bc008c9-openshift-service-ca\") pod \"maas-api-e2e-mt-1e560e69-69d489f879-qgztb\" (UID: \"e46d54df-2c33-4dcd-a817-30c71bc008c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1e560e69-69d489f879-qgztb" Jul 21 02:07:45.473984 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:07:45.473963 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e46d54df-2c33-4dcd-a817-30c71bc008c9-maas-api-tls\") pod \"maas-api-e2e-mt-1e560e69-69d489f879-qgztb\" (UID: \"e46d54df-2c33-4dcd-a817-30c71bc008c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1e560e69-69d489f879-qgztb" Jul 21 02:07:45.480394 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:07:45.480369 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqnbk\" (UniqueName: \"kubernetes.io/projected/e46d54df-2c33-4dcd-a817-30c71bc008c9-kube-api-access-hqnbk\") pod \"maas-api-e2e-mt-1e560e69-69d489f879-qgztb\" (UID: \"e46d54df-2c33-4dcd-a817-30c71bc008c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1e560e69-69d489f879-qgztb" Jul 21 02:07:45.620214 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:07:45.620124 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1e560e69-69d489f879-qgztb" Jul 21 02:07:45.782979 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:07:45.782894 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1e560e69-69d489f879-qgztb"] Jul 21 02:07:45.785549 ip-10-0-139-183 kubenswrapper[2580]: W0721 02:07:45.785514 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode46d54df_2c33_4dcd_a817_30c71bc008c9.slice/crio-55dfb15e964d4f3ad417c5d6982cb7623dee68453e32a2259ea18170ed051932 WatchSource:0}: Error finding container 55dfb15e964d4f3ad417c5d6982cb7623dee68453e32a2259ea18170ed051932: Status 404 returned error can't find the container with id 55dfb15e964d4f3ad417c5d6982cb7623dee68453e32a2259ea18170ed051932 Jul 21 02:07:46.321852 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:07:46.321817 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1e560e69-69d489f879-qgztb" event={"ID":"e46d54df-2c33-4dcd-a817-30c71bc008c9","Type":"ContainerStarted","Data":"76890c26891b5fa5e06ffe05301fd90cb6d87c269816f7c81fc75a4b3d585a2b"} Jul 21 02:07:46.322293 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:07:46.321866 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1e560e69-69d489f879-qgztb" event={"ID":"e46d54df-2c33-4dcd-a817-30c71bc008c9","Type":"ContainerStarted","Data":"55dfb15e964d4f3ad417c5d6982cb7623dee68453e32a2259ea18170ed051932"} Jul 21 02:07:46.322293 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:07:46.321897 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1e560e69-69d489f879-qgztb" Jul 21 02:07:46.347131 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:07:46.347079 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1e560e69-69d489f879-qgztb" podStartSLOduration=0.914460261 podStartE2EDuration="1.347060966s" totalImagesPullingTime="432.600705ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 02:07:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 02:07:46.34411616 +0000 UTC m=+1889.931282138" watchObservedRunningTime="2026-07-21 02:07:46.347060966 +0000 UTC m=+1889.934226943" Jul 21 02:07:50.627977 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:07:50.627937 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 02:07:52.333900 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:07:52.333869 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1e560e69-69d489f879-qgztb" Jul 21 02:08:08.255424 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:08:08.255387 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1e560e69-69d489f879-qgztb"] Jul 21 02:08:08.255996 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:08:08.255665 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1e560e69-69d489f879-qgztb" podUID="e46d54df-2c33-4dcd-a817-30c71bc008c9" containerName="maas-api" containerID="cri-o://76890c26891b5fa5e06ffe05301fd90cb6d87c269816f7c81fc75a4b3d585a2b" gracePeriod=30 Jul 21 02:08:08.454296 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:08:08.454264 2580 generic.go:356] "Generic (PLEG): container finished" podID="e46d54df-2c33-4dcd-a817-30c71bc008c9" containerID="76890c26891b5fa5e06ffe05301fd90cb6d87c269816f7c81fc75a4b3d585a2b" exitCode=0 Jul 21 02:08:08.454492 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:08:08.454352 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1e560e69-69d489f879-qgztb" event={"ID":"e46d54df-2c33-4dcd-a817-30c71bc008c9","Type":"ContainerDied","Data":"76890c26891b5fa5e06ffe05301fd90cb6d87c269816f7c81fc75a4b3d585a2b"} Jul 21 02:08:08.511579 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:08:08.511522 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1e560e69-69d489f879-qgztb" Jul 21 02:08:08.602721 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:08:08.602682 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e46d54df-2c33-4dcd-a817-30c71bc008c9-openshift-service-ca\") pod \"e46d54df-2c33-4dcd-a817-30c71bc008c9\" (UID: \"e46d54df-2c33-4dcd-a817-30c71bc008c9\") " Jul 21 02:08:08.602918 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:08:08.602803 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hqnbk\" (UniqueName: \"kubernetes.io/projected/e46d54df-2c33-4dcd-a817-30c71bc008c9-kube-api-access-hqnbk\") pod \"e46d54df-2c33-4dcd-a817-30c71bc008c9\" (UID: \"e46d54df-2c33-4dcd-a817-30c71bc008c9\") " Jul 21 02:08:08.602918 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:08:08.602844 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e46d54df-2c33-4dcd-a817-30c71bc008c9-maas-api-tls\") pod \"e46d54df-2c33-4dcd-a817-30c71bc008c9\" (UID: \"e46d54df-2c33-4dcd-a817-30c71bc008c9\") " Jul 21 02:08:08.603125 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:08:08.603098 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e46d54df-2c33-4dcd-a817-30c71bc008c9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e46d54df-2c33-4dcd-a817-30c71bc008c9" (UID: "e46d54df-2c33-4dcd-a817-30c71bc008c9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 02:08:08.603222 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:08:08.603204 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e46d54df-2c33-4dcd-a817-30c71bc008c9-openshift-service-ca\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 21 02:08:08.604936 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:08:08.604903 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e46d54df-2c33-4dcd-a817-30c71bc008c9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e46d54df-2c33-4dcd-a817-30c71bc008c9" (UID: "e46d54df-2c33-4dcd-a817-30c71bc008c9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 02:08:08.605036 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:08:08.604977 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e46d54df-2c33-4dcd-a817-30c71bc008c9-kube-api-access-hqnbk" (OuterVolumeSpecName: "kube-api-access-hqnbk") pod "e46d54df-2c33-4dcd-a817-30c71bc008c9" (UID: "e46d54df-2c33-4dcd-a817-30c71bc008c9"). InnerVolumeSpecName "kube-api-access-hqnbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 02:08:08.704011 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:08:08.703976 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqnbk\" (UniqueName: \"kubernetes.io/projected/e46d54df-2c33-4dcd-a817-30c71bc008c9-kube-api-access-hqnbk\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 21 02:08:08.704011 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:08:08.704005 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e46d54df-2c33-4dcd-a817-30c71bc008c9-maas-api-tls\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 21 02:08:09.460993 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:08:09.460960 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1e560e69-69d489f879-qgztb" Jul 21 02:08:09.461416 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:08:09.461004 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1e560e69-69d489f879-qgztb" event={"ID":"e46d54df-2c33-4dcd-a817-30c71bc008c9","Type":"ContainerDied","Data":"55dfb15e964d4f3ad417c5d6982cb7623dee68453e32a2259ea18170ed051932"} Jul 21 02:08:09.461416 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:08:09.461066 2580 scope.go:117] "RemoveContainer" containerID="76890c26891b5fa5e06ffe05301fd90cb6d87c269816f7c81fc75a4b3d585a2b" Jul 21 02:08:09.488182 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:08:09.488153 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1e560e69-69d489f879-qgztb"] Jul 21 02:08:09.495115 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:08:09.495091 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1e560e69-69d489f879-qgztb"] Jul 21 02:08:10.878686 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:08:10.878656 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e46d54df-2c33-4dcd-a817-30c71bc008c9" path="/var/lib/kubelet/pods/e46d54df-2c33-4dcd-a817-30c71bc008c9/volumes" Jul 21 02:08:11.343686 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:08:11.343648 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 02:08:44.323732 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:08:44.323697 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f3d1a053-7586c986bf-nfnlc"] Jul 21 02:08:44.324261 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:08:44.324245 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e46d54df-2c33-4dcd-a817-30c71bc008c9" containerName="maas-api" Jul 21 02:08:44.324307 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:08:44.324263 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e46d54df-2c33-4dcd-a817-30c71bc008c9" containerName="maas-api" Jul 21 02:08:44.324378 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:08:44.324367 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e46d54df-2c33-4dcd-a817-30c71bc008c9" containerName="maas-api" Jul 21 02:08:44.327887 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:08:44.327866 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3d1a053-7586c986bf-nfnlc" Jul 21 02:08:44.330641 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:08:44.330618 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 21 02:08:44.330761 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:08:44.330669 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 21 02:08:44.331337 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:08:44.331316 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-f3d1a053-serving-cert\"" Jul 21 02:08:44.331470 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:08:44.331384 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-z67q6\"" Jul 21 02:08:44.345959 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:08:44.345921 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f3d1a053-7586c986bf-nfnlc"] Jul 21 02:08:44.446059 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:08:44.446020 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c944c04-6f20-4db5-a3e1-58de5045fad4-openshift-service-ca\") pod \"maas-api-e2e-mt-f3d1a053-7586c986bf-nfnlc\" (UID: \"6c944c04-6f20-4db5-a3e1-58de5045fad4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3d1a053-7586c986bf-nfnlc" Jul 21 02:08:44.446251 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:08:44.446136 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c944c04-6f20-4db5-a3e1-58de5045fad4-maas-api-tls\") pod \"maas-api-e2e-mt-f3d1a053-7586c986bf-nfnlc\" (UID: \"6c944c04-6f20-4db5-a3e1-58de5045fad4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3d1a053-7586c986bf-nfnlc" Jul 21 02:08:44.446251 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:08:44.446193 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wccs6\" (UniqueName: \"kubernetes.io/projected/6c944c04-6f20-4db5-a3e1-58de5045fad4-kube-api-access-wccs6\") pod \"maas-api-e2e-mt-f3d1a053-7586c986bf-nfnlc\" (UID: \"6c944c04-6f20-4db5-a3e1-58de5045fad4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3d1a053-7586c986bf-nfnlc" Jul 21 02:08:44.547780 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:08:44.547740 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c944c04-6f20-4db5-a3e1-58de5045fad4-maas-api-tls\") pod \"maas-api-e2e-mt-f3d1a053-7586c986bf-nfnlc\" (UID: \"6c944c04-6f20-4db5-a3e1-58de5045fad4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3d1a053-7586c986bf-nfnlc" Jul 21 02:08:44.547944 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:08:44.547792 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wccs6\" (UniqueName: \"kubernetes.io/projected/6c944c04-6f20-4db5-a3e1-58de5045fad4-kube-api-access-wccs6\") pod \"maas-api-e2e-mt-f3d1a053-7586c986bf-nfnlc\" (UID: \"6c944c04-6f20-4db5-a3e1-58de5045fad4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3d1a053-7586c986bf-nfnlc" Jul 21 02:08:44.547944 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:08:44.547853 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c944c04-6f20-4db5-a3e1-58de5045fad4-openshift-service-ca\") pod \"maas-api-e2e-mt-f3d1a053-7586c986bf-nfnlc\" (UID: \"6c944c04-6f20-4db5-a3e1-58de5045fad4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3d1a053-7586c986bf-nfnlc" Jul 21 02:08:44.547944 ip-10-0-139-183 kubenswrapper[2580]: E0721 02:08:44.547894 2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-f3d1a053-serving-cert: secret "maas-api-e2e-mt-f3d1a053-serving-cert" not found Jul 21 02:08:44.548060 ip-10-0-139-183 kubenswrapper[2580]: E0721 02:08:44.547978 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c944c04-6f20-4db5-a3e1-58de5045fad4-maas-api-tls podName:6c944c04-6f20-4db5-a3e1-58de5045fad4 nodeName:}" failed. No retries permitted until 2026-07-21 02:08:45.047958573 +0000 UTC m=+1948.635124551 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6c944c04-6f20-4db5-a3e1-58de5045fad4-maas-api-tls") pod "maas-api-e2e-mt-f3d1a053-7586c986bf-nfnlc" (UID: "6c944c04-6f20-4db5-a3e1-58de5045fad4") : secret "maas-api-e2e-mt-f3d1a053-serving-cert" not found Jul 21 02:08:44.548568 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:08:44.548551 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c944c04-6f20-4db5-a3e1-58de5045fad4-openshift-service-ca\") pod \"maas-api-e2e-mt-f3d1a053-7586c986bf-nfnlc\" (UID: \"6c944c04-6f20-4db5-a3e1-58de5045fad4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3d1a053-7586c986bf-nfnlc" Jul 21 02:08:44.558114 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:08:44.558086 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wccs6\" (UniqueName: \"kubernetes.io/projected/6c944c04-6f20-4db5-a3e1-58de5045fad4-kube-api-access-wccs6\") pod \"maas-api-e2e-mt-f3d1a053-7586c986bf-nfnlc\" (UID: \"6c944c04-6f20-4db5-a3e1-58de5045fad4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3d1a053-7586c986bf-nfnlc" Jul 21 02:08:45.053259 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:08:45.053224 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c944c04-6f20-4db5-a3e1-58de5045fad4-maas-api-tls\") pod \"maas-api-e2e-mt-f3d1a053-7586c986bf-nfnlc\" (UID: \"6c944c04-6f20-4db5-a3e1-58de5045fad4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3d1a053-7586c986bf-nfnlc" Jul 21 02:08:45.055650 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:08:45.055626 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c944c04-6f20-4db5-a3e1-58de5045fad4-maas-api-tls\") pod \"maas-api-e2e-mt-f3d1a053-7586c986bf-nfnlc\" (UID: \"6c944c04-6f20-4db5-a3e1-58de5045fad4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3d1a053-7586c986bf-nfnlc" Jul 21 02:08:45.239477 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:08:45.239421 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3d1a053-7586c986bf-nfnlc" Jul 21 02:08:45.594742 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:08:45.594700 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f3d1a053-7586c986bf-nfnlc"] Jul 21 02:08:45.597080 ip-10-0-139-183 kubenswrapper[2580]: W0721 02:08:45.597048 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c944c04_6f20_4db5_a3e1_58de5045fad4.slice/crio-29a926df2fb102e09806472f5a6627bee2c44a0bdab36f48dec85f5d7882fa94 WatchSource:0}: Error finding container 29a926df2fb102e09806472f5a6627bee2c44a0bdab36f48dec85f5d7882fa94: Status 404 returned error can't find the container with id 29a926df2fb102e09806472f5a6627bee2c44a0bdab36f48dec85f5d7882fa94 Jul 21 02:08:45.598492 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:08:45.598476 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 02:08:45.679082 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:08:45.679045 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3d1a053-7586c986bf-nfnlc" event={"ID":"6c944c04-6f20-4db5-a3e1-58de5045fad4","Type":"ContainerStarted","Data":"29a926df2fb102e09806472f5a6627bee2c44a0bdab36f48dec85f5d7882fa94"} Jul 21 02:08:46.687115 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:08:46.687075 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3d1a053-7586c986bf-nfnlc" event={"ID":"6c944c04-6f20-4db5-a3e1-58de5045fad4","Type":"ContainerStarted","Data":"5d35d3c74373200b88e9e3d51eca388179bdaf1a54f6ccdfb9b61a02a5e1f1b1"} Jul 21 02:08:46.687573 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:08:46.687208 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3d1a053-7586c986bf-nfnlc" Jul 21 02:08:51.498128 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:08:51.498068 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3d1a053-7586c986bf-nfnlc" podStartSLOduration=7.103838724 podStartE2EDuration="7.498052191s" totalImagesPullingTime="394.213467ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 02:08:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 02:08:46.729894327 +0000 UTC m=+1950.317060306" watchObservedRunningTime="2026-07-21 02:08:51.498052191 +0000 UTC m=+1955.085218164" Jul 21 02:08:51.499604 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:08:51.499584 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 02:08:52.701299 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:08:52.701271 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3d1a053-7586c986bf-nfnlc" Jul 21 02:09:00.995683 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:09:00.995647 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 02:09:16.001584 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:09:16.001500 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 02:09:36.300054 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:09:36.300015 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 02:09:37.793176 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:09:37.793129 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f3d1a053-7586c986bf-nfnlc"] Jul 21 02:09:37.793816 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:09:37.793390 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3d1a053-7586c986bf-nfnlc" podUID="6c944c04-6f20-4db5-a3e1-58de5045fad4" containerName="maas-api" containerID="cri-o://5d35d3c74373200b88e9e3d51eca388179bdaf1a54f6ccdfb9b61a02a5e1f1b1" gracePeriod=30 Jul 21 02:09:37.976105 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:09:37.975712 2580 generic.go:356] "Generic (PLEG): container finished" podID="6c944c04-6f20-4db5-a3e1-58de5045fad4" containerID="5d35d3c74373200b88e9e3d51eca388179bdaf1a54f6ccdfb9b61a02a5e1f1b1" exitCode=0 Jul 21 02:09:37.976105 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:09:37.975778 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3d1a053-7586c986bf-nfnlc" event={"ID":"6c944c04-6f20-4db5-a3e1-58de5045fad4","Type":"ContainerDied","Data":"5d35d3c74373200b88e9e3d51eca388179bdaf1a54f6ccdfb9b61a02a5e1f1b1"} Jul 21 02:09:38.056872 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:09:38.056807 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3d1a053-7586c986bf-nfnlc" Jul 21 02:09:38.191106 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:09:38.191065 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wccs6\" (UniqueName: \"kubernetes.io/projected/6c944c04-6f20-4db5-a3e1-58de5045fad4-kube-api-access-wccs6\") pod \"6c944c04-6f20-4db5-a3e1-58de5045fad4\" (UID: \"6c944c04-6f20-4db5-a3e1-58de5045fad4\") " Jul 21 02:09:38.191305 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:09:38.191127 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c944c04-6f20-4db5-a3e1-58de5045fad4-openshift-service-ca\") pod \"6c944c04-6f20-4db5-a3e1-58de5045fad4\" (UID: \"6c944c04-6f20-4db5-a3e1-58de5045fad4\") " Jul 21 02:09:38.191305 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:09:38.191248 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c944c04-6f20-4db5-a3e1-58de5045fad4-maas-api-tls\") pod \"6c944c04-6f20-4db5-a3e1-58de5045fad4\" (UID: \"6c944c04-6f20-4db5-a3e1-58de5045fad4\") " Jul 21 02:09:38.191563 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:09:38.191541 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c944c04-6f20-4db5-a3e1-58de5045fad4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6c944c04-6f20-4db5-a3e1-58de5045fad4" (UID: "6c944c04-6f20-4db5-a3e1-58de5045fad4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 02:09:38.193217 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:09:38.193190 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c944c04-6f20-4db5-a3e1-58de5045fad4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6c944c04-6f20-4db5-a3e1-58de5045fad4" (UID: "6c944c04-6f20-4db5-a3e1-58de5045fad4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 02:09:38.193326 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:09:38.193272 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c944c04-6f20-4db5-a3e1-58de5045fad4-kube-api-access-wccs6" (OuterVolumeSpecName: "kube-api-access-wccs6") pod "6c944c04-6f20-4db5-a3e1-58de5045fad4" (UID: "6c944c04-6f20-4db5-a3e1-58de5045fad4"). InnerVolumeSpecName "kube-api-access-wccs6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 02:09:38.292837 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:09:38.292801 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wccs6\" (UniqueName: \"kubernetes.io/projected/6c944c04-6f20-4db5-a3e1-58de5045fad4-kube-api-access-wccs6\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 21 02:09:38.292837 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:09:38.292831 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c944c04-6f20-4db5-a3e1-58de5045fad4-openshift-service-ca\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 21 02:09:38.292837 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:09:38.292843 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c944c04-6f20-4db5-a3e1-58de5045fad4-maas-api-tls\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 21 02:09:38.982632 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:09:38.982596 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3d1a053-7586c986bf-nfnlc" Jul 21 02:09:38.983062 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:09:38.982648 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3d1a053-7586c986bf-nfnlc" event={"ID":"6c944c04-6f20-4db5-a3e1-58de5045fad4","Type":"ContainerDied","Data":"29a926df2fb102e09806472f5a6627bee2c44a0bdab36f48dec85f5d7882fa94"} Jul 21 02:09:38.983062 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:09:38.982702 2580 scope.go:117] "RemoveContainer" containerID="5d35d3c74373200b88e9e3d51eca388179bdaf1a54f6ccdfb9b61a02a5e1f1b1" Jul 21 02:09:38.998930 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:09:38.998905 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f3d1a053-7586c986bf-nfnlc"] Jul 21 02:09:39.002680 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:09:39.002655 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f3d1a053-7586c986bf-nfnlc"] Jul 21 02:09:40.877671 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:09:40.877631 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c944c04-6f20-4db5-a3e1-58de5045fad4" path="/var/lib/kubelet/pods/6c944c04-6f20-4db5-a3e1-58de5045fad4/volumes" Jul 21 02:09:46.998834 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:09:46.998797 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 02:10:17.075208 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:10:17.075170 2580 scope.go:117] "RemoveContainer" containerID="a276c1332a69d0d66d989352ab0c312079572226128fcc60130be76d8cc51394" Jul 21 02:10:36.713274 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:10:36.713234 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 02:10:49.594568 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:10:49.594482 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 02:11:16.979595 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:11:16.979568 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-9ctn2_1e930d67-d711-419c-af02-86b43027a95d/cluster-monitoring-operator/0.log" Jul 21 02:11:16.984471 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:11:16.984434 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-4mtp7_9c380009-9bf5-4b73-803a-5a1ea9c1abf0/console-operator/1.log" Jul 21 02:11:16.986621 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:11:16.986603 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-9ctn2_1e930d67-d711-419c-af02-86b43027a95d/cluster-monitoring-operator/0.log" Jul 21 02:11:16.991783 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:11:16.991762 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-4mtp7_9c380009-9bf5-4b73-803a-5a1ea9c1abf0/console-operator/1.log" Jul 21 02:11:32.120863 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:11:32.120826 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0eaad6-5cffd8f4f9-sqksn"] Jul 21 02:11:32.121283 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:11:32.121270 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c944c04-6f20-4db5-a3e1-58de5045fad4" containerName="maas-api" Jul 21 02:11:32.121283 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:11:32.121283 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c944c04-6f20-4db5-a3e1-58de5045fad4" containerName="maas-api" Jul 21 02:11:32.121367 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:11:32.121358 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c944c04-6f20-4db5-a3e1-58de5045fad4" containerName="maas-api" Jul 21 02:11:32.124736 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:11:32.124717 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0eaad6-5cffd8f4f9-sqksn" Jul 21 02:11:32.126975 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:11:32.126954 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 21 02:11:32.127116 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:11:32.126974 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-0eaad6-serving-cert\"" Jul 21 02:11:32.127116 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:11:32.127071 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 21 02:11:32.127782 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:11:32.127765 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-z67q6\"" Jul 21 02:11:32.135627 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:11:32.135599 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0eaad6-5cffd8f4f9-sqksn"] Jul 21 02:11:32.197365 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:11:32.197324 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7n6c\" (UniqueName: \"kubernetes.io/projected/e64ebd3d-1d83-4693-ab73-5180cb5f009e-kube-api-access-z7n6c\") pod \"maas-api-e2e-inf-a-0eaad6-5cffd8f4f9-sqksn\" (UID: \"e64ebd3d-1d83-4693-ab73-5180cb5f009e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0eaad6-5cffd8f4f9-sqksn" Jul 21 02:11:32.197613 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:11:32.197487 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e64ebd3d-1d83-4693-ab73-5180cb5f009e-maas-api-tls\") pod \"maas-api-e2e-inf-a-0eaad6-5cffd8f4f9-sqksn\" (UID: \"e64ebd3d-1d83-4693-ab73-5180cb5f009e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0eaad6-5cffd8f4f9-sqksn" Jul 21 02:11:32.197613 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:11:32.197532 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e64ebd3d-1d83-4693-ab73-5180cb5f009e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-0eaad6-5cffd8f4f9-sqksn\" (UID: \"e64ebd3d-1d83-4693-ab73-5180cb5f009e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0eaad6-5cffd8f4f9-sqksn" Jul 21 02:11:32.298970 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:11:32.298937 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e64ebd3d-1d83-4693-ab73-5180cb5f009e-maas-api-tls\") pod \"maas-api-e2e-inf-a-0eaad6-5cffd8f4f9-sqksn\" (UID: \"e64ebd3d-1d83-4693-ab73-5180cb5f009e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0eaad6-5cffd8f4f9-sqksn" Jul 21 02:11:32.299178 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:11:32.298988 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e64ebd3d-1d83-4693-ab73-5180cb5f009e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-0eaad6-5cffd8f4f9-sqksn\" (UID: \"e64ebd3d-1d83-4693-ab73-5180cb5f009e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0eaad6-5cffd8f4f9-sqksn" Jul 21 02:11:32.299178 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:11:32.299060 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7n6c\" (UniqueName: \"kubernetes.io/projected/e64ebd3d-1d83-4693-ab73-5180cb5f009e-kube-api-access-z7n6c\") pod \"maas-api-e2e-inf-a-0eaad6-5cffd8f4f9-sqksn\" (UID: \"e64ebd3d-1d83-4693-ab73-5180cb5f009e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0eaad6-5cffd8f4f9-sqksn" Jul 21 02:11:32.299749 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:11:32.299725 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e64ebd3d-1d83-4693-ab73-5180cb5f009e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-0eaad6-5cffd8f4f9-sqksn\" (UID: \"e64ebd3d-1d83-4693-ab73-5180cb5f009e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0eaad6-5cffd8f4f9-sqksn" Jul 21 02:11:32.301389 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:11:32.301368 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e64ebd3d-1d83-4693-ab73-5180cb5f009e-maas-api-tls\") pod \"maas-api-e2e-inf-a-0eaad6-5cffd8f4f9-sqksn\" (UID: \"e64ebd3d-1d83-4693-ab73-5180cb5f009e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0eaad6-5cffd8f4f9-sqksn" Jul 21 02:11:32.307210 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:11:32.307187 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7n6c\" (UniqueName: \"kubernetes.io/projected/e64ebd3d-1d83-4693-ab73-5180cb5f009e-kube-api-access-z7n6c\") pod \"maas-api-e2e-inf-a-0eaad6-5cffd8f4f9-sqksn\" (UID: \"e64ebd3d-1d83-4693-ab73-5180cb5f009e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0eaad6-5cffd8f4f9-sqksn" Jul 21 02:11:32.436616 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:11:32.436512 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0eaad6-5cffd8f4f9-sqksn" Jul 21 02:11:32.590303 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:11:32.590199 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0eaad6-5cffd8f4f9-sqksn"] Jul 21 02:11:32.593571 ip-10-0-139-183 kubenswrapper[2580]: W0721 02:11:32.593541 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode64ebd3d_1d83_4693_ab73_5180cb5f009e.slice/crio-2d2abc9897653d17b0819fea08c29a8f33b2fe7eb3b75c1180ca86990feac43d WatchSource:0}: Error finding container 2d2abc9897653d17b0819fea08c29a8f33b2fe7eb3b75c1180ca86990feac43d: Status 404 returned error can't find the container with id 2d2abc9897653d17b0819fea08c29a8f33b2fe7eb3b75c1180ca86990feac43d Jul 21 02:11:32.605733 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:11:32.605698 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0eaad6-5cffd8f4f9-sqksn" event={"ID":"e64ebd3d-1d83-4693-ab73-5180cb5f009e","Type":"ContainerStarted","Data":"2d2abc9897653d17b0819fea08c29a8f33b2fe7eb3b75c1180ca86990feac43d"} Jul 21 02:11:33.612330 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:11:33.612296 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0eaad6-5cffd8f4f9-sqksn" event={"ID":"e64ebd3d-1d83-4693-ab73-5180cb5f009e","Type":"ContainerStarted","Data":"ba5d0fedfd34f802c9aa988fa257e692563234d0ddd0902377715793e7f1b276"} Jul 21 02:11:33.612755 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:11:33.612412 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0eaad6-5cffd8f4f9-sqksn" Jul 21 02:11:33.629972 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:11:33.629918 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0eaad6-5cffd8f4f9-sqksn" podStartSLOduration=1.185895278 podStartE2EDuration="1.629900499s" totalImagesPullingTime="444.005221ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 02:11:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 02:11:33.629289738 +0000 UTC m=+2117.216455717" watchObservedRunningTime="2026-07-21 02:11:33.629900499 +0000 UTC m=+2117.217066479" Jul 21 02:11:38.011230 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:11:38.011189 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 02:11:39.623513 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:11:39.623484 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0eaad6-5cffd8f4f9-sqksn" Jul 21 02:11:50.893688 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:11:50.893652 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 02:12:00.906653 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:12:00.906615 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 02:12:22.992714 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:12:22.992633 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 02:12:37.098622 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:12:37.098591 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 02:13:24.502166 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:13:24.502130 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 02:13:31.183403 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:13:31.183368 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0eaad6-5cffd8f4f9-sqksn"] Jul 21 02:13:31.183847 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:13:31.183632 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0eaad6-5cffd8f4f9-sqksn" podUID="e64ebd3d-1d83-4693-ab73-5180cb5f009e" containerName="maas-api" containerID="cri-o://ba5d0fedfd34f802c9aa988fa257e692563234d0ddd0902377715793e7f1b276" gracePeriod=30 Jul 21 02:13:31.451718 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:13:31.451658 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0eaad6-5cffd8f4f9-sqksn" Jul 21 02:13:31.565716 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:13:31.565679 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e64ebd3d-1d83-4693-ab73-5180cb5f009e-openshift-service-ca\") pod \"e64ebd3d-1d83-4693-ab73-5180cb5f009e\" (UID: \"e64ebd3d-1d83-4693-ab73-5180cb5f009e\") " Jul 21 02:13:31.565901 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:13:31.565727 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e64ebd3d-1d83-4693-ab73-5180cb5f009e-maas-api-tls\") pod \"e64ebd3d-1d83-4693-ab73-5180cb5f009e\" (UID: \"e64ebd3d-1d83-4693-ab73-5180cb5f009e\") " Jul 21 02:13:31.565901 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:13:31.565777 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z7n6c\" (UniqueName: \"kubernetes.io/projected/e64ebd3d-1d83-4693-ab73-5180cb5f009e-kube-api-access-z7n6c\") pod \"e64ebd3d-1d83-4693-ab73-5180cb5f009e\" (UID: \"e64ebd3d-1d83-4693-ab73-5180cb5f009e\") " Jul 21 02:13:31.566105 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:13:31.566081 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e64ebd3d-1d83-4693-ab73-5180cb5f009e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e64ebd3d-1d83-4693-ab73-5180cb5f009e" (UID: "e64ebd3d-1d83-4693-ab73-5180cb5f009e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 02:13:31.567874 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:13:31.567851 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e64ebd3d-1d83-4693-ab73-5180cb5f009e-kube-api-access-z7n6c" (OuterVolumeSpecName: "kube-api-access-z7n6c") pod "e64ebd3d-1d83-4693-ab73-5180cb5f009e" (UID: "e64ebd3d-1d83-4693-ab73-5180cb5f009e"). InnerVolumeSpecName "kube-api-access-z7n6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 02:13:31.567985 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:13:31.567855 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e64ebd3d-1d83-4693-ab73-5180cb5f009e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e64ebd3d-1d83-4693-ab73-5180cb5f009e" (UID: "e64ebd3d-1d83-4693-ab73-5180cb5f009e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 02:13:31.666543 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:13:31.666501 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e64ebd3d-1d83-4693-ab73-5180cb5f009e-openshift-service-ca\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 21 02:13:31.666543 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:13:31.666534 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e64ebd3d-1d83-4693-ab73-5180cb5f009e-maas-api-tls\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 21 02:13:31.666543 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:13:31.666544 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7n6c\" (UniqueName: \"kubernetes.io/projected/e64ebd3d-1d83-4693-ab73-5180cb5f009e-kube-api-access-z7n6c\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 21 02:13:32.262290 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:13:32.262254 2580 generic.go:356] "Generic (PLEG): container finished" podID="e64ebd3d-1d83-4693-ab73-5180cb5f009e" containerID="ba5d0fedfd34f802c9aa988fa257e692563234d0ddd0902377715793e7f1b276" exitCode=0 Jul 21 02:13:32.262709 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:13:32.262304 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0eaad6-5cffd8f4f9-sqksn" Jul 21 02:13:32.262709 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:13:32.262342 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0eaad6-5cffd8f4f9-sqksn" event={"ID":"e64ebd3d-1d83-4693-ab73-5180cb5f009e","Type":"ContainerDied","Data":"ba5d0fedfd34f802c9aa988fa257e692563234d0ddd0902377715793e7f1b276"} Jul 21 02:13:32.262709 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:13:32.262401 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0eaad6-5cffd8f4f9-sqksn" event={"ID":"e64ebd3d-1d83-4693-ab73-5180cb5f009e","Type":"ContainerDied","Data":"2d2abc9897653d17b0819fea08c29a8f33b2fe7eb3b75c1180ca86990feac43d"} Jul 21 02:13:32.262709 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:13:32.262444 2580 scope.go:117] "RemoveContainer" containerID="ba5d0fedfd34f802c9aa988fa257e692563234d0ddd0902377715793e7f1b276" Jul 21 02:13:32.272854 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:13:32.272836 2580 scope.go:117] "RemoveContainer" containerID="ba5d0fedfd34f802c9aa988fa257e692563234d0ddd0902377715793e7f1b276" Jul 21 02:13:32.273110 ip-10-0-139-183 kubenswrapper[2580]: E0721 02:13:32.273092 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba5d0fedfd34f802c9aa988fa257e692563234d0ddd0902377715793e7f1b276\": container with ID starting with ba5d0fedfd34f802c9aa988fa257e692563234d0ddd0902377715793e7f1b276 not found: ID does not exist" containerID="ba5d0fedfd34f802c9aa988fa257e692563234d0ddd0902377715793e7f1b276" Jul 21 02:13:32.273173 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:13:32.273122 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba5d0fedfd34f802c9aa988fa257e692563234d0ddd0902377715793e7f1b276"} err="failed to get container status \"ba5d0fedfd34f802c9aa988fa257e692563234d0ddd0902377715793e7f1b276\": rpc error: code = NotFound desc = could not find container \"ba5d0fedfd34f802c9aa988fa257e692563234d0ddd0902377715793e7f1b276\": container with ID starting with ba5d0fedfd34f802c9aa988fa257e692563234d0ddd0902377715793e7f1b276 not found: ID does not exist" Jul 21 02:13:32.284849 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:13:32.284826 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0eaad6-5cffd8f4f9-sqksn"] Jul 21 02:13:32.288415 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:13:32.288391 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0eaad6-5cffd8f4f9-sqksn"] Jul 21 02:13:32.877058 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:13:32.877028 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e64ebd3d-1d83-4693-ab73-5180cb5f009e" path="/var/lib/kubelet/pods/e64ebd3d-1d83-4693-ab73-5180cb5f009e/volumes" Jul 21 02:13:39.998541 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:13:39.998457 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 02:13:56.496519 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:13:56.496485 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 02:14:08.887257 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:14:08.887226 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-91b59b-6456677d69-s77r4"] Jul 21 02:14:08.887827 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:14:08.887804 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e64ebd3d-1d83-4693-ab73-5180cb5f009e" containerName="maas-api" Jul 21 02:14:08.887994 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:14:08.887976 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e64ebd3d-1d83-4693-ab73-5180cb5f009e" containerName="maas-api" Jul 21 02:14:08.888145 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:14:08.888130 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e64ebd3d-1d83-4693-ab73-5180cb5f009e" containerName="maas-api" Jul 21 02:14:08.891715 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:14:08.891696 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-91b59b-6456677d69-s77r4" Jul 21 02:14:08.894409 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:14:08.894385 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-91b59b-serving-cert\"" Jul 21 02:14:08.894566 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:14:08.894550 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-z67q6\"" Jul 21 02:14:08.895375 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:14:08.895350 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 21 02:14:08.895504 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:14:08.895355 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 21 02:14:08.901772 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:14:08.901745 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-91b59b-6456677d69-s77r4"] Jul 21 02:14:09.014161 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:14:09.014122 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8f8afc3-2ace-4e51-b46a-5788f64d46a7-maas-api-tls\") pod \"maas-api-e2e-api-a-91b59b-6456677d69-s77r4\" (UID: \"e8f8afc3-2ace-4e51-b46a-5788f64d46a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-91b59b-6456677d69-s77r4" Jul 21 02:14:09.014351 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:14:09.014183 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8f8afc3-2ace-4e51-b46a-5788f64d46a7-openshift-service-ca\") pod \"maas-api-e2e-api-a-91b59b-6456677d69-s77r4\" (UID: \"e8f8afc3-2ace-4e51-b46a-5788f64d46a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-91b59b-6456677d69-s77r4" Jul 21 02:14:09.014351 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:14:09.014299 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj8vs\" (UniqueName: \"kubernetes.io/projected/e8f8afc3-2ace-4e51-b46a-5788f64d46a7-kube-api-access-zj8vs\") pod \"maas-api-e2e-api-a-91b59b-6456677d69-s77r4\" (UID: \"e8f8afc3-2ace-4e51-b46a-5788f64d46a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-91b59b-6456677d69-s77r4" Jul 21 02:14:09.115097 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:14:09.115063 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj8vs\" (UniqueName: \"kubernetes.io/projected/e8f8afc3-2ace-4e51-b46a-5788f64d46a7-kube-api-access-zj8vs\") pod \"maas-api-e2e-api-a-91b59b-6456677d69-s77r4\" (UID: \"e8f8afc3-2ace-4e51-b46a-5788f64d46a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-91b59b-6456677d69-s77r4" Jul 21 02:14:09.115280 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:14:09.115137 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8f8afc3-2ace-4e51-b46a-5788f64d46a7-maas-api-tls\") pod \"maas-api-e2e-api-a-91b59b-6456677d69-s77r4\" (UID: \"e8f8afc3-2ace-4e51-b46a-5788f64d46a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-91b59b-6456677d69-s77r4" Jul 21 02:14:09.115280 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:14:09.115209 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8f8afc3-2ace-4e51-b46a-5788f64d46a7-openshift-service-ca\") pod \"maas-api-e2e-api-a-91b59b-6456677d69-s77r4\" (UID: \"e8f8afc3-2ace-4e51-b46a-5788f64d46a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-91b59b-6456677d69-s77r4" Jul 21 02:14:09.115362 ip-10-0-139-183 kubenswrapper[2580]: E0721 02:14:09.115281 2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-91b59b-serving-cert: secret "maas-api-e2e-api-a-91b59b-serving-cert" not found Jul 21 02:14:09.115362 ip-10-0-139-183 kubenswrapper[2580]: E0721 02:14:09.115358 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8f8afc3-2ace-4e51-b46a-5788f64d46a7-maas-api-tls podName:e8f8afc3-2ace-4e51-b46a-5788f64d46a7 nodeName:}" failed. No retries permitted until 2026-07-21 02:14:09.615340774 +0000 UTC m=+2273.202506730 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e8f8afc3-2ace-4e51-b46a-5788f64d46a7-maas-api-tls") pod "maas-api-e2e-api-a-91b59b-6456677d69-s77r4" (UID: "e8f8afc3-2ace-4e51-b46a-5788f64d46a7") : secret "maas-api-e2e-api-a-91b59b-serving-cert" not found Jul 21 02:14:09.115917 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:14:09.115900 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8f8afc3-2ace-4e51-b46a-5788f64d46a7-openshift-service-ca\") pod \"maas-api-e2e-api-a-91b59b-6456677d69-s77r4\" (UID: \"e8f8afc3-2ace-4e51-b46a-5788f64d46a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-91b59b-6456677d69-s77r4" Jul 21 02:14:09.128608 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:14:09.128569 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj8vs\" (UniqueName: \"kubernetes.io/projected/e8f8afc3-2ace-4e51-b46a-5788f64d46a7-kube-api-access-zj8vs\") pod \"maas-api-e2e-api-a-91b59b-6456677d69-s77r4\" (UID: \"e8f8afc3-2ace-4e51-b46a-5788f64d46a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-91b59b-6456677d69-s77r4" Jul 21 02:14:09.621161 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:14:09.621123 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8f8afc3-2ace-4e51-b46a-5788f64d46a7-maas-api-tls\") pod \"maas-api-e2e-api-a-91b59b-6456677d69-s77r4\" (UID: \"e8f8afc3-2ace-4e51-b46a-5788f64d46a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-91b59b-6456677d69-s77r4" Jul 21 02:14:09.623586 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:14:09.623556 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8f8afc3-2ace-4e51-b46a-5788f64d46a7-maas-api-tls\") pod \"maas-api-e2e-api-a-91b59b-6456677d69-s77r4\" (UID: \"e8f8afc3-2ace-4e51-b46a-5788f64d46a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-91b59b-6456677d69-s77r4" Jul 21 02:14:09.804647 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:14:09.804616 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-91b59b-6456677d69-s77r4" Jul 21 02:14:09.962181 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:14:09.962154 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-91b59b-6456677d69-s77r4"] Jul 21 02:14:09.964181 ip-10-0-139-183 kubenswrapper[2580]: W0721 02:14:09.964154 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8f8afc3_2ace_4e51_b46a_5788f64d46a7.slice/crio-7b8e196c75dee489e4a5582f466f855130d4b843c83bd9dfe764592888c9fbc9 WatchSource:0}: Error finding container 7b8e196c75dee489e4a5582f466f855130d4b843c83bd9dfe764592888c9fbc9: Status 404 returned error can't find the container with id 7b8e196c75dee489e4a5582f466f855130d4b843c83bd9dfe764592888c9fbc9 Jul 21 02:14:09.965596 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:14:09.965577 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 02:14:10.474334 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:14:10.474300 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-91b59b-6456677d69-s77r4" event={"ID":"e8f8afc3-2ace-4e51-b46a-5788f64d46a7","Type":"ContainerStarted","Data":"7b8e196c75dee489e4a5582f466f855130d4b843c83bd9dfe764592888c9fbc9"} Jul 21 02:14:11.480751 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:14:11.480718 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-91b59b-6456677d69-s77r4" event={"ID":"e8f8afc3-2ace-4e51-b46a-5788f64d46a7","Type":"ContainerStarted","Data":"f0daa01f9137b1a6548c7ab877b1f178aea379f7941126e6536759506fdddd86"} Jul 21 02:14:11.481169 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:14:11.480855 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-91b59b-6456677d69-s77r4" Jul 21 02:14:11.497022 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:14:11.496974 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-91b59b-6456677d69-s77r4" podStartSLOduration=2.936893154 podStartE2EDuration="3.496957514s" totalImagesPullingTime="560.06436ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 02:14:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 02:14:11.496241907 +0000 UTC m=+2275.083407876" watchObservedRunningTime="2026-07-21 02:14:11.496957514 +0000 UTC m=+2275.084123492" Jul 21 02:14:15.804685 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:14:15.804645 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 02:14:17.491953 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:14:17.491883 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-91b59b-6456677d69-s77r4" Jul 21 02:14:31.504319 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:14:31.504282 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 02:14:45.496669 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:14:45.496628 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-91b59b-6456677d69-s77r4"] Jul 21 02:14:45.497143 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:14:45.496903 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-91b59b-6456677d69-s77r4" podUID="e8f8afc3-2ace-4e51-b46a-5788f64d46a7" containerName="maas-api" containerID="cri-o://f0daa01f9137b1a6548c7ab877b1f178aea379f7941126e6536759506fdddd86" gracePeriod=30 Jul 21 02:14:45.673330 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:14:45.673090 2580 generic.go:356] "Generic (PLEG): container finished" podID="e8f8afc3-2ace-4e51-b46a-5788f64d46a7" containerID="f0daa01f9137b1a6548c7ab877b1f178aea379f7941126e6536759506fdddd86" exitCode=0 Jul 21 02:14:45.673330 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:14:45.673260 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-91b59b-6456677d69-s77r4" event={"ID":"e8f8afc3-2ace-4e51-b46a-5788f64d46a7","Type":"ContainerDied","Data":"f0daa01f9137b1a6548c7ab877b1f178aea379f7941126e6536759506fdddd86"} Jul 21 02:14:45.757626 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:14:45.757602 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-91b59b-6456677d69-s77r4" Jul 21 02:14:45.885614 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:14:45.885579 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8f8afc3-2ace-4e51-b46a-5788f64d46a7-openshift-service-ca\") pod \"e8f8afc3-2ace-4e51-b46a-5788f64d46a7\" (UID: \"e8f8afc3-2ace-4e51-b46a-5788f64d46a7\") " Jul 21 02:14:45.885807 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:14:45.885669 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zj8vs\" (UniqueName: \"kubernetes.io/projected/e8f8afc3-2ace-4e51-b46a-5788f64d46a7-kube-api-access-zj8vs\") pod \"e8f8afc3-2ace-4e51-b46a-5788f64d46a7\" (UID: \"e8f8afc3-2ace-4e51-b46a-5788f64d46a7\") " Jul 21 02:14:45.885807 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:14:45.885736 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8f8afc3-2ace-4e51-b46a-5788f64d46a7-maas-api-tls\") pod \"e8f8afc3-2ace-4e51-b46a-5788f64d46a7\" (UID: \"e8f8afc3-2ace-4e51-b46a-5788f64d46a7\") " Jul 21 02:14:45.885932 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:14:45.885915 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8f8afc3-2ace-4e51-b46a-5788f64d46a7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e8f8afc3-2ace-4e51-b46a-5788f64d46a7" (UID: "e8f8afc3-2ace-4e51-b46a-5788f64d46a7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 02:14:45.887864 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:14:45.887831 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8f8afc3-2ace-4e51-b46a-5788f64d46a7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e8f8afc3-2ace-4e51-b46a-5788f64d46a7" (UID: "e8f8afc3-2ace-4e51-b46a-5788f64d46a7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 02:14:45.887864 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:14:45.887836 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8f8afc3-2ace-4e51-b46a-5788f64d46a7-kube-api-access-zj8vs" (OuterVolumeSpecName: "kube-api-access-zj8vs") pod "e8f8afc3-2ace-4e51-b46a-5788f64d46a7" (UID: "e8f8afc3-2ace-4e51-b46a-5788f64d46a7"). InnerVolumeSpecName "kube-api-access-zj8vs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 02:14:45.986958 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:14:45.986928 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8f8afc3-2ace-4e51-b46a-5788f64d46a7-maas-api-tls\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 21 02:14:45.986958 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:14:45.986953 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8f8afc3-2ace-4e51-b46a-5788f64d46a7-openshift-service-ca\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 21 02:14:45.986958 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:14:45.986963 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zj8vs\" (UniqueName: \"kubernetes.io/projected/e8f8afc3-2ace-4e51-b46a-5788f64d46a7-kube-api-access-zj8vs\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 21 02:14:46.680260 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:14:46.680229 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-91b59b-6456677d69-s77r4" event={"ID":"e8f8afc3-2ace-4e51-b46a-5788f64d46a7","Type":"ContainerDied","Data":"7b8e196c75dee489e4a5582f466f855130d4b843c83bd9dfe764592888c9fbc9"} Jul 21 02:14:46.680743 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:14:46.680296 2580 scope.go:117] "RemoveContainer" containerID="f0daa01f9137b1a6548c7ab877b1f178aea379f7941126e6536759506fdddd86" Jul 21 02:14:46.680743 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:14:46.680240 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-91b59b-6456677d69-s77r4" Jul 21 02:14:46.702726 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:14:46.702703 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-91b59b-6456677d69-s77r4"] Jul 21 02:14:46.707595 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:14:46.707571 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-91b59b-6456677d69-s77r4"] Jul 21 02:14:46.877933 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:14:46.877896 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8f8afc3-2ace-4e51-b46a-5788f64d46a7" path="/var/lib/kubelet/pods/e8f8afc3-2ace-4e51-b46a-5788f64d46a7/volumes" Jul 21 02:14:51.303270 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:14:51.303238 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 02:15:17.202189 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:15:17.202155 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 02:15:35.300641 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:15:35.300605 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c6538a-8484cb76c-w4kz4"] Jul 21 02:15:35.301437 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:15:35.301413 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8f8afc3-2ace-4e51-b46a-5788f64d46a7" containerName="maas-api" Jul 21 02:15:35.301437 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:15:35.301438 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8f8afc3-2ace-4e51-b46a-5788f64d46a7" containerName="maas-api" Jul 21 02:15:35.301642 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:15:35.301582 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8f8afc3-2ace-4e51-b46a-5788f64d46a7" containerName="maas-api" Jul 21 02:15:35.304975 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:15:35.304954 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c6538a-8484cb76c-w4kz4" Jul 21 02:15:35.307793 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:15:35.307766 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-c6538a-serving-cert\"" Jul 21 02:15:35.307910 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:15:35.307834 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 21 02:15:35.308731 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:15:35.308651 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 21 02:15:35.308731 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:15:35.308717 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-z67q6\"" Jul 21 02:15:35.318179 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:15:35.318133 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c6538a-8484cb76c-w4kz4"] Jul 21 02:15:35.470971 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:15:35.470934 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33e2b5ba-e528-4cad-903c-3fc6da9674f7-openshift-service-ca\") pod \"maas-api-e2e-shared-b-c6538a-8484cb76c-w4kz4\" (UID: \"33e2b5ba-e528-4cad-903c-3fc6da9674f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c6538a-8484cb76c-w4kz4" Jul 21 02:15:35.471167 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:15:35.471033 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxmgr\" (UniqueName: \"kubernetes.io/projected/33e2b5ba-e528-4cad-903c-3fc6da9674f7-kube-api-access-sxmgr\") pod \"maas-api-e2e-shared-b-c6538a-8484cb76c-w4kz4\" (UID: \"33e2b5ba-e528-4cad-903c-3fc6da9674f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c6538a-8484cb76c-w4kz4" Jul 21 02:15:35.471167 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:15:35.471088 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33e2b5ba-e528-4cad-903c-3fc6da9674f7-maas-api-tls\") pod \"maas-api-e2e-shared-b-c6538a-8484cb76c-w4kz4\" (UID: \"33e2b5ba-e528-4cad-903c-3fc6da9674f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c6538a-8484cb76c-w4kz4" Jul 21 02:15:35.519070 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:15:35.519037 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 02:15:35.571968 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:15:35.571884 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33e2b5ba-e528-4cad-903c-3fc6da9674f7-maas-api-tls\") pod \"maas-api-e2e-shared-b-c6538a-8484cb76c-w4kz4\" (UID: \"33e2b5ba-e528-4cad-903c-3fc6da9674f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c6538a-8484cb76c-w4kz4" Jul 21 02:15:35.571968 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:15:35.571928 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33e2b5ba-e528-4cad-903c-3fc6da9674f7-openshift-service-ca\") pod \"maas-api-e2e-shared-b-c6538a-8484cb76c-w4kz4\" (UID: \"33e2b5ba-e528-4cad-903c-3fc6da9674f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c6538a-8484cb76c-w4kz4" Jul 21 02:15:35.572163 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:15:35.571994 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxmgr\" (UniqueName: \"kubernetes.io/projected/33e2b5ba-e528-4cad-903c-3fc6da9674f7-kube-api-access-sxmgr\") pod \"maas-api-e2e-shared-b-c6538a-8484cb76c-w4kz4\" (UID: \"33e2b5ba-e528-4cad-903c-3fc6da9674f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c6538a-8484cb76c-w4kz4" Jul 21 02:15:35.572722 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:15:35.572696 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33e2b5ba-e528-4cad-903c-3fc6da9674f7-openshift-service-ca\") pod \"maas-api-e2e-shared-b-c6538a-8484cb76c-w4kz4\" (UID: \"33e2b5ba-e528-4cad-903c-3fc6da9674f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c6538a-8484cb76c-w4kz4" Jul 21 02:15:35.574393 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:15:35.574373 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33e2b5ba-e528-4cad-903c-3fc6da9674f7-maas-api-tls\") pod \"maas-api-e2e-shared-b-c6538a-8484cb76c-w4kz4\" (UID: \"33e2b5ba-e528-4cad-903c-3fc6da9674f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c6538a-8484cb76c-w4kz4" Jul 21 02:15:35.580399 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:15:35.580378 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxmgr\" (UniqueName: \"kubernetes.io/projected/33e2b5ba-e528-4cad-903c-3fc6da9674f7-kube-api-access-sxmgr\") pod \"maas-api-e2e-shared-b-c6538a-8484cb76c-w4kz4\" (UID: \"33e2b5ba-e528-4cad-903c-3fc6da9674f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c6538a-8484cb76c-w4kz4" Jul 21 02:15:35.619597 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:15:35.619557 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c6538a-8484cb76c-w4kz4" Jul 21 02:15:35.781005 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:15:35.780979 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c6538a-8484cb76c-w4kz4"] Jul 21 02:15:35.784065 ip-10-0-139-183 kubenswrapper[2580]: W0721 02:15:35.784038 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33e2b5ba_e528_4cad_903c_3fc6da9674f7.slice/crio-7a08467ec0dc70d1f36e66ba0017ea98df223d0a4ab2d096469f384b83cecaf9 WatchSource:0}: Error finding container 7a08467ec0dc70d1f36e66ba0017ea98df223d0a4ab2d096469f384b83cecaf9: Status 404 returned error can't find the container with id 7a08467ec0dc70d1f36e66ba0017ea98df223d0a4ab2d096469f384b83cecaf9 Jul 21 02:15:35.952292 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:15:35.952207 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c6538a-8484cb76c-w4kz4" event={"ID":"33e2b5ba-e528-4cad-903c-3fc6da9674f7","Type":"ContainerStarted","Data":"7a08467ec0dc70d1f36e66ba0017ea98df223d0a4ab2d096469f384b83cecaf9"} Jul 21 02:15:36.960075 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:15:36.960042 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c6538a-8484cb76c-w4kz4" event={"ID":"33e2b5ba-e528-4cad-903c-3fc6da9674f7","Type":"ContainerStarted","Data":"b52ea0cc1b577ca8daad5ee17c70814b7e85b0e8470e9b062d4bb50ccd4d9447"} Jul 21 02:15:36.960492 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:15:36.960136 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c6538a-8484cb76c-w4kz4" Jul 21 02:15:36.980097 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:15:36.980047 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c6538a-8484cb76c-w4kz4" podStartSLOduration=1.605988631 podStartE2EDuration="1.980027099s" totalImagesPullingTime="374.038468ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 02:15:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 02:15:36.978499189 +0000 UTC m=+2360.565665179" watchObservedRunningTime="2026-07-21 02:15:36.980027099 +0000 UTC m=+2360.567193076" Jul 21 02:15:42.971579 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:15:42.971547 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c6538a-8484cb76c-w4kz4" Jul 21 02:15:46.698320 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:15:46.698284 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 02:15:54.619283 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:15:54.619244 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 02:16:17.014566 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:16:17.014538 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-9ctn2_1e930d67-d711-419c-af02-86b43027a95d/cluster-monitoring-operator/0.log" Jul 21 02:16:17.023197 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:16:17.023166 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-4mtp7_9c380009-9bf5-4b73-803a-5a1ea9c1abf0/console-operator/1.log" Jul 21 02:16:17.026340 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:16:17.026320 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-9ctn2_1e930d67-d711-419c-af02-86b43027a95d/cluster-monitoring-operator/0.log" Jul 21 02:16:17.031643 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:16:17.031623 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-4mtp7_9c380009-9bf5-4b73-803a-5a1ea9c1abf0/console-operator/1.log" Jul 21 02:16:36.317594 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:16:36.317559 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 02:16:54.596357 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:16:54.596321 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 02:17:12.702284 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:17:12.702255 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 02:17:27.199387 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:17:27.199345 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 02:17:44.697110 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:17:44.697074 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 02:18:04.702325 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:18:04.702288 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 02:18:21.606161 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:18:21.606081 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 02:18:40.604010 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:18:40.603973 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 02:18:50.398729 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:18:50.398686 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 02:19:01.492894 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:19:01.492856 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 02:19:15.101649 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:19:15.101611 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 02:19:46.901134 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:19:46.901055 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 02:20:06.637801 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:20:06.637761 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 02:20:24.402400 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:20:24.402369 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 02:20:34.293545 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:20:34.293505 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 02:20:47.494325 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:20:47.494285 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 02:20:59.217201 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:20:59.217160 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 02:21:17.059628 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:17.059493 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-9ctn2_1e930d67-d711-419c-af02-86b43027a95d/cluster-monitoring-operator/0.log" Jul 21 02:21:17.064337 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:17.064317 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-9ctn2_1e930d67-d711-419c-af02-86b43027a95d/cluster-monitoring-operator/0.log" Jul 21 02:21:17.064483 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:17.064322 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-4mtp7_9c380009-9bf5-4b73-803a-5a1ea9c1abf0/console-operator/1.log" Jul 21 02:21:17.069166 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:17.069144 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-4mtp7_9c380009-9bf5-4b73-803a-5a1ea9c1abf0/console-operator/1.log" Jul 21 02:21:21.695332 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:21.695299 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 02:21:27.266522 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:27.266489 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7696-139869606577072"] Jul 21 02:21:27.269931 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:27.269914 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7696-139869606577072" Jul 21 02:21:27.272059 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:27.272036 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-s5sq8\"" Jul 21 02:21:27.280593 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:27.279759 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7696-139869606577072"] Jul 21 02:21:27.306943 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:27.306902 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw7g5\" (UniqueName: \"kubernetes.io/projected/f6429bb6-1b47-413f-b96b-1137580f6d48-kube-api-access-rw7g5\") pod \"test-curl-7696-139869606577072\" (UID: \"f6429bb6-1b47-413f-b96b-1137580f6d48\") " pod="opendatahub/test-curl-7696-139869606577072" Jul 21 02:21:27.408486 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:27.408412 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw7g5\" (UniqueName: \"kubernetes.io/projected/f6429bb6-1b47-413f-b96b-1137580f6d48-kube-api-access-rw7g5\") pod \"test-curl-7696-139869606577072\" (UID: \"f6429bb6-1b47-413f-b96b-1137580f6d48\") " pod="opendatahub/test-curl-7696-139869606577072" Jul 21 02:21:27.416742 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:27.416712 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw7g5\" (UniqueName: \"kubernetes.io/projected/f6429bb6-1b47-413f-b96b-1137580f6d48-kube-api-access-rw7g5\") pod \"test-curl-7696-139869606577072\" (UID: \"f6429bb6-1b47-413f-b96b-1137580f6d48\") " pod="opendatahub/test-curl-7696-139869606577072" Jul 21 02:21:27.582107 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:27.582004 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7696-139869606577072" Jul 21 02:21:27.733095 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:27.733070 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7696-139869606577072"] Jul 21 02:21:27.735367 ip-10-0-139-183 kubenswrapper[2580]: W0721 02:21:27.735338 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf6429bb6_1b47_413f_b96b_1137580f6d48.slice/crio-6083eeef8170864024c667e2ef1b08fd861dde4fdb496bb1ba5a28d05afe864a WatchSource:0}: Error finding container 6083eeef8170864024c667e2ef1b08fd861dde4fdb496bb1ba5a28d05afe864a: Status 404 returned error can't find the container with id 6083eeef8170864024c667e2ef1b08fd861dde4fdb496bb1ba5a28d05afe864a Jul 21 02:21:27.736795 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:27.736778 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 02:21:27.947820 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:27.947735 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7696-139869606577072" event={"ID":"f6429bb6-1b47-413f-b96b-1137580f6d48","Type":"ContainerStarted","Data":"6083eeef8170864024c667e2ef1b08fd861dde4fdb496bb1ba5a28d05afe864a"} Jul 21 02:21:28.954822 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:28.954787 2580 generic.go:356] "Generic (PLEG): container finished" podID="f6429bb6-1b47-413f-b96b-1137580f6d48" containerID="0b0a8597dc959eb22747b2e5fdad71d144597bab18133c1c6c825019a5daa5cd" exitCode=0 Jul 21 02:21:28.955206 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:28.954911 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7696-139869606577072" event={"ID":"f6429bb6-1b47-413f-b96b-1137580f6d48","Type":"ContainerDied","Data":"0b0a8597dc959eb22747b2e5fdad71d144597bab18133c1c6c825019a5daa5cd"} Jul 21 02:21:30.108270 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:30.108246 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7696-139869606577072" Jul 21 02:21:30.143770 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:30.143737 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7696-139869606577072_f6429bb6-1b47-413f-b96b-1137580f6d48/test-curl-7696-139869606577072/0.log" Jul 21 02:21:30.169196 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:30.169161 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7696-139869606577072"] Jul 21 02:21:30.174811 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:30.174788 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7696-139869606577072"] Jul 21 02:21:30.235793 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:30.235705 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rw7g5\" (UniqueName: \"kubernetes.io/projected/f6429bb6-1b47-413f-b96b-1137580f6d48-kube-api-access-rw7g5\") pod \"f6429bb6-1b47-413f-b96b-1137580f6d48\" (UID: \"f6429bb6-1b47-413f-b96b-1137580f6d48\") " Jul 21 02:21:30.237902 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:30.237862 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6429bb6-1b47-413f-b96b-1137580f6d48-kube-api-access-rw7g5" (OuterVolumeSpecName: "kube-api-access-rw7g5") pod "f6429bb6-1b47-413f-b96b-1137580f6d48" (UID: "f6429bb6-1b47-413f-b96b-1137580f6d48"). InnerVolumeSpecName "kube-api-access-rw7g5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 02:21:30.291101 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:30.291069 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7696-139869614192048"] Jul 21 02:21:30.291527 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:30.291514 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6429bb6-1b47-413f-b96b-1137580f6d48" containerName="test-curl-7696-139869606577072" Jul 21 02:21:30.291585 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:30.291528 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6429bb6-1b47-413f-b96b-1137580f6d48" containerName="test-curl-7696-139869606577072" Jul 21 02:21:30.291627 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:30.291617 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6429bb6-1b47-413f-b96b-1137580f6d48" containerName="test-curl-7696-139869606577072" Jul 21 02:21:30.294701 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:30.294686 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7696-139869614192048" Jul 21 02:21:30.300981 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:30.300952 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7696-139869614192048"] Jul 21 02:21:30.337066 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:30.337024 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjbpb\" (UniqueName: \"kubernetes.io/projected/beb768ed-dca1-4b61-9c42-1aed714016db-kube-api-access-hjbpb\") pod \"test-curl-7696-139869614192048\" (UID: \"beb768ed-dca1-4b61-9c42-1aed714016db\") " pod="opendatahub/test-curl-7696-139869614192048" Jul 21 02:21:30.337303 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:30.337203 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rw7g5\" (UniqueName: \"kubernetes.io/projected/f6429bb6-1b47-413f-b96b-1137580f6d48-kube-api-access-rw7g5\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 21 02:21:30.438474 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:30.438417 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjbpb\" (UniqueName: \"kubernetes.io/projected/beb768ed-dca1-4b61-9c42-1aed714016db-kube-api-access-hjbpb\") pod \"test-curl-7696-139869614192048\" (UID: \"beb768ed-dca1-4b61-9c42-1aed714016db\") " pod="opendatahub/test-curl-7696-139869614192048" Jul 21 02:21:30.446962 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:30.446936 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjbpb\" (UniqueName: \"kubernetes.io/projected/beb768ed-dca1-4b61-9c42-1aed714016db-kube-api-access-hjbpb\") pod \"test-curl-7696-139869614192048\" (UID: \"beb768ed-dca1-4b61-9c42-1aed714016db\") " pod="opendatahub/test-curl-7696-139869614192048" Jul 21 02:21:30.604957 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:30.604919 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7696-139869614192048" Jul 21 02:21:30.751525 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:30.751490 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7696-139869614192048"] Jul 21 02:21:30.754856 ip-10-0-139-183 kubenswrapper[2580]: W0721 02:21:30.754823 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbeb768ed_dca1_4b61_9c42_1aed714016db.slice/crio-1460fe30d2757850663336f58d5f07b54ca89499a292cf7c2f5f2b6b96f71797 WatchSource:0}: Error finding container 1460fe30d2757850663336f58d5f07b54ca89499a292cf7c2f5f2b6b96f71797: Status 404 returned error can't find the container with id 1460fe30d2757850663336f58d5f07b54ca89499a292cf7c2f5f2b6b96f71797 Jul 21 02:21:30.877879 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:30.877781 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6429bb6-1b47-413f-b96b-1137580f6d48" path="/var/lib/kubelet/pods/f6429bb6-1b47-413f-b96b-1137580f6d48/volumes" Jul 21 02:21:30.966832 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:30.966806 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7696-139869606577072" Jul 21 02:21:30.967033 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:30.966870 2580 scope.go:117] "RemoveContainer" containerID="0b0a8597dc959eb22747b2e5fdad71d144597bab18133c1c6c825019a5daa5cd" Jul 21 02:21:30.968268 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:30.968244 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7696-139869614192048" event={"ID":"beb768ed-dca1-4b61-9c42-1aed714016db","Type":"ContainerStarted","Data":"1460fe30d2757850663336f58d5f07b54ca89499a292cf7c2f5f2b6b96f71797"} Jul 21 02:21:31.974639 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:31.974605 2580 generic.go:356] "Generic (PLEG): container finished" podID="beb768ed-dca1-4b61-9c42-1aed714016db" containerID="f3633d8a10dd534fe9dea27628f8055b3c91b1d8c1716fc9403fd1fe118afbf7" exitCode=0 Jul 21 02:21:31.975056 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:31.974680 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7696-139869614192048" event={"ID":"beb768ed-dca1-4b61-9c42-1aed714016db","Type":"ContainerDied","Data":"f3633d8a10dd534fe9dea27628f8055b3c91b1d8c1716fc9403fd1fe118afbf7"} Jul 21 02:21:33.137070 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:33.137042 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7696-139869614192048" Jul 21 02:21:33.157880 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:33.157826 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7696-139869614192048_beb768ed-dca1-4b61-9c42-1aed714016db/test-curl-7696-139869614192048/0.log" Jul 21 02:21:33.183809 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:33.183779 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7696-139869614192048"] Jul 21 02:21:33.186088 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:33.186055 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7696-139869614192048"] Jul 21 02:21:33.265771 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:33.265742 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hjbpb\" (UniqueName: \"kubernetes.io/projected/beb768ed-dca1-4b61-9c42-1aed714016db-kube-api-access-hjbpb\") pod \"beb768ed-dca1-4b61-9c42-1aed714016db\" (UID: \"beb768ed-dca1-4b61-9c42-1aed714016db\") " Jul 21 02:21:33.267859 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:33.267829 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/beb768ed-dca1-4b61-9c42-1aed714016db-kube-api-access-hjbpb" (OuterVolumeSpecName: "kube-api-access-hjbpb") pod "beb768ed-dca1-4b61-9c42-1aed714016db" (UID: "beb768ed-dca1-4b61-9c42-1aed714016db"). InnerVolumeSpecName "kube-api-access-hjbpb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 02:21:33.348425 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:33.348391 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7696-139869606534144"] Jul 21 02:21:33.348865 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:33.348851 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="beb768ed-dca1-4b61-9c42-1aed714016db" containerName="test-curl-7696-139869614192048" Jul 21 02:21:33.348910 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:33.348866 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="beb768ed-dca1-4b61-9c42-1aed714016db" containerName="test-curl-7696-139869614192048" Jul 21 02:21:33.348955 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:33.348946 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="beb768ed-dca1-4b61-9c42-1aed714016db" containerName="test-curl-7696-139869614192048" Jul 21 02:21:33.352195 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:33.352178 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7696-139869606534144" Jul 21 02:21:33.359029 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:33.358997 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7696-139869606534144"] Jul 21 02:21:33.367100 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:33.367067 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjbpb\" (UniqueName: \"kubernetes.io/projected/beb768ed-dca1-4b61-9c42-1aed714016db-kube-api-access-hjbpb\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 21 02:21:33.467807 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:33.467769 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n956w\" (UniqueName: \"kubernetes.io/projected/6ddb18e8-8aed-43ae-9e26-e2a97f377a9b-kube-api-access-n956w\") pod \"test-curl-7696-139869606534144\" (UID: \"6ddb18e8-8aed-43ae-9e26-e2a97f377a9b\") " pod="opendatahub/test-curl-7696-139869606534144" Jul 21 02:21:33.568600 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:33.568507 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n956w\" (UniqueName: \"kubernetes.io/projected/6ddb18e8-8aed-43ae-9e26-e2a97f377a9b-kube-api-access-n956w\") pod \"test-curl-7696-139869606534144\" (UID: \"6ddb18e8-8aed-43ae-9e26-e2a97f377a9b\") " pod="opendatahub/test-curl-7696-139869606534144" Jul 21 02:21:33.576079 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:33.576045 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n956w\" (UniqueName: \"kubernetes.io/projected/6ddb18e8-8aed-43ae-9e26-e2a97f377a9b-kube-api-access-n956w\") pod \"test-curl-7696-139869606534144\" (UID: \"6ddb18e8-8aed-43ae-9e26-e2a97f377a9b\") " pod="opendatahub/test-curl-7696-139869606534144" Jul 21 02:21:33.664284 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:33.664241 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7696-139869606534144" Jul 21 02:21:33.810148 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:33.809942 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7696-139869606534144"] Jul 21 02:21:33.812409 ip-10-0-139-183 kubenswrapper[2580]: W0721 02:21:33.812376 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6ddb18e8_8aed_43ae_9e26_e2a97f377a9b.slice/crio-4bd8e3583bd243748055cc3f7220d5b639557be225e4c4d07984b6d10a7749d1 WatchSource:0}: Error finding container 4bd8e3583bd243748055cc3f7220d5b639557be225e4c4d07984b6d10a7749d1: Status 404 returned error can't find the container with id 4bd8e3583bd243748055cc3f7220d5b639557be225e4c4d07984b6d10a7749d1 Jul 21 02:21:33.988520 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:33.988492 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1460fe30d2757850663336f58d5f07b54ca89499a292cf7c2f5f2b6b96f71797" Jul 21 02:21:33.988520 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:33.988500 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7696-139869614192048" Jul 21 02:21:33.989662 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:33.989638 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7696-139869606534144" event={"ID":"6ddb18e8-8aed-43ae-9e26-e2a97f377a9b","Type":"ContainerStarted","Data":"4bd8e3583bd243748055cc3f7220d5b639557be225e4c4d07984b6d10a7749d1"} Jul 21 02:21:33.998931 ip-10-0-139-183 kubenswrapper[2580]: E0721 02:21:33.998902 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7696-139869614192048\" is forbidden: User \"system:node:ip-10-0-139-183.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-183.ec2.internal' and this object" podUID="beb768ed-dca1-4b61-9c42-1aed714016db" pod="opendatahub/test-curl-7696-139869614192048" Jul 21 02:21:34.877922 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:34.877888 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="beb768ed-dca1-4b61-9c42-1aed714016db" path="/var/lib/kubelet/pods/beb768ed-dca1-4b61-9c42-1aed714016db/volumes" Jul 21 02:21:34.996500 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:34.996467 2580 generic.go:356] "Generic (PLEG): container finished" podID="6ddb18e8-8aed-43ae-9e26-e2a97f377a9b" containerID="af59d67c7913abf219067df66b4fd1fb75c6cb006cdcf87c4d4f7764807e7011" exitCode=0 Jul 21 02:21:34.996666 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:34.996539 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7696-139869606534144" event={"ID":"6ddb18e8-8aed-43ae-9e26-e2a97f377a9b","Type":"ContainerDied","Data":"af59d67c7913abf219067df66b4fd1fb75c6cb006cdcf87c4d4f7764807e7011"} Jul 21 02:21:36.147663 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:36.147628 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7696-139869606534144" Jul 21 02:21:36.166818 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:36.166789 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7696-139869606534144_6ddb18e8-8aed-43ae-9e26-e2a97f377a9b/test-curl-7696-139869606534144/0.log" Jul 21 02:21:36.192981 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:36.192945 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7696-139869606534144"] Jul 21 02:21:36.196129 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:36.196102 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7696-139869606534144"] Jul 21 02:21:36.296958 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:36.296878 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n956w\" (UniqueName: \"kubernetes.io/projected/6ddb18e8-8aed-43ae-9e26-e2a97f377a9b-kube-api-access-n956w\") pod \"6ddb18e8-8aed-43ae-9e26-e2a97f377a9b\" (UID: \"6ddb18e8-8aed-43ae-9e26-e2a97f377a9b\") " Jul 21 02:21:36.298910 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:36.298879 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ddb18e8-8aed-43ae-9e26-e2a97f377a9b-kube-api-access-n956w" (OuterVolumeSpecName: "kube-api-access-n956w") pod "6ddb18e8-8aed-43ae-9e26-e2a97f377a9b" (UID: "6ddb18e8-8aed-43ae-9e26-e2a97f377a9b"). InnerVolumeSpecName "kube-api-access-n956w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 02:21:36.315753 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:36.315728 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7696-139869606534144"] Jul 21 02:21:36.316164 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:36.316150 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ddb18e8-8aed-43ae-9e26-e2a97f377a9b" containerName="test-curl-7696-139869606534144" Jul 21 02:21:36.316204 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:36.316166 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ddb18e8-8aed-43ae-9e26-e2a97f377a9b" containerName="test-curl-7696-139869606534144" Jul 21 02:21:36.316259 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:36.316250 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ddb18e8-8aed-43ae-9e26-e2a97f377a9b" containerName="test-curl-7696-139869606534144" Jul 21 02:21:36.319222 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:36.319208 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7696-139869606534144" Jul 21 02:21:36.325188 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:36.325166 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7696-139869606534144"] Jul 21 02:21:36.398408 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:36.398373 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74wl5\" (UniqueName: \"kubernetes.io/projected/50d85a12-84ee-4c3c-b3dc-122c5430f0e5-kube-api-access-74wl5\") pod \"test-curl-7696-139869606534144\" (UID: \"50d85a12-84ee-4c3c-b3dc-122c5430f0e5\") " pod="opendatahub/test-curl-7696-139869606534144" Jul 21 02:21:36.398616 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:36.398483 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n956w\" (UniqueName: \"kubernetes.io/projected/6ddb18e8-8aed-43ae-9e26-e2a97f377a9b-kube-api-access-n956w\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 21 02:21:36.499176 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:36.499136 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74wl5\" (UniqueName: \"kubernetes.io/projected/50d85a12-84ee-4c3c-b3dc-122c5430f0e5-kube-api-access-74wl5\") pod \"test-curl-7696-139869606534144\" (UID: \"50d85a12-84ee-4c3c-b3dc-122c5430f0e5\") " pod="opendatahub/test-curl-7696-139869606534144" Jul 21 02:21:36.508063 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:36.508038 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74wl5\" (UniqueName: \"kubernetes.io/projected/50d85a12-84ee-4c3c-b3dc-122c5430f0e5-kube-api-access-74wl5\") pod \"test-curl-7696-139869606534144\" (UID: \"50d85a12-84ee-4c3c-b3dc-122c5430f0e5\") " pod="opendatahub/test-curl-7696-139869606534144" Jul 21 02:21:36.630907 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:36.630816 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7696-139869606534144" Jul 21 02:21:36.775145 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:36.775113 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7696-139869606534144"] Jul 21 02:21:36.778832 ip-10-0-139-183 kubenswrapper[2580]: W0721 02:21:36.778798 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod50d85a12_84ee_4c3c_b3dc_122c5430f0e5.slice/crio-10fd18ee7de1488acd59f22a75ee7ae0bd05c7ee96eebb27dd6ca926df654668 WatchSource:0}: Error finding container 10fd18ee7de1488acd59f22a75ee7ae0bd05c7ee96eebb27dd6ca926df654668: Status 404 returned error can't find the container with id 10fd18ee7de1488acd59f22a75ee7ae0bd05c7ee96eebb27dd6ca926df654668 Jul 21 02:21:36.878218 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:36.878183 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ddb18e8-8aed-43ae-9e26-e2a97f377a9b" path="/var/lib/kubelet/pods/6ddb18e8-8aed-43ae-9e26-e2a97f377a9b/volumes" Jul 21 02:21:37.008578 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:37.008552 2580 scope.go:117] "RemoveContainer" containerID="af59d67c7913abf219067df66b4fd1fb75c6cb006cdcf87c4d4f7764807e7011" Jul 21 02:21:37.008578 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:37.008572 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7696-139869606534144" Jul 21 02:21:37.009760 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:37.009731 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7696-139869606534144" event={"ID":"50d85a12-84ee-4c3c-b3dc-122c5430f0e5","Type":"ContainerStarted","Data":"10fd18ee7de1488acd59f22a75ee7ae0bd05c7ee96eebb27dd6ca926df654668"} Jul 21 02:21:38.017887 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:38.017856 2580 generic.go:356] "Generic (PLEG): container finished" podID="50d85a12-84ee-4c3c-b3dc-122c5430f0e5" containerID="cb0ae0d9e184d04fcc206f1e629805a0807cb247d8f7d6a733b251478df78295" exitCode=0 Jul 21 02:21:38.018284 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:38.017944 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7696-139869606534144" event={"ID":"50d85a12-84ee-4c3c-b3dc-122c5430f0e5","Type":"ContainerDied","Data":"cb0ae0d9e184d04fcc206f1e629805a0807cb247d8f7d6a733b251478df78295"} Jul 21 02:21:38.401487 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:38.401393 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 02:21:39.172363 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:39.172335 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7696-139869606534144" Jul 21 02:21:39.193949 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:39.193919 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7696-139869606534144_50d85a12-84ee-4c3c-b3dc-122c5430f0e5/test-curl-7696-139869606534144/0.log" Jul 21 02:21:39.216825 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:39.216792 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7696-139869606534144"] Jul 21 02:21:39.220306 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:39.220281 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7696-139869606534144"] Jul 21 02:21:39.324627 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:39.324526 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-74wl5\" (UniqueName: \"kubernetes.io/projected/50d85a12-84ee-4c3c-b3dc-122c5430f0e5-kube-api-access-74wl5\") pod \"50d85a12-84ee-4c3c-b3dc-122c5430f0e5\" (UID: \"50d85a12-84ee-4c3c-b3dc-122c5430f0e5\") " Jul 21 02:21:39.326594 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:39.326567 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50d85a12-84ee-4c3c-b3dc-122c5430f0e5-kube-api-access-74wl5" (OuterVolumeSpecName: "kube-api-access-74wl5") pod "50d85a12-84ee-4c3c-b3dc-122c5430f0e5" (UID: "50d85a12-84ee-4c3c-b3dc-122c5430f0e5"). InnerVolumeSpecName "kube-api-access-74wl5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 02:21:39.340541 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:39.340511 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7696-139869606531552"] Jul 21 02:21:39.341027 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:39.341010 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50d85a12-84ee-4c3c-b3dc-122c5430f0e5" containerName="test-curl-7696-139869606534144" Jul 21 02:21:39.341125 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:39.341031 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="50d85a12-84ee-4c3c-b3dc-122c5430f0e5" containerName="test-curl-7696-139869606534144" Jul 21 02:21:39.341183 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:39.341137 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="50d85a12-84ee-4c3c-b3dc-122c5430f0e5" containerName="test-curl-7696-139869606534144" Jul 21 02:21:39.343936 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:39.343916 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7696-139869606531552" Jul 21 02:21:39.351378 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:39.351348 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7696-139869606531552"] Jul 21 02:21:39.425722 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:39.425685 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn6mq\" (UniqueName: \"kubernetes.io/projected/153bd56e-338a-4ab7-a0a3-3288a7a21ce2-kube-api-access-tn6mq\") pod \"test-curl-7696-139869606531552\" (UID: \"153bd56e-338a-4ab7-a0a3-3288a7a21ce2\") " pod="opendatahub/test-curl-7696-139869606531552" Jul 21 02:21:39.425908 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:39.425844 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74wl5\" (UniqueName: \"kubernetes.io/projected/50d85a12-84ee-4c3c-b3dc-122c5430f0e5-kube-api-access-74wl5\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 21 02:21:39.526850 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:39.526815 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn6mq\" (UniqueName: \"kubernetes.io/projected/153bd56e-338a-4ab7-a0a3-3288a7a21ce2-kube-api-access-tn6mq\") pod \"test-curl-7696-139869606531552\" (UID: \"153bd56e-338a-4ab7-a0a3-3288a7a21ce2\") " pod="opendatahub/test-curl-7696-139869606531552" Jul 21 02:21:39.534932 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:39.534894 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn6mq\" (UniqueName: \"kubernetes.io/projected/153bd56e-338a-4ab7-a0a3-3288a7a21ce2-kube-api-access-tn6mq\") pod \"test-curl-7696-139869606531552\" (UID: \"153bd56e-338a-4ab7-a0a3-3288a7a21ce2\") " pod="opendatahub/test-curl-7696-139869606531552" Jul 21 02:21:39.656287 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:39.656202 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7696-139869606531552" Jul 21 02:21:39.796607 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:39.796585 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7696-139869606531552"] Jul 21 02:21:39.798865 ip-10-0-139-183 kubenswrapper[2580]: W0721 02:21:39.798834 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod153bd56e_338a_4ab7_a0a3_3288a7a21ce2.slice/crio-528cac9c52abc8c7a06ff8d856c1966fe22d55327c79d26709154cca9d10674b WatchSource:0}: Error finding container 528cac9c52abc8c7a06ff8d856c1966fe22d55327c79d26709154cca9d10674b: Status 404 returned error can't find the container with id 528cac9c52abc8c7a06ff8d856c1966fe22d55327c79d26709154cca9d10674b Jul 21 02:21:40.030745 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:40.030710 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7696-139869606531552" event={"ID":"153bd56e-338a-4ab7-a0a3-3288a7a21ce2","Type":"ContainerStarted","Data":"528cac9c52abc8c7a06ff8d856c1966fe22d55327c79d26709154cca9d10674b"} Jul 21 02:21:40.032016 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:40.031993 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7696-139869606534144" Jul 21 02:21:40.032016 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:40.032005 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10fd18ee7de1488acd59f22a75ee7ae0bd05c7ee96eebb27dd6ca926df654668" Jul 21 02:21:40.042639 ip-10-0-139-183 kubenswrapper[2580]: E0721 02:21:40.042605 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7696-139869606534144\" is forbidden: User \"system:node:ip-10-0-139-183.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-183.ec2.internal' and this object" podUID="50d85a12-84ee-4c3c-b3dc-122c5430f0e5" pod="opendatahub/test-curl-7696-139869606534144" Jul 21 02:21:40.878828 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:40.878794 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50d85a12-84ee-4c3c-b3dc-122c5430f0e5" path="/var/lib/kubelet/pods/50d85a12-84ee-4c3c-b3dc-122c5430f0e5/volumes" Jul 21 02:21:41.038682 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:41.038652 2580 generic.go:356] "Generic (PLEG): container finished" podID="153bd56e-338a-4ab7-a0a3-3288a7a21ce2" containerID="dd05162be6e421161d7c62c2c263cb66c2d29c9d4af06afc2aa2147f5fc9632c" exitCode=0 Jul 21 02:21:41.038682 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:41.038688 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7696-139869606531552" event={"ID":"153bd56e-338a-4ab7-a0a3-3288a7a21ce2","Type":"ContainerDied","Data":"dd05162be6e421161d7c62c2c263cb66c2d29c9d4af06afc2aa2147f5fc9632c"} Jul 21 02:21:42.202023 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:42.201996 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7696-139869606531552" Jul 21 02:21:42.222839 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:42.222810 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7696-139869606531552_153bd56e-338a-4ab7-a0a3-3288a7a21ce2/test-curl-7696-139869606531552/0.log" Jul 21 02:21:42.247913 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:42.247879 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7696-139869606531552"] Jul 21 02:21:42.251422 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:42.251398 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7696-139869606531552"] Jul 21 02:21:42.352344 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:42.352251 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tn6mq\" (UniqueName: \"kubernetes.io/projected/153bd56e-338a-4ab7-a0a3-3288a7a21ce2-kube-api-access-tn6mq\") pod \"153bd56e-338a-4ab7-a0a3-3288a7a21ce2\" (UID: \"153bd56e-338a-4ab7-a0a3-3288a7a21ce2\") " Jul 21 02:21:42.354481 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:42.354438 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/153bd56e-338a-4ab7-a0a3-3288a7a21ce2-kube-api-access-tn6mq" (OuterVolumeSpecName: "kube-api-access-tn6mq") pod "153bd56e-338a-4ab7-a0a3-3288a7a21ce2" (UID: "153bd56e-338a-4ab7-a0a3-3288a7a21ce2"). InnerVolumeSpecName "kube-api-access-tn6mq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 02:21:42.374823 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:42.374797 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7696-139869606531552"] Jul 21 02:21:42.375263 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:42.375250 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="153bd56e-338a-4ab7-a0a3-3288a7a21ce2" containerName="test-curl-7696-139869606531552" Jul 21 02:21:42.375317 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:42.375264 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="153bd56e-338a-4ab7-a0a3-3288a7a21ce2" containerName="test-curl-7696-139869606531552" Jul 21 02:21:42.375353 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:42.375334 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="153bd56e-338a-4ab7-a0a3-3288a7a21ce2" containerName="test-curl-7696-139869606531552" Jul 21 02:21:42.378395 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:42.378378 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7696-139869606531552" Jul 21 02:21:42.382161 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:42.382132 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7696-139869606531552"] Jul 21 02:21:42.453750 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:42.453716 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rt9m\" (UniqueName: \"kubernetes.io/projected/34573feb-9668-4cf0-a569-613f2f504d36-kube-api-access-8rt9m\") pod \"test-curl-7696-139869606531552\" (UID: \"34573feb-9668-4cf0-a569-613f2f504d36\") " pod="opendatahub/test-curl-7696-139869606531552" Jul 21 02:21:42.453904 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:42.453808 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tn6mq\" (UniqueName: \"kubernetes.io/projected/153bd56e-338a-4ab7-a0a3-3288a7a21ce2-kube-api-access-tn6mq\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 21 02:21:42.555171 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:42.555138 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rt9m\" (UniqueName: \"kubernetes.io/projected/34573feb-9668-4cf0-a569-613f2f504d36-kube-api-access-8rt9m\") pod \"test-curl-7696-139869606531552\" (UID: \"34573feb-9668-4cf0-a569-613f2f504d36\") " pod="opendatahub/test-curl-7696-139869606531552" Jul 21 02:21:42.563250 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:42.563218 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rt9m\" (UniqueName: \"kubernetes.io/projected/34573feb-9668-4cf0-a569-613f2f504d36-kube-api-access-8rt9m\") pod \"test-curl-7696-139869606531552\" (UID: \"34573feb-9668-4cf0-a569-613f2f504d36\") " pod="opendatahub/test-curl-7696-139869606531552" Jul 21 02:21:42.691088 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:42.690998 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7696-139869606531552" Jul 21 02:21:42.829883 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:42.829857 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7696-139869606531552"] Jul 21 02:21:42.831432 ip-10-0-139-183 kubenswrapper[2580]: W0721 02:21:42.831403 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod34573feb_9668_4cf0_a569_613f2f504d36.slice/crio-e6437103400c3f5ff3f9e2495d541b54c74c7274c6e0a0e97394c58e52dc6e8b WatchSource:0}: Error finding container e6437103400c3f5ff3f9e2495d541b54c74c7274c6e0a0e97394c58e52dc6e8b: Status 404 returned error can't find the container with id e6437103400c3f5ff3f9e2495d541b54c74c7274c6e0a0e97394c58e52dc6e8b Jul 21 02:21:42.881899 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:42.881868 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="153bd56e-338a-4ab7-a0a3-3288a7a21ce2" path="/var/lib/kubelet/pods/153bd56e-338a-4ab7-a0a3-3288a7a21ce2/volumes" Jul 21 02:21:43.051367 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:43.051319 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7696-139869606531552" event={"ID":"34573feb-9668-4cf0-a569-613f2f504d36","Type":"ContainerStarted","Data":"e6437103400c3f5ff3f9e2495d541b54c74c7274c6e0a0e97394c58e52dc6e8b"} Jul 21 02:21:43.052692 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:43.052673 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7696-139869606531552" Jul 21 02:21:43.052805 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:43.052725 2580 scope.go:117] "RemoveContainer" containerID="dd05162be6e421161d7c62c2c263cb66c2d29c9d4af06afc2aa2147f5fc9632c" Jul 21 02:21:44.061621 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:44.061585 2580 generic.go:356] "Generic (PLEG): container finished" podID="34573feb-9668-4cf0-a569-613f2f504d36" containerID="94dc389c014126ee2ad095745c2ce273f05d98d8e3c067313d492ebb57a4cc45" exitCode=0 Jul 21 02:21:44.061621 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:44.061628 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7696-139869606531552" event={"ID":"34573feb-9668-4cf0-a569-613f2f504d36","Type":"ContainerDied","Data":"94dc389c014126ee2ad095745c2ce273f05d98d8e3c067313d492ebb57a4cc45"} Jul 21 02:21:45.235087 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:45.235059 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7696-139869606531552" Jul 21 02:21:45.255104 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:45.255064 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7696-139869606531552_34573feb-9668-4cf0-a569-613f2f504d36/test-curl-7696-139869606531552/0.log" Jul 21 02:21:45.281865 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:45.281837 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7696-139869606531552"] Jul 21 02:21:45.286492 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:45.286466 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7696-139869606531552"] Jul 21 02:21:45.400080 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:45.399989 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8rt9m\" (UniqueName: \"kubernetes.io/projected/34573feb-9668-4cf0-a569-613f2f504d36-kube-api-access-8rt9m\") pod \"34573feb-9668-4cf0-a569-613f2f504d36\" (UID: \"34573feb-9668-4cf0-a569-613f2f504d36\") " Jul 21 02:21:45.402213 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:45.402192 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34573feb-9668-4cf0-a569-613f2f504d36-kube-api-access-8rt9m" (OuterVolumeSpecName: "kube-api-access-8rt9m") pod "34573feb-9668-4cf0-a569-613f2f504d36" (UID: "34573feb-9668-4cf0-a569-613f2f504d36"). InnerVolumeSpecName "kube-api-access-8rt9m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 02:21:45.501850 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:45.501814 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rt9m\" (UniqueName: \"kubernetes.io/projected/34573feb-9668-4cf0-a569-613f2f504d36-kube-api-access-8rt9m\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 21 02:21:46.074294 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:46.074268 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6437103400c3f5ff3f9e2495d541b54c74c7274c6e0a0e97394c58e52dc6e8b" Jul 21 02:21:46.074507 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:46.074304 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7696-139869606531552" Jul 21 02:21:46.083929 ip-10-0-139-183 kubenswrapper[2580]: E0721 02:21:46.083898 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7696-139869606531552\" is forbidden: User \"system:node:ip-10-0-139-183.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-183.ec2.internal' and this object" podUID="34573feb-9668-4cf0-a569-613f2f504d36" pod="opendatahub/test-curl-7696-139869606531552" Jul 21 02:21:46.875435 ip-10-0-139-183 kubenswrapper[2580]: E0721 02:21:46.875400 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7696-139869606531552\" is forbidden: User \"system:node:ip-10-0-139-183.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-183.ec2.internal' and this object" podUID="34573feb-9668-4cf0-a569-613f2f504d36" pod="opendatahub/test-curl-7696-139869606531552" Jul 21 02:21:46.877394 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:46.877373 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34573feb-9668-4cf0-a569-613f2f504d36" path="/var/lib/kubelet/pods/34573feb-9668-4cf0-a569-613f2f504d36/volumes" Jul 21 02:21:59.299283 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:21:59.299243 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 02:22:18.429329 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:22:18.429282 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c6538a-8484cb76c-w4kz4"] Jul 21 02:22:18.429869 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:22:18.429603 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c6538a-8484cb76c-w4kz4" podUID="33e2b5ba-e528-4cad-903c-3fc6da9674f7" containerName="maas-api" containerID="cri-o://b52ea0cc1b577ca8daad5ee17c70814b7e85b0e8470e9b062d4bb50ccd4d9447" gracePeriod=30 Jul 21 02:22:18.694440 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:22:18.694375 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c6538a-8484cb76c-w4kz4" Jul 21 02:22:18.815081 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:22:18.815046 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sxmgr\" (UniqueName: \"kubernetes.io/projected/33e2b5ba-e528-4cad-903c-3fc6da9674f7-kube-api-access-sxmgr\") pod \"33e2b5ba-e528-4cad-903c-3fc6da9674f7\" (UID: \"33e2b5ba-e528-4cad-903c-3fc6da9674f7\") " Jul 21 02:22:18.815255 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:22:18.815113 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33e2b5ba-e528-4cad-903c-3fc6da9674f7-openshift-service-ca\") pod \"33e2b5ba-e528-4cad-903c-3fc6da9674f7\" (UID: \"33e2b5ba-e528-4cad-903c-3fc6da9674f7\") " Jul 21 02:22:18.815255 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:22:18.815176 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33e2b5ba-e528-4cad-903c-3fc6da9674f7-maas-api-tls\") pod \"33e2b5ba-e528-4cad-903c-3fc6da9674f7\" (UID: \"33e2b5ba-e528-4cad-903c-3fc6da9674f7\") " Jul 21 02:22:18.815498 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:22:18.815443 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33e2b5ba-e528-4cad-903c-3fc6da9674f7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "33e2b5ba-e528-4cad-903c-3fc6da9674f7" (UID: "33e2b5ba-e528-4cad-903c-3fc6da9674f7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 02:22:18.817176 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:22:18.817143 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33e2b5ba-e528-4cad-903c-3fc6da9674f7-kube-api-access-sxmgr" (OuterVolumeSpecName: "kube-api-access-sxmgr") pod "33e2b5ba-e528-4cad-903c-3fc6da9674f7" (UID: "33e2b5ba-e528-4cad-903c-3fc6da9674f7"). InnerVolumeSpecName "kube-api-access-sxmgr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 02:22:18.817263 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:22:18.817215 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33e2b5ba-e528-4cad-903c-3fc6da9674f7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "33e2b5ba-e528-4cad-903c-3fc6da9674f7" (UID: "33e2b5ba-e528-4cad-903c-3fc6da9674f7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 02:22:18.917124 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:22:18.917086 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33e2b5ba-e528-4cad-903c-3fc6da9674f7-openshift-service-ca\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 21 02:22:18.917124 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:22:18.917115 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33e2b5ba-e528-4cad-903c-3fc6da9674f7-maas-api-tls\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 21 02:22:18.917124 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:22:18.917128 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxmgr\" (UniqueName: \"kubernetes.io/projected/33e2b5ba-e528-4cad-903c-3fc6da9674f7-kube-api-access-sxmgr\") on node \"ip-10-0-139-183.ec2.internal\" DevicePath \"\"" Jul 21 02:22:19.260346 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:22:19.260309 2580 generic.go:356] "Generic (PLEG): container finished" podID="33e2b5ba-e528-4cad-903c-3fc6da9674f7" containerID="b52ea0cc1b577ca8daad5ee17c70814b7e85b0e8470e9b062d4bb50ccd4d9447" exitCode=0 Jul 21 02:22:19.260550 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:22:19.260353 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c6538a-8484cb76c-w4kz4" Jul 21 02:22:19.260550 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:22:19.260394 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c6538a-8484cb76c-w4kz4" event={"ID":"33e2b5ba-e528-4cad-903c-3fc6da9674f7","Type":"ContainerDied","Data":"b52ea0cc1b577ca8daad5ee17c70814b7e85b0e8470e9b062d4bb50ccd4d9447"} Jul 21 02:22:19.260550 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:22:19.260476 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c6538a-8484cb76c-w4kz4" event={"ID":"33e2b5ba-e528-4cad-903c-3fc6da9674f7","Type":"ContainerDied","Data":"7a08467ec0dc70d1f36e66ba0017ea98df223d0a4ab2d096469f384b83cecaf9"} Jul 21 02:22:19.260550 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:22:19.260508 2580 scope.go:117] "RemoveContainer" containerID="b52ea0cc1b577ca8daad5ee17c70814b7e85b0e8470e9b062d4bb50ccd4d9447" Jul 21 02:22:19.269767 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:22:19.269746 2580 scope.go:117] "RemoveContainer" containerID="b52ea0cc1b577ca8daad5ee17c70814b7e85b0e8470e9b062d4bb50ccd4d9447" Jul 21 02:22:19.270053 ip-10-0-139-183 kubenswrapper[2580]: E0721 02:22:19.270032 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b52ea0cc1b577ca8daad5ee17c70814b7e85b0e8470e9b062d4bb50ccd4d9447\": container with ID starting with b52ea0cc1b577ca8daad5ee17c70814b7e85b0e8470e9b062d4bb50ccd4d9447 not found: ID does not exist" containerID="b52ea0cc1b577ca8daad5ee17c70814b7e85b0e8470e9b062d4bb50ccd4d9447" Jul 21 02:22:19.270148 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:22:19.270062 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b52ea0cc1b577ca8daad5ee17c70814b7e85b0e8470e9b062d4bb50ccd4d9447"} err="failed to get container status \"b52ea0cc1b577ca8daad5ee17c70814b7e85b0e8470e9b062d4bb50ccd4d9447\": rpc error: code = NotFound desc = could not find container \"b52ea0cc1b577ca8daad5ee17c70814b7e85b0e8470e9b062d4bb50ccd4d9447\": container with ID starting with b52ea0cc1b577ca8daad5ee17c70814b7e85b0e8470e9b062d4bb50ccd4d9447 not found: ID does not exist" Jul 21 02:22:19.275995 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:22:19.275974 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c6538a-8484cb76c-w4kz4"] Jul 21 02:22:19.279348 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:22:19.279323 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c6538a-8484cb76c-w4kz4"] Jul 21 02:22:20.878237 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:22:20.878202 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33e2b5ba-e528-4cad-903c-3fc6da9674f7" path="/var/lib/kubelet/pods/33e2b5ba-e528-4cad-903c-3fc6da9674f7/volumes" Jul 21 02:22:21.593350 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:22:21.593314 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4glr6"] Jul 21 02:22:35.883307 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:22:35.883278 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-pkkpq_e604ba86-e513-4b2c-88e4-0e7b90974b72/manager/0.log" Jul 21 02:22:36.003113 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:22:36.003081 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-pgxz4_a5b32c0e-f65a-48ba-a752-1b78cb1950e3/manager/0.log" Jul 21 02:22:36.130067 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:22:36.130036 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5cd8fcb89-nf9zb_3c2ad5c2-e85c-49c3-8e07-adfec292999c/manager/0.log" Jul 21 02:22:36.244660 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:22:36.244581 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-8dt74_e079cc2b-d540-422b-a89c-f59f6c812680/server/0.log" Jul 21 02:22:36.362976 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:22:36.362951 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-bb4gc_6a7381ef-2c27-451a-9516-fab7f9c42e87/manager/0.log" Jul 21 02:22:36.479007 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:22:36.478981 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7d7c89d5b6-722r9_60d07d21-eb68-49ee-b399-5e1acb9b11cd/copy-manifests/0.log" Jul 21 02:22:36.493593 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:22:36.493560 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7d7c89d5b6-722r9_60d07d21-eb68-49ee-b399-5e1acb9b11cd/manager/0.log" Jul 21 02:22:38.979416 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:22:38.979337 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-kxb76_270a391d-e4cd-4d02-8b67-66f6c9715e8e/manager/0.log" Jul 21 02:22:39.101211 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:22:39.101183 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-7gfwm_3f0675f5-b65c-4a80-acda-d1744325c0e9/manager/0.log" Jul 21 02:22:39.218270 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:22:39.218240 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-xtgcn_0a12d9c8-e70f-4f03-8465-926cdc4645f5/kuadrant-console-plugin/0.log" Jul 21 02:22:39.340946 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:22:39.340911 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-r79j5_e8374b93-553a-4491-9eb4-e8282bb527b1/registry-server/0.log" Jul 21 02:22:39.468734 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:22:39.468703 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-8j6sk_0b10c41e-3302-47f3-8b6b-4057c89a130e/manager/0.log" Jul 21 02:22:39.592389 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:22:39.592306 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-4glr6_81f38466-10f7-408f-ad65-44bdbb3b9e91/limitador/0.log" Jul 21 02:22:40.469387 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:22:40.469356 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5cc474c686-6vvdm_f478270d-44e8-4bb0-bec0-fda93118abb3/kube-auth-proxy/0.log" Jul 21 02:22:40.594606 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:22:40.594579 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-5zsqr_b3cd4166-77e6-4b24-ae7d-b0d823eba80d/istio-proxy/1.log" Jul 21 02:22:40.720552 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:22:40.720474 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-frgrf_7d04fe9b-af81-49cd-91c3-ed777f12282a/payload-pre-processing/0.log" Jul 21 02:22:40.863535 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:22:40.863507 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-bxgxv_9775328a-6f07-4ded-b3a0-1484ed0969b6/payload-processing/0.log" Jul 21 02:22:41.004822 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:22:41.004795 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6f7df7d7cb-gcx5w_e6d9baf0-743b-48f5-9374-4443c65ea048/router/0.log" Jul 21 02:22:41.369333 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:22:41.369252 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-z7r2c_098b3243-a8a5-4ee6-8e82-2e7b27ac6671/main/0.log" Jul 21 02:22:41.995557 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:22:41.995522 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-qvwzz_c02cf82d-8313-4edb-9f25-e7a93dabecba/main/0.log" Jul 21 02:22:48.138058 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:22:48.138022 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-gfbpz/must-gather-59tnj"] Jul 21 02:22:48.138534 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:22:48.138510 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33e2b5ba-e528-4cad-903c-3fc6da9674f7" containerName="maas-api" Jul 21 02:22:48.138534 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:22:48.138523 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="33e2b5ba-e528-4cad-903c-3fc6da9674f7" containerName="maas-api" Jul 21 02:22:48.138628 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:22:48.138541 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34573feb-9668-4cf0-a569-613f2f504d36" containerName="test-curl-7696-139869606531552" Jul 21 02:22:48.138628 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:22:48.138548 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="34573feb-9668-4cf0-a569-613f2f504d36" containerName="test-curl-7696-139869606531552" Jul 21 02:22:48.138628 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:22:48.138621 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="34573feb-9668-4cf0-a569-613f2f504d36" containerName="test-curl-7696-139869606531552" Jul 21 02:22:48.138738 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:22:48.138631 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="33e2b5ba-e528-4cad-903c-3fc6da9674f7" containerName="maas-api" Jul 21 02:22:48.142021 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:22:48.142005 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gfbpz/must-gather-59tnj" Jul 21 02:22:48.149028 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:22:48.149010 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-gfbpz\"/\"kube-root-ca.crt\"" Jul 21 02:22:48.149500 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:22:48.149483 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-gfbpz\"/\"openshift-service-ca.crt\"" Jul 21 02:22:48.149555 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:22:48.149483 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-gfbpz\"/\"default-dockercfg-blw8z\"" Jul 21 02:22:48.160210 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:22:48.160190 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gfbpz/must-gather-59tnj"] Jul 21 02:22:48.203404 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:22:48.203367 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/eb6d1e06-13a4-44db-a4e5-61d765515ab2-must-gather-output\") pod \"must-gather-59tnj\" (UID: \"eb6d1e06-13a4-44db-a4e5-61d765515ab2\") " pod="openshift-must-gather-gfbpz/must-gather-59tnj" Jul 21 02:22:48.203578 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:22:48.203422 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrtrs\" (UniqueName: \"kubernetes.io/projected/eb6d1e06-13a4-44db-a4e5-61d765515ab2-kube-api-access-hrtrs\") pod \"must-gather-59tnj\" (UID: \"eb6d1e06-13a4-44db-a4e5-61d765515ab2\") " pod="openshift-must-gather-gfbpz/must-gather-59tnj" Jul 21 02:22:48.304643 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:22:48.304596 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/eb6d1e06-13a4-44db-a4e5-61d765515ab2-must-gather-output\") pod \"must-gather-59tnj\" (UID: \"eb6d1e06-13a4-44db-a4e5-61d765515ab2\") " pod="openshift-must-gather-gfbpz/must-gather-59tnj" Jul 21 02:22:48.304823 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:22:48.304664 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrtrs\" (UniqueName: \"kubernetes.io/projected/eb6d1e06-13a4-44db-a4e5-61d765515ab2-kube-api-access-hrtrs\") pod \"must-gather-59tnj\" (UID: \"eb6d1e06-13a4-44db-a4e5-61d765515ab2\") " pod="openshift-must-gather-gfbpz/must-gather-59tnj" Jul 21 02:22:48.304947 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:22:48.304926 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/eb6d1e06-13a4-44db-a4e5-61d765515ab2-must-gather-output\") pod \"must-gather-59tnj\" (UID: \"eb6d1e06-13a4-44db-a4e5-61d765515ab2\") " pod="openshift-must-gather-gfbpz/must-gather-59tnj" Jul 21 02:22:48.315796 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:22:48.315766 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrtrs\" (UniqueName: \"kubernetes.io/projected/eb6d1e06-13a4-44db-a4e5-61d765515ab2-kube-api-access-hrtrs\") pod \"must-gather-59tnj\" (UID: \"eb6d1e06-13a4-44db-a4e5-61d765515ab2\") " pod="openshift-must-gather-gfbpz/must-gather-59tnj" Jul 21 02:22:48.451169 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:22:48.451088 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gfbpz/must-gather-59tnj" Jul 21 02:22:48.591118 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:22:48.591094 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gfbpz/must-gather-59tnj"] Jul 21 02:22:48.593043 ip-10-0-139-183 kubenswrapper[2580]: W0721 02:22:48.593017 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeb6d1e06_13a4_44db_a4e5_61d765515ab2.slice/crio-0adec4561fe04e2761775cd7fe8fa82758562bc11e64bc838ac66b3f1988278b WatchSource:0}: Error finding container 0adec4561fe04e2761775cd7fe8fa82758562bc11e64bc838ac66b3f1988278b: Status 404 returned error can't find the container with id 0adec4561fe04e2761775cd7fe8fa82758562bc11e64bc838ac66b3f1988278b Jul 21 02:22:49.425492 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:22:49.425433 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gfbpz/must-gather-59tnj" event={"ID":"eb6d1e06-13a4-44db-a4e5-61d765515ab2","Type":"ContainerStarted","Data":"0adec4561fe04e2761775cd7fe8fa82758562bc11e64bc838ac66b3f1988278b"} Jul 21 02:22:50.437352 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:22:50.436427 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gfbpz/must-gather-59tnj" event={"ID":"eb6d1e06-13a4-44db-a4e5-61d765515ab2","Type":"ContainerStarted","Data":"6dbcb43444df97027d936f355fa82fe32fa17ae25af7e2ebc896e974a41d55a1"} Jul 21 02:22:50.437352 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:22:50.436511 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gfbpz/must-gather-59tnj" event={"ID":"eb6d1e06-13a4-44db-a4e5-61d765515ab2","Type":"ContainerStarted","Data":"bc961adca0f00a6cecf0dee635edaf64e87deaeaff7df46befe91f6708c7bb60"} Jul 21 02:22:50.455725 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:22:50.455672 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-gfbpz/must-gather-59tnj" podStartSLOduration=1.6724188660000001 podStartE2EDuration="2.455654131s" totalImagesPullingTime="783.235265ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 02:22:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 02:22:50.455522112 +0000 UTC m=+2794.042688092" watchObservedRunningTime="2026-07-21 02:22:50.455654131 +0000 UTC m=+2794.042820138" Jul 21 02:22:50.984308 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:22:50.984277 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-24nj2_17bdceb8-8016-4d0b-8b3c-f36507279428/global-pull-secret-syncer/0.log" Jul 21 02:22:51.226688 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:22:51.226655 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-smxdt_e548f5b6-6507-4431-b027-88917b43191f/konnectivity-agent/0.log" Jul 21 02:22:51.309378 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:22:51.309345 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-183.ec2.internal_fb35f72ee7f4b217fd778d25ec504876/haproxy/0.log" Jul 21 02:22:56.738152 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:22:56.738116 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-kxb76_270a391d-e4cd-4d02-8b67-66f6c9715e8e/manager/0.log" Jul 21 02:22:56.766274 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:22:56.766237 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-7gfwm_3f0675f5-b65c-4a80-acda-d1744325c0e9/manager/0.log" Jul 21 02:22:56.799975 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:22:56.799940 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-xtgcn_0a12d9c8-e70f-4f03-8465-926cdc4645f5/kuadrant-console-plugin/0.log" Jul 21 02:22:56.858410 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:22:56.858371 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-r79j5_e8374b93-553a-4491-9eb4-e8282bb527b1/registry-server/0.log" Jul 21 02:22:57.000368 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:22:56.998442 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-8j6sk_0b10c41e-3302-47f3-8b6b-4057c89a130e/manager/0.log" Jul 21 02:22:57.064352 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:22:57.064318 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-4glr6_81f38466-10f7-408f-ad65-44bdbb3b9e91/limitador/0.log" Jul 21 02:22:59.053653 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:22:59.053612 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-9ctn2_1e930d67-d711-419c-af02-86b43027a95d/cluster-monitoring-operator/1.log" Jul 21 02:22:59.168327 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:22:59.168288 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-9ctn2_1e930d67-d711-419c-af02-86b43027a95d/cluster-monitoring-operator/0.log" Jul 21 02:22:59.199841 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:22:59.199741 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-7gzg9_016c43a1-1619-4b20-aa34-37f9a61b0016/kube-state-metrics/0.log" Jul 21 02:22:59.226895 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:22:59.226859 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-7gzg9_016c43a1-1619-4b20-aa34-37f9a61b0016/kube-rbac-proxy-main/0.log" Jul 21 02:22:59.256216 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:22:59.256188 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-7gzg9_016c43a1-1619-4b20-aa34-37f9a61b0016/kube-rbac-proxy-self/0.log" Jul 21 02:22:59.289732 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:22:59.289705 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6b98d6767c-27dq7_7244033a-1bcd-4621-9e12-7cead4e87eea/metrics-server/0.log" Jul 21 02:22:59.320466 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:22:59.320361 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-7b7b7bc8bf-68gpx_d04afbe3-2081-44fc-985b-2ca888bfb6f3/monitoring-plugin/0.log" Jul 21 02:22:59.560560 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:22:59.560529 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7vvrs_6484e153-4568-44ee-ba8d-4d4b13b5defd/node-exporter/0.log" Jul 21 02:22:59.583437 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:22:59.583359 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7vvrs_6484e153-4568-44ee-ba8d-4d4b13b5defd/kube-rbac-proxy/0.log" Jul 21 02:22:59.609397 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:22:59.609372 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7vvrs_6484e153-4568-44ee-ba8d-4d4b13b5defd/init-textfile/0.log" Jul 21 02:23:00.003086 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:23:00.003048 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-d976688db-wk24w_7312b0ab-bfff-4bce-976e-ba6e6167ba3e/telemeter-client/0.log" Jul 21 02:23:00.033325 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:23:00.033301 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-d976688db-wk24w_7312b0ab-bfff-4bce-976e-ba6e6167ba3e/reload/0.log" Jul 21 02:23:00.060622 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:23:00.060593 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-d976688db-wk24w_7312b0ab-bfff-4bce-976e-ba6e6167ba3e/kube-rbac-proxy/0.log" Jul 21 02:23:00.099208 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:23:00.099168 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7655d55f5f-rhfvc_0b59c9f0-3074-40cb-9188-14225a950785/thanos-query/0.log" Jul 21 02:23:00.132163 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:23:00.132123 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7655d55f5f-rhfvc_0b59c9f0-3074-40cb-9188-14225a950785/kube-rbac-proxy-web/0.log" Jul 21 02:23:00.156641 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:23:00.156610 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7655d55f5f-rhfvc_0b59c9f0-3074-40cb-9188-14225a950785/kube-rbac-proxy/0.log" Jul 21 02:23:00.185464 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:23:00.185425 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7655d55f5f-rhfvc_0b59c9f0-3074-40cb-9188-14225a950785/prom-label-proxy/0.log" Jul 21 02:23:00.213685 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:23:00.213651 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7655d55f5f-rhfvc_0b59c9f0-3074-40cb-9188-14225a950785/kube-rbac-proxy-rules/0.log" Jul 21 02:23:00.239054 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:23:00.239025 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7655d55f5f-rhfvc_0b59c9f0-3074-40cb-9188-14225a950785/kube-rbac-proxy-metrics/0.log" Jul 21 02:23:01.230646 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:23:01.230607 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-gfbpz/perf-node-gather-daemonset-kv4dj"] Jul 21 02:23:01.236874 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:23:01.236834 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gfbpz/perf-node-gather-daemonset-kv4dj" Jul 21 02:23:01.244477 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:23:01.244419 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gfbpz/perf-node-gather-daemonset-kv4dj"] Jul 21 02:23:01.359106 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:23:01.359062 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/608c0fbb-e662-4141-8616-2b292ac8e262-sys\") pod \"perf-node-gather-daemonset-kv4dj\" (UID: \"608c0fbb-e662-4141-8616-2b292ac8e262\") " pod="openshift-must-gather-gfbpz/perf-node-gather-daemonset-kv4dj" Jul 21 02:23:01.359106 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:23:01.359107 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/608c0fbb-e662-4141-8616-2b292ac8e262-podres\") pod \"perf-node-gather-daemonset-kv4dj\" (UID: \"608c0fbb-e662-4141-8616-2b292ac8e262\") " pod="openshift-must-gather-gfbpz/perf-node-gather-daemonset-kv4dj" Jul 21 02:23:01.359352 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:23:01.359228 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/608c0fbb-e662-4141-8616-2b292ac8e262-lib-modules\") pod \"perf-node-gather-daemonset-kv4dj\" (UID: \"608c0fbb-e662-4141-8616-2b292ac8e262\") " pod="openshift-must-gather-gfbpz/perf-node-gather-daemonset-kv4dj" Jul 21 02:23:01.359352 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:23:01.359268 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/608c0fbb-e662-4141-8616-2b292ac8e262-proc\") pod \"perf-node-gather-daemonset-kv4dj\" (UID: \"608c0fbb-e662-4141-8616-2b292ac8e262\") " pod="openshift-must-gather-gfbpz/perf-node-gather-daemonset-kv4dj" Jul 21 02:23:01.359352 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:23:01.359306 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jspl\" (UniqueName: \"kubernetes.io/projected/608c0fbb-e662-4141-8616-2b292ac8e262-kube-api-access-9jspl\") pod \"perf-node-gather-daemonset-kv4dj\" (UID: \"608c0fbb-e662-4141-8616-2b292ac8e262\") " pod="openshift-must-gather-gfbpz/perf-node-gather-daemonset-kv4dj" Jul 21 02:23:01.444917 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:23:01.444886 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-7bc7489cb5-5gfm8_15628691-46c2-418e-8eca-47f963849e13/networking-console-plugin/0.log" Jul 21 02:23:01.460007 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:23:01.459976 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/608c0fbb-e662-4141-8616-2b292ac8e262-proc\") pod \"perf-node-gather-daemonset-kv4dj\" (UID: \"608c0fbb-e662-4141-8616-2b292ac8e262\") " pod="openshift-must-gather-gfbpz/perf-node-gather-daemonset-kv4dj" Jul 21 02:23:01.460209 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:23:01.460030 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jspl\" (UniqueName: \"kubernetes.io/projected/608c0fbb-e662-4141-8616-2b292ac8e262-kube-api-access-9jspl\") pod \"perf-node-gather-daemonset-kv4dj\" (UID: \"608c0fbb-e662-4141-8616-2b292ac8e262\") " pod="openshift-must-gather-gfbpz/perf-node-gather-daemonset-kv4dj" Jul 21 02:23:01.460209 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:23:01.460090 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/608c0fbb-e662-4141-8616-2b292ac8e262-sys\") pod \"perf-node-gather-daemonset-kv4dj\" (UID: \"608c0fbb-e662-4141-8616-2b292ac8e262\") " pod="openshift-must-gather-gfbpz/perf-node-gather-daemonset-kv4dj" Jul 21 02:23:01.460209 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:23:01.460118 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/608c0fbb-e662-4141-8616-2b292ac8e262-podres\") pod \"perf-node-gather-daemonset-kv4dj\" (UID: \"608c0fbb-e662-4141-8616-2b292ac8e262\") " pod="openshift-must-gather-gfbpz/perf-node-gather-daemonset-kv4dj" Jul 21 02:23:01.460209 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:23:01.460133 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/608c0fbb-e662-4141-8616-2b292ac8e262-proc\") pod \"perf-node-gather-daemonset-kv4dj\" (UID: \"608c0fbb-e662-4141-8616-2b292ac8e262\") " pod="openshift-must-gather-gfbpz/perf-node-gather-daemonset-kv4dj" Jul 21 02:23:01.460410 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:23:01.460211 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/608c0fbb-e662-4141-8616-2b292ac8e262-sys\") pod \"perf-node-gather-daemonset-kv4dj\" (UID: \"608c0fbb-e662-4141-8616-2b292ac8e262\") " pod="openshift-must-gather-gfbpz/perf-node-gather-daemonset-kv4dj" Jul 21 02:23:01.460410 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:23:01.460225 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/608c0fbb-e662-4141-8616-2b292ac8e262-lib-modules\") pod \"perf-node-gather-daemonset-kv4dj\" (UID: \"608c0fbb-e662-4141-8616-2b292ac8e262\") " pod="openshift-must-gather-gfbpz/perf-node-gather-daemonset-kv4dj" Jul 21 02:23:01.460410 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:23:01.460316 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/608c0fbb-e662-4141-8616-2b292ac8e262-podres\") pod \"perf-node-gather-daemonset-kv4dj\" (UID: \"608c0fbb-e662-4141-8616-2b292ac8e262\") " pod="openshift-must-gather-gfbpz/perf-node-gather-daemonset-kv4dj" Jul 21 02:23:01.460410 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:23:01.460336 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/608c0fbb-e662-4141-8616-2b292ac8e262-lib-modules\") pod \"perf-node-gather-daemonset-kv4dj\" (UID: \"608c0fbb-e662-4141-8616-2b292ac8e262\") " pod="openshift-must-gather-gfbpz/perf-node-gather-daemonset-kv4dj" Jul 21 02:23:01.468438 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:23:01.468407 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jspl\" (UniqueName: \"kubernetes.io/projected/608c0fbb-e662-4141-8616-2b292ac8e262-kube-api-access-9jspl\") pod \"perf-node-gather-daemonset-kv4dj\" (UID: \"608c0fbb-e662-4141-8616-2b292ac8e262\") " pod="openshift-must-gather-gfbpz/perf-node-gather-daemonset-kv4dj" Jul 21 02:23:01.551754 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:23:01.551699 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gfbpz/perf-node-gather-daemonset-kv4dj" Jul 21 02:23:01.755821 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:23:01.755471 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gfbpz/perf-node-gather-daemonset-kv4dj"] Jul 21 02:23:01.757385 ip-10-0-139-183 kubenswrapper[2580]: W0721 02:23:01.757317 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod608c0fbb_e662_4141_8616_2b292ac8e262.slice/crio-7dc40a3cf19966ba187396474df34e5162567701af0b2569ba0de5a0c5e04295 WatchSource:0}: Error finding container 7dc40a3cf19966ba187396474df34e5162567701af0b2569ba0de5a0c5e04295: Status 404 returned error can't find the container with id 7dc40a3cf19966ba187396474df34e5162567701af0b2569ba0de5a0c5e04295 Jul 21 02:23:02.082407 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:23:02.082320 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-4mtp7_9c380009-9bf5-4b73-803a-5a1ea9c1abf0/console-operator/1.log" Jul 21 02:23:02.088912 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:23:02.088875 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-4mtp7_9c380009-9bf5-4b73-803a-5a1ea9c1abf0/console-operator/2.log" Jul 21 02:23:02.537249 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:23:02.537202 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gfbpz/perf-node-gather-daemonset-kv4dj" event={"ID":"608c0fbb-e662-4141-8616-2b292ac8e262","Type":"ContainerStarted","Data":"07aa551afd06ec1d70192743f963fd36352fbbc7593130628c3f9dfa524ab367"} Jul 21 02:23:02.537645 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:23:02.537273 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-gfbpz/perf-node-gather-daemonset-kv4dj" Jul 21 02:23:02.537645 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:23:02.537298 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gfbpz/perf-node-gather-daemonset-kv4dj" event={"ID":"608c0fbb-e662-4141-8616-2b292ac8e262","Type":"ContainerStarted","Data":"7dc40a3cf19966ba187396474df34e5162567701af0b2569ba0de5a0c5e04295"} Jul 21 02:23:02.553350 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:23:02.553300 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-gfbpz/perf-node-gather-daemonset-kv4dj" podStartSLOduration=1.553284104 podStartE2EDuration="1.553284104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 02:23:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 02:23:02.552160627 +0000 UTC m=+2806.139326627" watchObservedRunningTime="2026-07-21 02:23:02.553284104 +0000 UTC m=+2806.140450082" Jul 21 02:23:02.556533 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:23:02.556506 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-8fdf8bfcb-qdwmd_01a8b910-97b0-4384-a18e-bf3188092b4b/console/0.log" Jul 21 02:23:03.190105 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:23:03.190072 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78cf4c6f6-vxvqz_ea46ff20-2a04-4b13-92ed-371a4eb06987/volume-data-source-validator/0.log" Jul 21 02:23:04.348311 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:23:04.348280 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-v899s_0add4762-9fef-4072-a05a-cdca30f4cdcf/dns/0.log" Jul 21 02:23:04.373967 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:23:04.373938 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-v899s_0add4762-9fef-4072-a05a-cdca30f4cdcf/kube-rbac-proxy/0.log" Jul 21 02:23:04.402592 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:23:04.402554 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5n4nl_a1ca57f9-11f8-48c3-9ff6-a7b2561fede0/dns-node-resolver/0.log" Jul 21 02:23:05.118495 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:23:05.118443 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zxcjm_1489d404-17a6-4bf1-9a85-9d3787aece54/node-ca/0.log" Jul 21 02:23:06.270318 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:23:06.270292 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5cc474c686-6vvdm_f478270d-44e8-4bb0-bec0-fda93118abb3/kube-auth-proxy/0.log" Jul 21 02:23:06.304660 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:23:06.304625 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-5zsqr_b3cd4166-77e6-4b24-ae7d-b0d823eba80d/istio-proxy/0.log" Jul 21 02:23:06.333648 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:23:06.333616 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-5zsqr_b3cd4166-77e6-4b24-ae7d-b0d823eba80d/istio-proxy/1.log" Jul 21 02:23:06.367646 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:23:06.367620 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-frgrf_7d04fe9b-af81-49cd-91c3-ed777f12282a/payload-pre-processing/0.log" Jul 21 02:23:06.399729 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:23:06.399688 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-bxgxv_9775328a-6f07-4ded-b3a0-1484ed0969b6/payload-processing/0.log" Jul 21 02:23:06.425083 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:23:06.425053 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6f7df7d7cb-gcx5w_e6d9baf0-743b-48f5-9374-4443c65ea048/router/0.log" Jul 21 02:23:07.173131 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:23:07.173106 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bmgfc_1d402931-b366-48df-acda-2e144e7c6c70/serve-healthcheck-canary/0.log" Jul 21 02:23:07.688964 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:23:07.688929 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-wnjpr_d67dd362-e3a6-4ad1-9f9f-8f628ffce9a2/insights-operator/0.log" Jul 21 02:23:07.690293 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:23:07.690272 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-wnjpr_d67dd362-e3a6-4ad1-9f9f-8f628ffce9a2/insights-operator/1.log" Jul 21 02:23:07.890347 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:23:07.890310 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mqxgn_e0343507-90fb-4245-bd91-73476ed1a05b/kube-rbac-proxy/0.log" Jul 21 02:23:07.919866 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:23:07.919838 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mqxgn_e0343507-90fb-4245-bd91-73476ed1a05b/exporter/0.log" Jul 21 02:23:07.944280 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:23:07.944206 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mqxgn_e0343507-90fb-4245-bd91-73476ed1a05b/extractor/0.log" Jul 21 02:23:08.559381 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:23:08.559343 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-gfbpz/perf-node-gather-daemonset-kv4dj" Jul 21 02:23:10.212813 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:23:10.212714 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-pkkpq_e604ba86-e513-4b2c-88e4-0e7b90974b72/manager/0.log" Jul 21 02:23:10.480215 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:23:10.480123 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-pgxz4_a5b32c0e-f65a-48ba-a752-1b78cb1950e3/manager/0.log" Jul 21 02:23:10.634704 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:23:10.634671 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5cd8fcb89-nf9zb_3c2ad5c2-e85c-49c3-8e07-adfec292999c/manager/0.log" Jul 21 02:23:10.657066 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:23:10.657022 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-8dt74_e079cc2b-d540-422b-a89c-f59f6c812680/server/0.log" Jul 21 02:23:10.769619 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:23:10.769564 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-bb4gc_6a7381ef-2c27-451a-9516-fab7f9c42e87/manager/0.log" Jul 21 02:23:10.865794 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:23:10.865121 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7d7c89d5b6-722r9_60d07d21-eb68-49ee-b399-5e1acb9b11cd/manager/0.log" Jul 21 02:23:10.883962 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:23:10.883892 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7d7c89d5b6-722r9_60d07d21-eb68-49ee-b399-5e1acb9b11cd/copy-manifests/0.log" Jul 21 02:23:12.157276 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:23:12.157199 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5cc474c686-6vvdm_f478270d-44e8-4bb0-bec0-fda93118abb3/kube-auth-proxy/0.log" Jul 21 02:23:12.181894 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:23:12.181864 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-5zsqr_b3cd4166-77e6-4b24-ae7d-b0d823eba80d/istio-proxy/0.log" Jul 21 02:23:12.216898 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:23:12.216870 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-5zsqr_b3cd4166-77e6-4b24-ae7d-b0d823eba80d/istio-proxy/1.log" Jul 21 02:23:12.243976 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:23:12.243949 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-frgrf_7d04fe9b-af81-49cd-91c3-ed777f12282a/payload-pre-processing/0.log" Jul 21 02:23:12.268190 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:23:12.268160 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-bxgxv_9775328a-6f07-4ded-b3a0-1484ed0969b6/payload-processing/0.log" Jul 21 02:23:12.284323 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:23:12.284299 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6f7df7d7cb-gcx5w_e6d9baf0-743b-48f5-9374-4443c65ea048/router/0.log" Jul 21 02:23:13.701397 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:23:13.701311 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5cc474c686-6vvdm_f478270d-44e8-4bb0-bec0-fda93118abb3/kube-auth-proxy/0.log" Jul 21 02:23:13.732318 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:23:13.732290 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-5zsqr_b3cd4166-77e6-4b24-ae7d-b0d823eba80d/istio-proxy/0.log" Jul 21 02:23:13.765468 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:23:13.765424 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-5zsqr_b3cd4166-77e6-4b24-ae7d-b0d823eba80d/istio-proxy/1.log" Jul 21 02:23:13.789665 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:23:13.789634 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-frgrf_7d04fe9b-af81-49cd-91c3-ed777f12282a/payload-pre-processing/0.log" Jul 21 02:23:13.813032 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:23:13.812998 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-bxgxv_9775328a-6f07-4ded-b3a0-1484ed0969b6/payload-processing/0.log" Jul 21 02:23:13.826405 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:23:13.826379 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6f7df7d7cb-gcx5w_e6d9baf0-743b-48f5-9374-4443c65ea048/router/0.log" Jul 21 02:23:17.858319 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:23:17.858284 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-9h5pm_6e08e8c3-dd3e-4792-8096-cf150dbcb58b/kube-storage-version-migrator-operator/1.log" Jul 21 02:23:17.859320 ip-10-0-139-183 kubenswrapper[2580]: I0721 02:23:17.859300 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-9h5pm_6e08e8c3-dd3e-4792-8096-cf150dbcb58b/kube-storage-version-migrator-operator/0.log"