Jul 21 00:39:04.984736 ip-10-0-129-137 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 21 00:39:04.984751 ip-10-0-129-137 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 21 00:39:04.984760 ip-10-0-129-137 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 21 00:39:04.985420 ip-10-0-129-137 systemd[1]: Failed to start Kubernetes Kubelet. Jul 21 00:39:15.214687 ip-10-0-129-137 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 21 00:39:15.214702 ip-10-0-129-137 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot ce697dd873c7497a8116a68b1274c039 -- Jul 21 00:40:52.258509 ip-10-0-129-137 systemd[1]: Starting Kubernetes Kubelet... Jul 21 00:40:52.678133 ip-10-0-129-137 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 21 00:40:52.678133 ip-10-0-129-137 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 21 00:40:52.678133 ip-10-0-129-137 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 21 00:40:52.678133 ip-10-0-129-137 kubenswrapper[2580]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 21 00:40:52.678133 ip-10-0-129-137 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 21 00:40:52.678133 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.640910 2580 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 21 00:40:52.678133 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647459 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 00:40:52.678133 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647556 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 00:40:52.678133 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647560 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 00:40:52.678133 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647564 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 00:40:52.678133 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647567 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 00:40:52.678133 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647569 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 00:40:52.678133 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647572 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 00:40:52.678133 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647577 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 00:40:52.680194 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647581 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 00:40:52.680194 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647584 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 00:40:52.680194 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647587 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 00:40:52.680194 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647590 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 00:40:52.680194 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647594 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 00:40:52.680194 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647598 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 00:40:52.680194 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647601 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 00:40:52.680194 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647604 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 00:40:52.680194 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647607 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 00:40:52.680194 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647609 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 00:40:52.680194 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647612 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 00:40:52.680194 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647615 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 00:40:52.680194 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647617 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 00:40:52.680194 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647620 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 00:40:52.680194 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647622 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 00:40:52.680194 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647625 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 00:40:52.680194 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647628 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 00:40:52.680194 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647630 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 00:40:52.680194 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647633 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 00:40:52.680194 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647635 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 00:40:52.699573 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647638 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 00:40:52.699573 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647641 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 00:40:52.699573 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647644 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 00:40:52.699573 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647647 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 21 00:40:52.699573 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647650 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 00:40:52.699573 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647657 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 00:40:52.699573 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647660 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 00:40:52.699573 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647662 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 00:40:52.699573 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647665 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 00:40:52.699573 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647667 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 00:40:52.699573 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647672 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 00:40:52.699573 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647675 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 00:40:52.699573 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647678 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 00:40:52.699573 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647680 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 00:40:52.699573 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647683 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 00:40:52.699573 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647685 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 00:40:52.699573 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647688 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 00:40:52.699573 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647691 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 00:40:52.699573 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647693 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 00:40:52.699573 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647696 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 00:40:52.734093 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647698 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 00:40:52.734093 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647701 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 00:40:52.734093 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647703 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 00:40:52.734093 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647706 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 00:40:52.734093 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647708 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 00:40:52.734093 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647711 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 00:40:52.734093 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647713 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 00:40:52.734093 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647716 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 00:40:52.734093 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647720 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 00:40:52.734093 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647722 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 00:40:52.734093 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647725 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 00:40:52.734093 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647727 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 00:40:52.734093 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647730 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 00:40:52.734093 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647732 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 00:40:52.734093 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647735 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 00:40:52.734093 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647737 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 00:40:52.734093 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647740 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 00:40:52.734093 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647743 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 00:40:52.734093 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647746 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 00:40:52.711271 ip-10-0-129-137 systemd[1]: Started Kubernetes Kubelet. Jul 21 00:40:52.738884 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647748 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 00:40:52.738884 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647751 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 00:40:52.738884 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647754 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 00:40:52.738884 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647757 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 00:40:52.738884 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647760 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 00:40:52.738884 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647762 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 00:40:52.738884 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647764 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 00:40:52.738884 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647767 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 00:40:52.738884 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647770 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 00:40:52.738884 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647772 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 00:40:52.738884 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647775 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 00:40:52.738884 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647777 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 00:40:52.738884 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647779 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 00:40:52.738884 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647783 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 00:40:52.738884 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647786 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 00:40:52.738884 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647790 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 00:40:52.738884 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647795 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 00:40:52.738884 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647797 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 00:40:52.738884 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647800 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 00:40:52.755695 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647803 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 00:40:52.755695 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647806 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 00:40:52.755695 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647808 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 00:40:52.755695 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647811 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 00:40:52.755695 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647814 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 00:40:52.755695 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647816 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 00:40:52.755695 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647819 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 00:40:52.755695 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647822 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 00:40:52.755695 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647824 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 00:40:52.755695 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647828 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 00:40:52.755695 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647830 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 00:40:52.755695 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647833 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 00:40:52.755695 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.647836 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 00:40:52.755695 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648520 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 00:40:52.755695 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648527 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 00:40:52.755695 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648530 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 00:40:52.755695 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648535 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 00:40:52.755695 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648538 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 00:40:52.755695 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648540 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 00:40:52.755695 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648543 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 00:40:52.769824 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648545 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 00:40:52.769824 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648548 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 00:40:52.769824 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648551 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 00:40:52.769824 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648553 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 00:40:52.769824 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648558 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 00:40:52.769824 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648561 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 00:40:52.769824 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648565 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 00:40:52.769824 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648568 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 00:40:52.769824 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648571 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 00:40:52.769824 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648574 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 00:40:52.769824 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648577 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 00:40:52.769824 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648580 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 00:40:52.769824 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648583 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 00:40:52.769824 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648586 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 00:40:52.769824 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648589 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 00:40:52.769824 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648592 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 00:40:52.769824 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648594 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 00:40:52.769824 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648597 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 00:40:52.769824 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648599 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 00:40:52.769824 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648602 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 00:40:52.788348 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648604 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 00:40:52.788348 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648607 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 00:40:52.788348 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648609 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 00:40:52.788348 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648612 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 00:40:52.788348 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648614 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 21 00:40:52.788348 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648617 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 00:40:52.788348 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648619 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 00:40:52.788348 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648622 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 00:40:52.788348 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648625 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 00:40:52.788348 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648628 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 00:40:52.788348 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648630 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 00:40:52.788348 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648633 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 00:40:52.788348 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648636 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 00:40:52.788348 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648638 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 00:40:52.788348 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648640 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 00:40:52.788348 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648643 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 00:40:52.788348 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648646 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 00:40:52.788348 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648648 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 00:40:52.788348 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648653 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 00:40:52.931582 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648656 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 00:40:52.931582 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648659 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 00:40:52.931582 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648662 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 00:40:52.931582 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648665 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 00:40:52.931582 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648668 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 00:40:52.931582 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648670 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 00:40:52.931582 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648673 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 00:40:52.931582 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648677 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 00:40:52.931582 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648680 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 00:40:52.931582 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648683 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 00:40:52.931582 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648685 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 00:40:52.931582 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648687 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 00:40:52.931582 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648690 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 00:40:52.931582 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648692 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 00:40:52.931582 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648695 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 00:40:52.931582 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648697 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 00:40:52.931582 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648700 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 00:40:52.931582 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648702 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 00:40:52.931582 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648705 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 00:40:52.931582 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648707 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 00:40:52.932749 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648710 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 00:40:52.932749 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648713 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 00:40:52.932749 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648716 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 00:40:52.932749 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648718 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 00:40:52.932749 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648722 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 00:40:52.932749 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648725 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 00:40:52.932749 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648727 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 00:40:52.932749 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648730 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 00:40:52.932749 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648732 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 00:40:52.932749 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648735 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 00:40:52.932749 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648737 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 00:40:52.932749 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648740 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 00:40:52.932749 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648743 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 00:40:52.932749 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648745 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 00:40:52.932749 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648748 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 00:40:52.932749 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648750 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 00:40:52.932749 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648753 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 00:40:52.932749 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648756 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 00:40:52.932749 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648758 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 00:40:52.934538 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648762 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 00:40:52.934538 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648765 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 00:40:52.934538 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648768 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 00:40:52.934538 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648770 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 00:40:52.934538 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648773 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 00:40:52.934538 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648775 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 00:40:52.934538 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648778 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 00:40:52.934538 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648780 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 00:40:52.934538 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648783 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 00:40:52.934538 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648785 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 00:40:52.934538 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648788 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 00:40:52.934538 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648790 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 00:40:52.934538 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648792 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 00:40:52.934538 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.648795 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 00:40:52.934538 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649216 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 00:40:52.934538 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649221 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 00:40:52.934538 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649224 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 00:40:52.934538 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649228 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 00:40:52.934538 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649245 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 00:40:52.934538 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649248 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 00:40:52.935173 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649250 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 00:40:52.935173 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649253 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 00:40:52.935173 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649256 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 00:40:52.935173 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649258 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 00:40:52.935173 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649261 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 00:40:52.935173 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649264 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 00:40:52.935173 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649266 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 00:40:52.935173 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649269 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 00:40:52.935173 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649271 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 00:40:52.935173 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649274 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 00:40:52.935173 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649277 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 00:40:52.935173 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649279 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 00:40:52.935173 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649283 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 00:40:52.935173 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649286 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 00:40:52.935173 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649288 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 00:40:52.935173 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649291 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 00:40:52.935173 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649294 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 00:40:52.935173 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649298 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 00:40:52.935173 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649301 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 00:40:53.179499 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649303 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 00:40:53.179499 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649306 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 00:40:53.179499 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649308 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 00:40:53.179499 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649311 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 00:40:53.179499 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649313 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 00:40:53.179499 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649316 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 00:40:53.179499 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649318 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 00:40:53.179499 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649322 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 00:40:53.179499 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649324 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 00:40:53.179499 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649327 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 00:40:53.179499 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649330 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 00:40:53.179499 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649333 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 00:40:53.179499 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649335 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 00:40:53.179499 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649338 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 00:40:53.179499 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649340 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 00:40:53.179499 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649343 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 00:40:53.179499 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649345 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 00:40:53.179499 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649347 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 00:40:53.179499 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649350 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 00:40:53.179499 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649352 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 00:40:53.180179 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649355 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 00:40:53.180179 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649357 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 00:40:53.180179 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649360 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 00:40:53.180179 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649363 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 00:40:53.180179 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649366 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 00:40:53.180179 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649370 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 00:40:53.180179 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649372 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 00:40:53.180179 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649374 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 00:40:53.180179 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649377 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 00:40:53.180179 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649379 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 00:40:53.180179 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649382 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 00:40:53.180179 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649384 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 00:40:53.180179 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649386 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 00:40:53.180179 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649389 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 21 00:40:53.180179 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649391 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 00:40:53.180179 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649395 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 00:40:53.180179 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649398 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 00:40:53.180179 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649401 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 00:40:53.180179 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649404 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 00:40:53.180179 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649408 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 00:40:53.180822 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649411 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 00:40:53.180822 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649413 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 00:40:53.180822 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649416 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 00:40:53.180822 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649419 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 00:40:53.180822 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649423 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 00:40:53.180822 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649426 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 00:40:53.180822 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649429 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 00:40:53.180822 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649432 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 00:40:53.180822 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649434 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 00:40:53.180822 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649437 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 00:40:53.180822 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649440 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 00:40:53.180822 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649444 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 00:40:53.180822 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649446 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 00:40:53.180822 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649449 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 00:40:53.180822 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649451 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 00:40:53.180822 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649454 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 00:40:53.180822 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649457 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 00:40:53.180822 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649461 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 00:40:53.180822 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649464 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 00:40:53.181436 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649467 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 00:40:53.181436 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649469 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 00:40:53.181436 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649472 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 00:40:53.181436 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649475 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 00:40:53.181436 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649477 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 00:40:53.181436 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649480 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 00:40:53.181436 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649482 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 00:40:53.181436 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649485 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 00:40:53.181436 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649487 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 00:40:53.181436 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649490 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 00:40:53.181436 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649492 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 00:40:53.181436 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649495 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 00:40:53.181436 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649498 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 00:40:53.181436 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649501 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 00:40:53.181436 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.649503 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 00:40:53.181436 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649574 2580 flags.go:64] FLAG: --address="0.0.0.0" Jul 21 00:40:53.181436 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649581 2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 21 00:40:53.181436 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649587 2580 flags.go:64] FLAG: --anonymous-auth="true" Jul 21 00:40:53.181436 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649591 2580 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 21 00:40:53.181436 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649595 2580 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 21 00:40:53.181436 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649599 2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 21 00:40:53.182048 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649602 2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 21 00:40:53.182048 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649606 2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 21 00:40:53.182048 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649609 2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 21 00:40:53.182048 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649612 2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 21 00:40:53.182048 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649616 2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 21 00:40:53.182048 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649619 2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 21 00:40:53.182048 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649622 2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 21 00:40:53.182048 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649625 2580 flags.go:64] FLAG: --cgroup-root="" Jul 21 00:40:53.182048 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649628 2580 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 21 00:40:53.182048 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649632 2580 flags.go:64] FLAG: --client-ca-file="" Jul 21 00:40:53.182048 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649636 2580 flags.go:64] FLAG: --cloud-provider="external" Jul 21 00:40:53.182048 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649639 2580 flags.go:64] FLAG: --cluster-dns="[]" Jul 21 00:40:53.182048 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649643 2580 flags.go:64] FLAG: --cluster-domain="" Jul 21 00:40:53.182048 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649646 2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 21 00:40:53.182048 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649649 2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 21 00:40:53.182048 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649652 2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 21 00:40:53.182048 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649655 2580 flags.go:64] FLAG: --container-log-max-files="5" Jul 21 00:40:53.182048 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649658 2580 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 21 00:40:53.182048 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649661 2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 21 00:40:53.182048 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649664 2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 21 00:40:53.182048 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649668 2580 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 21 00:40:53.182048 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649671 2580 flags.go:64] FLAG: --contention-profiling="false" Jul 21 00:40:53.182048 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649673 2580 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 21 00:40:53.182048 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649678 2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 21 00:40:53.182762 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649681 2580 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 21 00:40:53.182762 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649684 2580 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 21 00:40:53.182762 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649688 2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 21 00:40:53.182762 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649691 2580 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 21 00:40:53.182762 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649694 2580 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 21 00:40:53.182762 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649697 2580 flags.go:64] FLAG: --enable-load-reader="false" Jul 21 00:40:53.182762 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649700 2580 flags.go:64] FLAG: --enable-server="true" Jul 21 00:40:53.182762 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649702 2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 21 00:40:53.182762 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649707 2580 flags.go:64] FLAG: --event-burst="100" Jul 21 00:40:53.182762 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649710 2580 flags.go:64] FLAG: --event-qps="50" Jul 21 00:40:53.182762 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649713 2580 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 21 00:40:53.182762 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649716 2580 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 21 00:40:53.182762 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649719 2580 flags.go:64] FLAG: --eviction-hard="" Jul 21 00:40:53.182762 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649722 2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 21 00:40:53.182762 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649726 2580 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 21 00:40:53.182762 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649729 2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 21 00:40:53.182762 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649731 2580 flags.go:64] FLAG: --eviction-soft="" Jul 21 00:40:53.182762 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649735 2580 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 21 00:40:53.182762 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649738 2580 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 21 00:40:53.182762 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649741 2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 21 00:40:53.182762 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649744 2580 flags.go:64] FLAG: --experimental-mounter-path="" Jul 21 00:40:53.182762 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649747 2580 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 21 00:40:53.182762 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649750 2580 flags.go:64] FLAG: --fail-swap-on="true" Jul 21 00:40:53.182762 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649753 2580 flags.go:64] FLAG: --feature-gates="" Jul 21 00:40:53.182762 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649756 2580 flags.go:64] FLAG: --file-check-frequency="20s" Jul 21 00:40:53.183474 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649759 2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 21 00:40:53.183474 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649762 2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 21 00:40:53.183474 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649765 2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 21 00:40:53.183474 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649768 2580 flags.go:64] FLAG: --healthz-port="10248" Jul 21 00:40:53.183474 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649771 2580 flags.go:64] FLAG: --help="false" Jul 21 00:40:53.183474 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649774 2580 flags.go:64] FLAG: --hostname-override="ip-10-0-129-137.ec2.internal" Jul 21 00:40:53.183474 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649779 2580 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 21 00:40:53.183474 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649781 2580 flags.go:64] FLAG: --http-check-frequency="20s" Jul 21 00:40:53.183474 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649784 2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 21 00:40:53.183474 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649787 2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 21 00:40:53.183474 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649790 2580 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 21 00:40:53.183474 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649793 2580 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 21 00:40:53.183474 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649796 2580 flags.go:64] FLAG: --image-service-endpoint="" Jul 21 00:40:53.183474 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649799 2580 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 21 00:40:53.183474 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649802 2580 flags.go:64] FLAG: --kube-api-burst="100" Jul 21 00:40:53.183474 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649804 2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 21 00:40:53.183474 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649807 2580 flags.go:64] FLAG: --kube-api-qps="50" Jul 21 00:40:53.183474 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649810 2580 flags.go:64] FLAG: --kube-reserved="" Jul 21 00:40:53.183474 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649813 2580 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 21 00:40:53.183474 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649816 2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 21 00:40:53.183474 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649819 2580 flags.go:64] FLAG: --kubelet-cgroups="" Jul 21 00:40:53.183474 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649821 2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 21 00:40:53.183474 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649824 2580 flags.go:64] FLAG: --lock-file="" Jul 21 00:40:53.183474 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649827 2580 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 21 00:40:53.184128 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649831 2580 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 21 00:40:53.184128 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649834 2580 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 21 00:40:53.184128 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649839 2580 flags.go:64] FLAG: --log-json-split-stream="false" Jul 21 00:40:53.184128 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649842 2580 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 21 00:40:53.184128 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649845 2580 flags.go:64] FLAG: --log-text-split-stream="false" Jul 21 00:40:53.184128 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649847 2580 flags.go:64] FLAG: --logging-format="text" Jul 21 00:40:53.184128 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649850 2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 21 00:40:53.184128 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649853 2580 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 21 00:40:53.184128 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649856 2580 flags.go:64] FLAG: --manifest-url="" Jul 21 00:40:53.184128 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649859 2580 flags.go:64] FLAG: --manifest-url-header="" Jul 21 00:40:53.184128 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649862 2580 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 21 00:40:53.184128 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649865 2580 flags.go:64] FLAG: --max-open-files="1000000" Jul 21 00:40:53.184128 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649869 2580 flags.go:64] FLAG: --max-pods="110" Jul 21 00:40:53.184128 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649872 2580 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 21 00:40:53.184128 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649876 2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 21 00:40:53.184128 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649879 2580 flags.go:64] FLAG: --memory-manager-policy="None" Jul 21 00:40:53.184128 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649882 2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 21 00:40:53.184128 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649885 2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 21 00:40:53.184128 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649887 2580 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 21 00:40:53.184128 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649890 2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 21 00:40:53.184128 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649897 2580 flags.go:64] FLAG: --node-status-max-images="50" Jul 21 00:40:53.184128 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649900 2580 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 21 00:40:53.184128 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649903 2580 flags.go:64] FLAG: --oom-score-adj="-999" Jul 21 00:40:53.184128 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649906 2580 flags.go:64] FLAG: --pod-cidr="" Jul 21 00:40:53.184841 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649909 2580 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 21 00:40:53.184841 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649915 2580 flags.go:64] FLAG: --pod-manifest-path="" Jul 21 00:40:53.184841 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649917 2580 flags.go:64] FLAG: --pod-max-pids="-1" Jul 21 00:40:53.184841 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649920 2580 flags.go:64] FLAG: --pods-per-core="0" Jul 21 00:40:53.184841 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649923 2580 flags.go:64] FLAG: --port="10250" Jul 21 00:40:53.184841 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649926 2580 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 21 00:40:53.184841 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649929 2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0bf1765099878ea56" Jul 21 00:40:53.184841 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649932 2580 flags.go:64] FLAG: --qos-reserved="" Jul 21 00:40:53.184841 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649937 2580 flags.go:64] FLAG: --read-only-port="10255" Jul 21 00:40:53.184841 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649940 2580 flags.go:64] FLAG: --register-node="true" Jul 21 00:40:53.184841 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649943 2580 flags.go:64] FLAG: --register-with-taints="" Jul 21 00:40:53.184841 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649946 2580 flags.go:64] FLAG: --registry-burst="10" Jul 21 00:40:53.184841 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649950 2580 flags.go:64] FLAG: --registry-qps="5" Jul 21 00:40:53.184841 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649952 2580 flags.go:64] FLAG: --reserved-cpus="" Jul 21 00:40:53.184841 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649955 2580 flags.go:64] FLAG: --reserved-memory="" Jul 21 00:40:53.184841 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649959 2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 21 00:40:53.184841 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649962 2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 21 00:40:53.184841 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649965 2580 flags.go:64] FLAG: --rotate-certificates="false" Jul 21 00:40:53.184841 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649968 2580 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 21 00:40:53.184841 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649971 2580 flags.go:64] FLAG: --runonce="false" Jul 21 00:40:53.184841 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649973 2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 21 00:40:53.184841 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649976 2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 21 00:40:53.184841 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649979 2580 flags.go:64] FLAG: --seccomp-default="false" Jul 21 00:40:53.184841 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649983 2580 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 21 00:40:53.184841 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649986 2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 21 00:40:53.185541 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649989 2580 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 21 00:40:53.185541 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649992 2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 21 00:40:53.185541 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649995 2580 flags.go:64] FLAG: --storage-driver-password="root" Jul 21 00:40:53.185541 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.649997 2580 flags.go:64] FLAG: --storage-driver-secure="false" Jul 21 00:40:53.185541 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.650000 2580 flags.go:64] FLAG: --storage-driver-table="stats" Jul 21 00:40:53.185541 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.650003 2580 flags.go:64] FLAG: --storage-driver-user="root" Jul 21 00:40:53.185541 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.650006 2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 21 00:40:53.185541 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.650009 2580 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 21 00:40:53.185541 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.650011 2580 flags.go:64] FLAG: --system-cgroups="" Jul 21 00:40:53.185541 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.650014 2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 21 00:40:53.185541 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.650019 2580 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 21 00:40:53.185541 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.650022 2580 flags.go:64] FLAG: --tls-cert-file="" Jul 21 00:40:53.185541 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.650025 2580 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 21 00:40:53.185541 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.650029 2580 flags.go:64] FLAG: --tls-min-version="" Jul 21 00:40:53.185541 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.650031 2580 flags.go:64] FLAG: --tls-private-key-file="" Jul 21 00:40:53.185541 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.650036 2580 flags.go:64] FLAG: --topology-manager-policy="none" Jul 21 00:40:53.185541 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.650039 2580 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 21 00:40:53.185541 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.650042 2580 flags.go:64] FLAG: --topology-manager-scope="container" Jul 21 00:40:53.185541 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.650045 2580 flags.go:64] FLAG: --v="2" Jul 21 00:40:53.185541 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.650048 2580 flags.go:64] FLAG: --version="false" Jul 21 00:40:53.185541 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.650055 2580 flags.go:64] FLAG: --vmodule="" Jul 21 00:40:53.185541 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.650059 2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 21 00:40:53.185541 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.650062 2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 21 00:40:53.185541 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650176 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 00:40:53.185541 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650180 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 00:40:53.186182 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650183 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 00:40:53.186182 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650186 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 00:40:53.186182 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650189 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 00:40:53.186182 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650191 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 00:40:53.186182 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650194 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 00:40:53.186182 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650199 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 00:40:53.186182 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650202 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 00:40:53.186182 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650204 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 00:40:53.186182 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650207 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 00:40:53.186182 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650210 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 00:40:53.186182 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650212 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 00:40:53.186182 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650215 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 00:40:53.186182 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650217 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 00:40:53.186182 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650220 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 00:40:53.186182 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650223 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 00:40:53.186182 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650225 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 00:40:53.186182 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650227 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 00:40:53.186182 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650245 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 00:40:53.186182 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650258 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 00:40:53.186182 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650260 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 00:40:53.186761 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650263 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 00:40:53.186761 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650265 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 00:40:53.186761 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650268 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 00:40:53.186761 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650271 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 00:40:53.186761 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650274 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 00:40:53.186761 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650277 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 00:40:53.186761 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650279 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 00:40:53.186761 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650282 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 00:40:53.186761 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650285 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 00:40:53.186761 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650288 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 00:40:53.186761 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650290 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 00:40:53.186761 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650293 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 00:40:53.186761 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650295 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 00:40:53.186761 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650299 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 00:40:53.186761 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650302 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 00:40:53.186761 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650306 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 00:40:53.186761 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650309 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 00:40:53.186761 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650314 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 00:40:53.186761 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650317 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 00:40:53.187271 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650320 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 00:40:53.187271 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650323 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 00:40:53.187271 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650326 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 00:40:53.187271 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650328 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 00:40:53.187271 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650331 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 00:40:53.187271 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650334 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 00:40:53.187271 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650337 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 00:40:53.187271 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650339 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 00:40:53.187271 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650342 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 21 00:40:53.187271 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650345 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 00:40:53.187271 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650347 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 00:40:53.187271 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650350 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 00:40:53.187271 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650352 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 00:40:53.187271 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650355 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 00:40:53.187271 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650357 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 00:40:53.187271 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650360 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 00:40:53.187271 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650364 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 00:40:53.187271 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650367 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 00:40:53.187271 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650369 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 00:40:53.187271 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650372 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 00:40:53.187836 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650375 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 00:40:53.187836 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650377 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 00:40:53.187836 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650380 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 00:40:53.187836 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650384 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 00:40:53.187836 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650387 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 00:40:53.187836 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650390 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 00:40:53.187836 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650393 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 00:40:53.187836 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650396 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 00:40:53.187836 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650399 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 00:40:53.187836 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650401 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 00:40:53.187836 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650405 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 00:40:53.187836 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650408 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 00:40:53.187836 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650411 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 00:40:53.187836 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650414 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 00:40:53.187836 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650416 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 00:40:53.187836 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650419 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 00:40:53.187836 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650422 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 00:40:53.187836 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650424 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 00:40:53.187836 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650427 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 00:40:53.188422 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650429 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 00:40:53.188422 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650432 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 00:40:53.188422 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650434 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 00:40:53.188422 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650437 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 00:40:53.188422 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650439 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 00:40:53.188422 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650442 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 00:40:53.188422 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650444 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 00:40:53.188422 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650447 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 00:40:53.188422 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650449 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 00:40:53.188422 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650453 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 00:40:53.188422 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650455 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 00:40:53.188422 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650458 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 00:40:53.188422 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650460 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 00:40:53.188422 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650463 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 00:40:53.188422 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650465 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 00:40:53.188422 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650468 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 00:40:53.188422 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650471 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 00:40:53.188422 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650473 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 00:40:53.188422 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.650476 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 00:40:53.188952 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.651019 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 21 00:40:53.188952 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.657630 2580 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 21 00:40:53.188952 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.657645 2580 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 21 00:40:53.188952 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.657664 2580 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 21 00:40:53.188952 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.657671 2580 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 21 00:40:53.188952 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657725 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 00:40:53.188952 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657730 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 00:40:53.188952 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657733 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 00:40:53.188952 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657736 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 00:40:53.188952 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657739 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 00:40:53.188952 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657742 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 00:40:53.188952 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657745 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 00:40:53.188952 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657748 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 00:40:53.188952 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657751 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 00:40:53.188952 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657754 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 00:40:53.189389 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657756 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 00:40:53.189389 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657759 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 00:40:53.189389 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657761 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 00:40:53.189389 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657764 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 00:40:53.189389 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657767 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 00:40:53.189389 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657770 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 00:40:53.189389 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657772 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 00:40:53.189389 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657775 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 00:40:53.189389 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657778 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 00:40:53.189389 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657780 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 00:40:53.189389 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657783 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 00:40:53.189389 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657786 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 00:40:53.189389 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657790 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 00:40:53.189389 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657792 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 00:40:53.189389 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657795 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 00:40:53.189389 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657798 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 00:40:53.189389 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657800 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 00:40:53.189389 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657803 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 00:40:53.189389 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657807 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 00:40:53.189912 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657811 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 00:40:53.189912 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657814 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 00:40:53.189912 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657817 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 00:40:53.189912 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657819 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 00:40:53.189912 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657822 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 00:40:53.189912 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657825 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 00:40:53.189912 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657827 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 00:40:53.189912 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657830 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 00:40:53.189912 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657832 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 00:40:53.189912 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657836 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 00:40:53.189912 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657840 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 00:40:53.189912 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657843 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 00:40:53.189912 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657846 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 00:40:53.189912 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657849 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 00:40:53.189912 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657852 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 00:40:53.189912 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657855 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 00:40:53.189912 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657858 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 00:40:53.189912 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657861 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 00:40:53.189912 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657864 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 21 00:40:53.189912 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657867 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 00:40:53.190441 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657869 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 00:40:53.190441 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657872 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 00:40:53.190441 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657875 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 00:40:53.190441 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657878 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 00:40:53.190441 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657880 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 00:40:53.190441 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657883 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 00:40:53.190441 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657886 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 00:40:53.190441 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657889 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 00:40:53.190441 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657892 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 00:40:53.190441 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657894 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 00:40:53.190441 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657897 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 00:40:53.190441 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657900 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 00:40:53.190441 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657902 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 00:40:53.190441 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657905 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 00:40:53.190441 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657910 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 00:40:53.190441 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657913 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 00:40:53.190441 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657916 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 00:40:53.190441 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657918 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 00:40:53.190441 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657921 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 00:40:53.190441 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657923 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 00:40:53.190994 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657926 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 00:40:53.190994 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657929 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 00:40:53.190994 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657931 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 00:40:53.190994 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657934 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 00:40:53.190994 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657936 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 00:40:53.190994 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657939 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 00:40:53.190994 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657942 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 00:40:53.190994 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657945 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 00:40:53.190994 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657947 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 00:40:53.190994 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657950 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 00:40:53.190994 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657952 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 00:40:53.190994 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657955 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 00:40:53.190994 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657957 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 00:40:53.190994 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657960 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 00:40:53.190994 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657962 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 00:40:53.190994 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657965 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 00:40:53.190994 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657967 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 00:40:53.190994 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657970 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 00:40:53.190994 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657972 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 00:40:53.191769 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657975 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 00:40:53.191769 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657977 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 00:40:53.191769 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657980 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 00:40:53.191769 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657983 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 00:40:53.191769 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657986 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 00:40:53.191769 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657989 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 00:40:53.191769 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657992 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 00:40:53.191769 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.657997 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 00:40:53.191769 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658000 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 00:40:53.191769 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658003 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 00:40:53.191769 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658006 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 00:40:53.191769 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.658011 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 21 00:40:53.191769 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658115 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 00:40:53.191769 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658120 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 00:40:53.191769 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658123 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 00:40:53.192217 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658126 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 00:40:53.192217 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658129 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 00:40:53.192217 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658132 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 00:40:53.192217 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658134 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 00:40:53.192217 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658137 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 00:40:53.192217 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658139 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 00:40:53.192217 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658142 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 00:40:53.192217 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658144 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 00:40:53.192217 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658147 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 00:40:53.192217 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658150 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 00:40:53.192217 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658153 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 00:40:53.192217 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658155 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 00:40:53.192217 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658158 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 00:40:53.192217 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658160 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 00:40:53.192217 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658163 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 00:40:53.192217 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658166 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 00:40:53.192217 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658168 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 00:40:53.192217 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658171 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 00:40:53.192217 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658173 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 00:40:53.192217 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658176 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 00:40:53.452675 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658181 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 00:40:53.452675 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658183 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 00:40:53.452675 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658186 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 00:40:53.452675 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658188 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 00:40:53.452675 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658193 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 00:40:53.452675 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658196 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 00:40:53.452675 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658199 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 00:40:53.452675 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658201 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 00:40:53.452675 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658204 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 00:40:53.452675 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658208 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 00:40:53.452675 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658212 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 00:40:53.452675 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658215 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 00:40:53.452675 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658218 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 00:40:53.452675 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658221 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 00:40:53.452675 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658224 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 00:40:53.452675 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658227 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 00:40:53.452675 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658249 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 00:40:53.452675 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658252 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 00:40:53.452675 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658255 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 00:40:53.453921 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658257 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 00:40:53.453921 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658260 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 00:40:53.453921 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658263 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 00:40:53.453921 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658267 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 00:40:53.453921 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658270 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 00:40:53.453921 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658272 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 00:40:53.453921 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658275 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 00:40:53.453921 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658277 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 00:40:53.453921 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658280 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 00:40:53.453921 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658282 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 00:40:53.453921 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658285 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 00:40:53.453921 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658288 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 00:40:53.453921 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658290 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 00:40:53.453921 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658296 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 00:40:53.453921 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658298 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 00:40:53.453921 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658301 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 00:40:53.453921 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658303 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 00:40:53.453921 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658307 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 00:40:53.453921 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658310 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 00:40:53.454480 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658313 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 00:40:53.454480 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658316 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 00:40:53.454480 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658318 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 00:40:53.454480 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658321 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 00:40:53.454480 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658324 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 00:40:53.454480 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658327 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 00:40:53.454480 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658329 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 00:40:53.454480 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658332 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 00:40:53.454480 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658335 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 00:40:53.454480 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658337 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 21 00:40:53.454480 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658340 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 00:40:53.454480 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658342 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 00:40:53.454480 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658344 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 00:40:53.454480 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658348 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 00:40:53.454480 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658351 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 00:40:53.454480 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658354 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 00:40:53.454480 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658356 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 00:40:53.454480 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658359 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 00:40:53.454480 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658361 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 00:40:53.454480 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658364 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 00:40:53.455032 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658366 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 00:40:53.455032 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658369 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 00:40:53.455032 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658371 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 00:40:53.455032 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658374 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 00:40:53.455032 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658376 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 00:40:53.455032 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658379 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 00:40:53.455032 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658382 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 00:40:53.455032 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658385 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 00:40:53.455032 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658387 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 00:40:53.455032 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658390 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 00:40:53.455032 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658394 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 00:40:53.455032 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658396 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 00:40:53.455032 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658399 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 00:40:53.455032 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658402 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 00:40:53.455032 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658405 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 00:40:53.455032 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658407 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 00:40:53.455032 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658409 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 00:40:53.455032 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:52.658412 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 00:40:53.455526 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.658417 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 21 00:40:53.455526 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.658515 2580 server.go:962] "Client rotation is on, will bootstrap in background" Jul 21 00:40:53.455526 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.660642 2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 21 00:40:53.455526 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.662673 2580 server.go:1019] "Starting client certificate rotation" Jul 21 00:40:53.455526 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.662785 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 21 00:40:53.455526 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.662819 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 21 00:40:53.455526 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.665974 2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 21 00:40:53.455526 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.671397 2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 21 00:40:53.455526 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.681439 2580 log.go:25] "Validated CRI v1 runtime API" Jul 21 00:40:53.455526 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.685333 2580 log.go:25] "Validated CRI v1 image API" Jul 21 00:40:53.455526 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.686101 2580 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 21 00:40:53.455526 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.688141 2580 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 b0e32356-4413-4935-98f7-de4ee4e35fe7:/dev/nvme0n1p4 b1ba4a06-bb5d-4460-9d72-397f3e870339:/dev/nvme0n1p3] Jul 21 00:40:53.455899 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.688153 2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 21 00:40:53.455965 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.692735 2580 manager.go:216] Machine: {Timestamp:2026-07-21 00:40:52.691571611 +0000 UTC m=+0.297680642 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3203213 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ee8de65d81931a15fd255d5d2281e SystemUUID:ec2ee8de-65d8-1931-a15f-d255d5d2281e BootID:ce697dd8-73c7-497a-8116-a68b1274c039 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:81:a1:8d:bc:b1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:81:a1:8d:bc:b1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:2e:a1:a0:ec:72:87 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 21 00:40:53.455965 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.692821 2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 21 00:40:53.455965 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.692888 2580 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 21 00:40:53.455965 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.694420 2580 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 21 00:40:53.455965 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.694443 2580 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-137.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 21 00:40:53.455965 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.695115 2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 21 00:40:53.455965 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.695122 2580 topology_manager.go:138] "Creating topology manager with none policy" Jul 21 00:40:53.456255 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.695144 2580 container_manager_linux.go:309] "Creating device plugin manager" Jul 21 00:40:53.456255 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.695152 2580 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 21 00:40:53.456255 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.695755 2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 21 00:40:53.456255 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.697134 2580 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 21 00:40:53.456255 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.701333 2580 state_mem.go:36] "Initialized new in-memory state store" Jul 21 00:40:53.456255 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.701431 2580 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 21 00:40:53.456255 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.701458 2580 kubelet.go:486] "Attempting to sync node with API server" Jul 21 00:40:53.456255 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.701467 2580 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 21 00:40:53.456255 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.701477 2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 21 00:40:53.456255 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.701497 2580 kubelet.go:398] "Adding apiserver pod source" Jul 21 00:40:53.456255 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.701505 2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 21 00:40:53.456255 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.701757 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 21 00:40:53.456255 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.701777 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 21 00:40:53.456255 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.703468 2580 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 21 00:40:53.456255 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.705550 2580 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 21 00:40:53.456255 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.705577 2580 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 21 00:40:53.456255 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.706622 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 21 00:40:53.456255 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.706635 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 21 00:40:53.456255 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.706642 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 21 00:40:53.456255 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.706648 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 21 00:40:53.456255 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.706659 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 21 00:40:53.456255 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.706668 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 21 00:40:53.456255 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.706678 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 21 00:40:53.456255 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.706683 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 21 00:40:53.456255 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.706689 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 21 00:40:53.456255 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.706694 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 21 00:40:53.456255 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.706707 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 21 00:40:53.456255 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.706712 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 21 00:40:53.456255 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.707401 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 21 00:40:53.456255 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.707410 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 21 00:40:53.456255 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.710535 2580 server.go:1268] "Started kubelet" Jul 21 00:40:53.456255 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.710613 2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 21 00:40:53.456255 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.710668 2580 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 21 00:40:53.456255 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.710723 2580 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 21 00:40:53.456255 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.714420 2580 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 21 00:40:53.456255 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:40:52.715129 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-137.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 21 00:40:53.456255 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.715348 2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-137.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 21 00:40:53.456255 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:40:52.715489 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 21 00:40:53.456255 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.715942 2580 server.go:310] "Adding debug handlers to kubelet server" Jul 21 00:40:53.457259 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:40:52.717702 2580 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 21 00:40:53.457259 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:40:52.720359 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-137.ec2.internal.18c4273315f1387e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-137.ec2.internal,UID:ip-10-0-129-137.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-137.ec2.internal,},FirstTimestamp:2026-07-21 00:40:52.710504574 +0000 UTC m=+0.316613607,LastTimestamp:2026-07-21 00:40:52.710504574 +0000 UTC m=+0.316613607,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-137.ec2.internal,}" Jul 21 00:40:53.457259 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.724010 2580 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 21 00:40:53.457259 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.724029 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 21 00:40:53.457259 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:40:52.724118 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-137.ec2.internal\" not found" Jul 21 00:40:53.457259 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.724707 2580 factory.go:55] Registering systemd factory Jul 21 00:40:53.457259 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.724721 2580 factory.go:223] Registration of the systemd container factory successfully Jul 21 00:40:53.457259 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.724854 2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 21 00:40:53.457259 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.724862 2580 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 21 00:40:53.457259 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.724875 2580 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 21 00:40:53.457259 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.724911 2580 factory.go:153] Registering CRI-O factory Jul 21 00:40:53.457259 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.724925 2580 factory.go:223] Registration of the crio container factory successfully Jul 21 00:40:53.457259 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.724973 2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 21 00:40:53.457259 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.724997 2580 factory.go:103] Registering Raw factory Jul 21 00:40:53.457259 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.725016 2580 manager.go:1196] Started watching for new ooms in manager Jul 21 00:40:53.457259 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.724973 2580 reconstruct.go:97] "Volume reconstruction finished" Jul 21 00:40:53.457259 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.725034 2580 reconciler.go:29] "Reconciler: start to sync state" Jul 21 00:40:53.457259 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.725367 2580 manager.go:349] Starting recovery of all containers Jul 21 00:40:53.457259 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.736699 2580 manager.go:354] Recovery completed Jul 21 00:40:53.457259 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:40:52.737632 2580 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-137.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 21 00:40:53.457259 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:40:52.737672 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 21 00:40:53.457259 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.741017 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 00:40:53.457259 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.742206 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-n8lgs" Jul 21 00:40:53.457259 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.742790 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeHasSufficientMemory" Jul 21 00:40:53.457259 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.742814 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 00:40:53.457259 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.742839 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeHasSufficientPID" Jul 21 00:40:53.457259 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.743357 2580 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 21 00:40:53.457259 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.743366 2580 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 21 00:40:53.457259 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.743381 2580 state_mem.go:36] "Initialized new in-memory state store" Jul 21 00:40:53.457259 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.743398 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 00:40:53.457259 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.744154 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeHasSufficientMemory" Jul 21 00:40:53.457259 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.744178 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 00:40:53.458143 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.744188 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeHasSufficientPID" Jul 21 00:40:53.458143 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.745495 2580 policy_none.go:49] "None policy: Start" Jul 21 00:40:53.458143 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.745515 2580 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 21 00:40:53.458143 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.745529 2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 21 00:40:53.458143 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.747455 2580 policy_none.go:47] "Start" Jul 21 00:40:53.458143 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.749521 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-n8lgs" Jul 21 00:40:53.458143 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:40:52.754722 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-137.ec2.internal.18c4273317de07c6 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-137.ec2.internal,UID:ip-10-0-129-137.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-137.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-137.ec2.internal,},FirstTimestamp:2026-07-21 00:40:52.74280135 +0000 UTC m=+0.348910382,LastTimestamp:2026-07-21 00:40:52.74280135 +0000 UTC m=+0.348910382,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-137.ec2.internal,}" Jul 21 00:40:53.458143 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.809769 2580 manager.go:337] "Starting Device Plugin manager" Jul 21 00:40:53.458143 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:40:52.809796 2580 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 21 00:40:53.458143 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.809805 2580 server.go:87] "Starting device plugin registration server" Jul 21 00:40:53.458143 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.810034 2580 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 21 00:40:53.458143 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.810047 2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 21 00:40:53.458143 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.810158 2580 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 21 00:40:53.458143 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.810256 2580 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 21 00:40:53.458143 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.810268 2580 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 21 00:40:53.458143 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:40:52.810744 2580 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 21 00:40:53.458143 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:40:52.810781 2580 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-137.ec2.internal\" not found" Jul 21 00:40:53.458143 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.904367 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 21 00:40:53.458143 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.905499 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 21 00:40:53.458143 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.905531 2580 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 21 00:40:53.458143 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.905550 2580 kubelet.go:2448] "Starting kubelet main sync loop" Jul 21 00:40:53.458143 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:40:52.905588 2580 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 21 00:40:53.458143 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.907797 2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 00:40:53.458143 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.910637 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 00:40:53.458143 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.911370 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeHasSufficientMemory" Jul 21 00:40:53.458143 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.911395 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 00:40:53.458143 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.911404 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeHasSufficientPID" Jul 21 00:40:53.458143 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.911425 2580 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-137.ec2.internal" Jul 21 00:40:53.458143 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:52.920162 2580 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-137.ec2.internal" Jul 21 00:40:53.458143 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:40:52.920179 2580 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-137.ec2.internal\": node \"ip-10-0-129-137.ec2.internal\" not found" Jul 21 00:40:53.458143 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:40:52.944503 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-137.ec2.internal\" not found" Jul 21 00:40:53.458143 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:53.006004 2580 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-137.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-137.ec2.internal"] Jul 21 00:40:53.458143 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:53.006086 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 00:40:53.458996 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:53.007083 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeHasSufficientMemory" Jul 21 00:40:53.458996 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:53.007108 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 00:40:53.458996 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:53.007118 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeHasSufficientPID" Jul 21 00:40:53.458996 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:53.009427 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 00:40:53.458996 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:53.009539 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-137.ec2.internal" Jul 21 00:40:53.458996 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:53.009565 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 00:40:53.458996 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:53.010628 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeHasSufficientMemory" Jul 21 00:40:53.458996 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:53.010652 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 00:40:53.458996 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:53.010662 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeHasSufficientPID" Jul 21 00:40:53.458996 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:53.010628 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeHasSufficientMemory" Jul 21 00:40:53.458996 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:53.010685 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 00:40:53.458996 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:53.010698 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeHasSufficientPID" Jul 21 00:40:53.458996 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:53.012806 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-137.ec2.internal" Jul 21 00:40:53.458996 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:53.012836 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 00:40:53.458996 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:53.013574 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeHasSufficientMemory" Jul 21 00:40:53.458996 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:53.013599 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 00:40:53.458996 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:53.013613 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeHasSufficientPID" Jul 21 00:40:53.458996 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:53.027429 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2cf3a63f5e16fba7955768004cdb0f7c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-137.ec2.internal\" (UID: \"2cf3a63f5e16fba7955768004cdb0f7c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-137.ec2.internal" Jul 21 00:40:53.458996 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:53.027448 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2cf3a63f5e16fba7955768004cdb0f7c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-137.ec2.internal\" (UID: \"2cf3a63f5e16fba7955768004cdb0f7c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-137.ec2.internal" Jul 21 00:40:53.458996 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:53.027472 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f449d6f92faa6a51a14c54944141985e-config\") pod \"kube-apiserver-proxy-ip-10-0-129-137.ec2.internal\" (UID: \"f449d6f92faa6a51a14c54944141985e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-137.ec2.internal" Jul 21 00:40:53.458996 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:40:53.034820 2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-137.ec2.internal\" not found" node="ip-10-0-129-137.ec2.internal" Jul 21 00:40:53.458996 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:40:53.039919 2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-137.ec2.internal\" not found" node="ip-10-0-129-137.ec2.internal" Jul 21 00:40:53.458996 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:40:53.045194 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-137.ec2.internal\" not found" Jul 21 00:40:53.458996 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:53.128244 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2cf3a63f5e16fba7955768004cdb0f7c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-137.ec2.internal\" (UID: \"2cf3a63f5e16fba7955768004cdb0f7c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-137.ec2.internal" Jul 21 00:40:53.459664 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:53.128330 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2cf3a63f5e16fba7955768004cdb0f7c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-137.ec2.internal\" (UID: \"2cf3a63f5e16fba7955768004cdb0f7c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-137.ec2.internal" Jul 21 00:40:53.459664 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:53.128347 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f449d6f92faa6a51a14c54944141985e-config\") pod \"kube-apiserver-proxy-ip-10-0-129-137.ec2.internal\" (UID: \"f449d6f92faa6a51a14c54944141985e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-137.ec2.internal" Jul 21 00:40:53.459664 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:53.128301 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2cf3a63f5e16fba7955768004cdb0f7c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-137.ec2.internal\" (UID: \"2cf3a63f5e16fba7955768004cdb0f7c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-137.ec2.internal" Jul 21 00:40:53.459664 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:53.128377 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f449d6f92faa6a51a14c54944141985e-config\") pod \"kube-apiserver-proxy-ip-10-0-129-137.ec2.internal\" (UID: \"f449d6f92faa6a51a14c54944141985e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-137.ec2.internal" Jul 21 00:40:53.459664 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:53.128403 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2cf3a63f5e16fba7955768004cdb0f7c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-137.ec2.internal\" (UID: \"2cf3a63f5e16fba7955768004cdb0f7c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-137.ec2.internal" Jul 21 00:40:53.459664 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:40:53.145275 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-137.ec2.internal\" not found" Jul 21 00:40:53.459664 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:40:53.246058 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-137.ec2.internal\" not found" Jul 21 00:40:53.459664 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:53.338535 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-137.ec2.internal" Jul 21 00:40:53.459664 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:53.343086 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-137.ec2.internal" Jul 21 00:40:53.459664 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:40:53.346641 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-137.ec2.internal\" not found" Jul 21 00:40:53.459664 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:40:53.447203 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-137.ec2.internal\" not found" Jul 21 00:40:53.676617 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:40:53.547747 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-137.ec2.internal\" not found" Jul 21 00:40:53.676617 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:40:53.648202 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-137.ec2.internal\" not found" Jul 21 00:40:53.676617 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:53.661343 2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 21 00:40:53.676617 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:53.661484 2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 21 00:40:53.931552 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:53.720340 2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 00:40:53.931552 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:40:53.748587 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-137.ec2.internal\" not found" Jul 21 00:40:53.931552 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:53.754680 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-20 00:35:53 +0000 UTC" deadline="2028-04-04 22:01:19.924920648 +0000 UTC" Jul 21 00:40:53.931552 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:53.754718 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14973h20m26.170206907s" Jul 21 00:40:53.931552 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:40:53.849301 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-137.ec2.internal\" not found" Jul 21 00:40:54.235681 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:53.943570 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 00:40:54.235681 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:53.944453 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeHasSufficientMemory" Jul 21 00:40:54.235681 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:53.944481 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 00:40:54.235681 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:53.944491 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeHasSufficientPID" Jul 21 00:40:54.235681 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:40:53.949474 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-137.ec2.internal\" not found" Jul 21 00:40:54.235681 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.028256 2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 00:40:54.235681 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:40:54.050124 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-137.ec2.internal\" not found" Jul 21 00:40:54.235681 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.092754 2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 00:40:54.235681 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.126269 2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-137.ec2.internal" Jul 21 00:40:54.235681 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.139251 2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 21 00:40:54.235681 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.140108 2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-137.ec2.internal" Jul 21 00:40:54.235681 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.146641 2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 21 00:40:54.738752 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.703124 2580 apiserver.go:52] "Watching apiserver" Jul 21 00:40:54.738752 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.713184 2580 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 21 00:40:54.738752 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.714926 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-858wv","openshift-image-registry/node-ca-2rxh4","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-137.ec2.internal","openshift-multus/network-metrics-daemon-4nq5b","openshift-network-operator/iptables-alerter-fj98d","kube-system/global-pull-secret-syncer-dzcmw","openshift-cluster-node-tuning-operator/tuned-jwqsv","openshift-multus/multus-additional-cni-plugins-6zg78","openshift-multus/multus-krnqh","openshift-network-diagnostics/network-check-target-9r2ff","openshift-ovn-kubernetes/ovnkube-node-bctst","kube-system/konnectivity-agent-ntk2c","kube-system/kube-apiserver-proxy-ip-10-0-129-137.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qthms"] Jul 21 00:40:54.738752 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.717527 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jwqsv" Jul 21 00:40:54.738752 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.722907 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 21 00:40:54.738752 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.722947 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 21 00:40:54.738752 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.722966 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-r55jd\"" Jul 21 00:40:54.738752 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.724625 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 21 00:40:54.738752 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.725595 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bctst" Jul 21 00:40:54.738752 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.727845 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 21 00:40:54.738752 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.728566 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qthms" Jul 21 00:40:54.738752 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.728571 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 21 00:40:54.738752 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.728623 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 21 00:40:54.738752 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.728660 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 21 00:40:54.738752 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.728919 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-m5dbx\"" Jul 21 00:40:54.738752 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.728952 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 21 00:40:54.738752 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.729972 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 21 00:40:54.738752 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.731138 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 21 00:40:54.738752 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.731192 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4nq5b" Jul 21 00:40:54.738752 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:40:54.731286 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4nq5b" podUID="6cfb338e-19eb-45ee-a3de-ea7311469109" Jul 21 00:40:54.738752 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.731549 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-cv2qw\"" Jul 21 00:40:54.738752 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.731560 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 21 00:40:54.738752 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.731631 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 21 00:40:54.738752 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.733879 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fj98d" Jul 21 00:40:54.738752 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.736029 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/71312c5c-a477-4500-996f-4c7cffd7fdc6-host-slash\") pod \"ovnkube-node-bctst\" (UID: \"71312c5c-a477-4500-996f-4c7cffd7fdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bctst" Jul 21 00:40:54.740174 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.736046 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/71312c5c-a477-4500-996f-4c7cffd7fdc6-node-log\") pod \"ovnkube-node-bctst\" (UID: \"71312c5c-a477-4500-996f-4c7cffd7fdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bctst" Jul 21 00:40:54.740174 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.736063 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0b7ed92b-7e21-40b9-9ac6-f233a11168d0-etc-sysctl-d\") pod \"tuned-jwqsv\" (UID: \"0b7ed92b-7e21-40b9-9ac6-f233a11168d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jwqsv" Jul 21 00:40:54.740174 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.736080 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0b7ed92b-7e21-40b9-9ac6-f233a11168d0-etc-sysctl-conf\") pod \"tuned-jwqsv\" (UID: \"0b7ed92b-7e21-40b9-9ac6-f233a11168d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jwqsv" Jul 21 00:40:54.740174 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.736095 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0b7ed92b-7e21-40b9-9ac6-f233a11168d0-lib-modules\") pod \"tuned-jwqsv\" (UID: \"0b7ed92b-7e21-40b9-9ac6-f233a11168d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jwqsv" Jul 21 00:40:54.740174 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.736110 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b7ed92b-7e21-40b9-9ac6-f233a11168d0-tmp\") pod \"tuned-jwqsv\" (UID: \"0b7ed92b-7e21-40b9-9ac6-f233a11168d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jwqsv" Jul 21 00:40:54.740174 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.736125 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0b7ed92b-7e21-40b9-9ac6-f233a11168d0-etc-systemd\") pod \"tuned-jwqsv\" (UID: \"0b7ed92b-7e21-40b9-9ac6-f233a11168d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jwqsv" Jul 21 00:40:54.740174 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.736138 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0b7ed92b-7e21-40b9-9ac6-f233a11168d0-run\") pod \"tuned-jwqsv\" (UID: \"0b7ed92b-7e21-40b9-9ac6-f233a11168d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jwqsv" Jul 21 00:40:54.740174 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.736154 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2f98\" (UniqueName: \"kubernetes.io/projected/71312c5c-a477-4500-996f-4c7cffd7fdc6-kube-api-access-m2f98\") pod \"ovnkube-node-bctst\" (UID: \"71312c5c-a477-4500-996f-4c7cffd7fdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bctst" Jul 21 00:40:54.740174 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.736169 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0b7ed92b-7e21-40b9-9ac6-f233a11168d0-etc-modprobe-d\") pod \"tuned-jwqsv\" (UID: \"0b7ed92b-7e21-40b9-9ac6-f233a11168d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jwqsv" Jul 21 00:40:54.740174 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.736184 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b7ed92b-7e21-40b9-9ac6-f233a11168d0-etc-kubernetes\") pod \"tuned-jwqsv\" (UID: \"0b7ed92b-7e21-40b9-9ac6-f233a11168d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jwqsv" Jul 21 00:40:54.740174 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.736199 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71312c5c-a477-4500-996f-4c7cffd7fdc6-var-lib-openvswitch\") pod \"ovnkube-node-bctst\" (UID: \"71312c5c-a477-4500-996f-4c7cffd7fdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bctst" Jul 21 00:40:54.740174 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.736213 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/71312c5c-a477-4500-996f-4c7cffd7fdc6-log-socket\") pod \"ovnkube-node-bctst\" (UID: \"71312c5c-a477-4500-996f-4c7cffd7fdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bctst" Jul 21 00:40:54.740174 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.738387 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/71312c5c-a477-4500-996f-4c7cffd7fdc6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bctst\" (UID: \"71312c5c-a477-4500-996f-4c7cffd7fdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bctst" Jul 21 00:40:54.740174 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.738415 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/71312c5c-a477-4500-996f-4c7cffd7fdc6-env-overrides\") pod \"ovnkube-node-bctst\" (UID: \"71312c5c-a477-4500-996f-4c7cffd7fdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bctst" Jul 21 00:40:54.740829 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.738436 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/71312c5c-a477-4500-996f-4c7cffd7fdc6-systemd-units\") pod \"ovnkube-node-bctst\" (UID: \"71312c5c-a477-4500-996f-4c7cffd7fdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bctst" Jul 21 00:40:54.740829 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.738456 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/71312c5c-a477-4500-996f-4c7cffd7fdc6-run-systemd\") pod \"ovnkube-node-bctst\" (UID: \"71312c5c-a477-4500-996f-4c7cffd7fdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bctst" Jul 21 00:40:54.740829 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.738481 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0b7ed92b-7e21-40b9-9ac6-f233a11168d0-sys\") pod \"tuned-jwqsv\" (UID: \"0b7ed92b-7e21-40b9-9ac6-f233a11168d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jwqsv" Jul 21 00:40:54.740829 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.738505 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0b7ed92b-7e21-40b9-9ac6-f233a11168d0-host\") pod \"tuned-jwqsv\" (UID: \"0b7ed92b-7e21-40b9-9ac6-f233a11168d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jwqsv" Jul 21 00:40:54.740829 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.738534 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kvss\" (UniqueName: \"kubernetes.io/projected/0b7ed92b-7e21-40b9-9ac6-f233a11168d0-kube-api-access-8kvss\") pod \"tuned-jwqsv\" (UID: \"0b7ed92b-7e21-40b9-9ac6-f233a11168d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jwqsv" Jul 21 00:40:54.740829 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.738581 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/71312c5c-a477-4500-996f-4c7cffd7fdc6-host-kubelet\") pod \"ovnkube-node-bctst\" (UID: \"71312c5c-a477-4500-996f-4c7cffd7fdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bctst" Jul 21 00:40:54.740829 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.738606 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/71312c5c-a477-4500-996f-4c7cffd7fdc6-host-cni-bin\") pod \"ovnkube-node-bctst\" (UID: \"71312c5c-a477-4500-996f-4c7cffd7fdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bctst" Jul 21 00:40:54.740829 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.738631 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71312c5c-a477-4500-996f-4c7cffd7fdc6-etc-openvswitch\") pod \"ovnkube-node-bctst\" (UID: \"71312c5c-a477-4500-996f-4c7cffd7fdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bctst" Jul 21 00:40:54.740829 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.738674 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/71312c5c-a477-4500-996f-4c7cffd7fdc6-run-ovn\") pod \"ovnkube-node-bctst\" (UID: \"71312c5c-a477-4500-996f-4c7cffd7fdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bctst" Jul 21 00:40:54.740829 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.738781 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 21 00:40:54.740829 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.738792 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/71312c5c-a477-4500-996f-4c7cffd7fdc6-host-cni-netd\") pod \"ovnkube-node-bctst\" (UID: \"71312c5c-a477-4500-996f-4c7cffd7fdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bctst" Jul 21 00:40:54.740829 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.738824 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/71312c5c-a477-4500-996f-4c7cffd7fdc6-ovnkube-config\") pod \"ovnkube-node-bctst\" (UID: \"71312c5c-a477-4500-996f-4c7cffd7fdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bctst" Jul 21 00:40:54.740829 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.738855 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/71312c5c-a477-4500-996f-4c7cffd7fdc6-ovnkube-script-lib\") pod \"ovnkube-node-bctst\" (UID: \"71312c5c-a477-4500-996f-4c7cffd7fdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bctst" Jul 21 00:40:54.740829 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.738902 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/71312c5c-a477-4500-996f-4c7cffd7fdc6-host-run-netns\") pod \"ovnkube-node-bctst\" (UID: \"71312c5c-a477-4500-996f-4c7cffd7fdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bctst" Jul 21 00:40:54.741474 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.738930 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71312c5c-a477-4500-996f-4c7cffd7fdc6-run-openvswitch\") pod \"ovnkube-node-bctst\" (UID: \"71312c5c-a477-4500-996f-4c7cffd7fdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bctst" Jul 21 00:40:54.741474 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.738965 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/71312c5c-a477-4500-996f-4c7cffd7fdc6-ovn-node-metrics-cert\") pod \"ovnkube-node-bctst\" (UID: \"71312c5c-a477-4500-996f-4c7cffd7fdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bctst" Jul 21 00:40:54.741474 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.738971 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-hxcdg\"" Jul 21 00:40:54.741474 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.738992 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0b7ed92b-7e21-40b9-9ac6-f233a11168d0-etc-sysconfig\") pod \"tuned-jwqsv\" (UID: \"0b7ed92b-7e21-40b9-9ac6-f233a11168d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jwqsv" Jul 21 00:40:54.741474 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.739012 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 21 00:40:54.741474 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.739030 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b7ed92b-7e21-40b9-9ac6-f233a11168d0-var-lib-kubelet\") pod \"tuned-jwqsv\" (UID: \"0b7ed92b-7e21-40b9-9ac6-f233a11168d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jwqsv" Jul 21 00:40:54.741474 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.739056 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0b7ed92b-7e21-40b9-9ac6-f233a11168d0-etc-tuned\") pod \"tuned-jwqsv\" (UID: \"0b7ed92b-7e21-40b9-9ac6-f233a11168d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jwqsv" Jul 21 00:40:54.741474 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.739105 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/71312c5c-a477-4500-996f-4c7cffd7fdc6-host-run-ovn-kubernetes\") pod \"ovnkube-node-bctst\" (UID: \"71312c5c-a477-4500-996f-4c7cffd7fdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bctst" Jul 21 00:40:54.741474 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.739193 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 21 00:40:54.741474 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.739387 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzcmw" Jul 21 00:40:54.741474 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:40:54.739457 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dzcmw" podUID="fda1cd2e-3d5d-4dc1-94a1-1b8ebbb4a199" Jul 21 00:40:54.742733 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.741722 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-krnqh" Jul 21 00:40:54.742733 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.742226 2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 21 00:40:54.993910 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.744454 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-jbtdf\"" Jul 21 00:40:54.993910 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.744761 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6zg78" Jul 21 00:40:54.993910 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.744810 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 21 00:40:54.993910 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.745112 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 21 00:40:54.993910 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.745328 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 21 00:40:54.993910 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.745362 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 21 00:40:54.993910 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.747380 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2rxh4" Jul 21 00:40:54.993910 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.747970 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 21 00:40:54.993910 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.748381 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-gttlc\"" Jul 21 00:40:54.993910 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.749208 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 21 00:40:54.993910 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.750140 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9r2ff" Jul 21 00:40:54.993910 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:40:54.750205 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9r2ff" podUID="e11587d4-f449-4b99-8355-787a52243ba4" Jul 21 00:40:54.993910 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.752719 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-858wv" Jul 21 00:40:54.993910 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.753894 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 21 00:40:54.993910 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.753961 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 21 00:40:54.993910 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.754079 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-b824b\"" Jul 21 00:40:54.993910 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.754455 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 21 00:40:54.993910 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.755424 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 21 00:40:54.993910 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.755654 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 21 00:40:54.993910 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.756573 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ntk2c" Jul 21 00:40:54.993910 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.757798 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-tzdgk\"" Jul 21 00:40:54.993910 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.759386 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-9l79b\"" Jul 21 00:40:54.993910 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.759562 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 21 00:40:54.993910 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.760060 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 21 00:40:54.993910 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.774921 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-scxlp" Jul 21 00:40:54.993910 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.792345 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-scxlp" Jul 21 00:40:54.993910 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.827149 2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 21 00:40:54.993910 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.839830 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/245f1c61-31ff-4104-8fa1-5d970dd40533-etc-selinux\") pod \"aws-ebs-csi-driver-node-qthms\" (UID: \"245f1c61-31ff-4104-8fa1-5d970dd40533\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qthms" Jul 21 00:40:54.993910 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.839860 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0b7ed92b-7e21-40b9-9ac6-f233a11168d0-etc-sysconfig\") pod \"tuned-jwqsv\" (UID: \"0b7ed92b-7e21-40b9-9ac6-f233a11168d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jwqsv" Jul 21 00:40:54.995845 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.839879 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b7ed92b-7e21-40b9-9ac6-f233a11168d0-var-lib-kubelet\") pod \"tuned-jwqsv\" (UID: \"0b7ed92b-7e21-40b9-9ac6-f233a11168d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jwqsv" Jul 21 00:40:54.995845 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.839897 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f3e26871-c996-46c1-adfb-c96c1869814b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6zg78\" (UID: \"f3e26871-c996-46c1-adfb-c96c1869814b\") " pod="openshift-multus/multus-additional-cni-plugins-6zg78" Jul 21 00:40:54.995845 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.839929 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e7629fc1-b058-4431-b0d3-1b465dee3b48-konnectivity-ca\") pod \"konnectivity-agent-ntk2c\" (UID: \"e7629fc1-b058-4431-b0d3-1b465dee3b48\") " pod="kube-system/konnectivity-agent-ntk2c" Jul 21 00:40:54.995845 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.839947 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/70177b3d-5d28-4be4-a20d-63c49744b652-iptables-alerter-script\") pod \"iptables-alerter-fj98d\" (UID: \"70177b3d-5d28-4be4-a20d-63c49744b652\") " pod="openshift-network-operator/iptables-alerter-fj98d" Jul 21 00:40:54.995845 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.839959 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b7ed92b-7e21-40b9-9ac6-f233a11168d0-var-lib-kubelet\") pod \"tuned-jwqsv\" (UID: \"0b7ed92b-7e21-40b9-9ac6-f233a11168d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jwqsv" Jul 21 00:40:54.995845 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.839960 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0b7ed92b-7e21-40b9-9ac6-f233a11168d0-etc-sysconfig\") pod \"tuned-jwqsv\" (UID: \"0b7ed92b-7e21-40b9-9ac6-f233a11168d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jwqsv" Jul 21 00:40:54.995845 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.839991 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/71312c5c-a477-4500-996f-4c7cffd7fdc6-node-log\") pod \"ovnkube-node-bctst\" (UID: \"71312c5c-a477-4500-996f-4c7cffd7fdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bctst" Jul 21 00:40:54.995845 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.839965 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/71312c5c-a477-4500-996f-4c7cffd7fdc6-node-log\") pod \"ovnkube-node-bctst\" (UID: \"71312c5c-a477-4500-996f-4c7cffd7fdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bctst" Jul 21 00:40:54.995845 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.840022 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f3e26871-c996-46c1-adfb-c96c1869814b-os-release\") pod \"multus-additional-cni-plugins-6zg78\" (UID: \"f3e26871-c996-46c1-adfb-c96c1869814b\") " pod="openshift-multus/multus-additional-cni-plugins-6zg78" Jul 21 00:40:54.995845 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.840039 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df3aad65-414d-4dd0-9e23-c0c0e46e4f06-tmp-dir\") pod \"node-resolver-858wv\" (UID: \"df3aad65-414d-4dd0-9e23-c0c0e46e4f06\") " pod="openshift-dns/node-resolver-858wv" Jul 21 00:40:54.995845 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.840053 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9aeff916-a90a-49ae-9ac5-0e7a46d7f272-host-var-lib-cni-multus\") pod \"multus-krnqh\" (UID: \"9aeff916-a90a-49ae-9ac5-0e7a46d7f272\") " pod="openshift-multus/multus-krnqh" Jul 21 00:40:54.995845 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.840068 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/245f1c61-31ff-4104-8fa1-5d970dd40533-registration-dir\") pod \"aws-ebs-csi-driver-node-qthms\" (UID: \"245f1c61-31ff-4104-8fa1-5d970dd40533\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qthms" Jul 21 00:40:54.995845 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.840085 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f3e26871-c996-46c1-adfb-c96c1869814b-system-cni-dir\") pod \"multus-additional-cni-plugins-6zg78\" (UID: \"f3e26871-c996-46c1-adfb-c96c1869814b\") " pod="openshift-multus/multus-additional-cni-plugins-6zg78" Jul 21 00:40:54.995845 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.840117 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtd5t\" (UniqueName: \"kubernetes.io/projected/f3e26871-c996-46c1-adfb-c96c1869814b-kube-api-access-jtd5t\") pod \"multus-additional-cni-plugins-6zg78\" (UID: \"f3e26871-c996-46c1-adfb-c96c1869814b\") " pod="openshift-multus/multus-additional-cni-plugins-6zg78" Jul 21 00:40:54.996393 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.840133 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9aeff916-a90a-49ae-9ac5-0e7a46d7f272-host-var-lib-cni-bin\") pod \"multus-krnqh\" (UID: \"9aeff916-a90a-49ae-9ac5-0e7a46d7f272\") " pod="openshift-multus/multus-krnqh" Jul 21 00:40:54.996393 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.840148 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/245f1c61-31ff-4104-8fa1-5d970dd40533-device-dir\") pod \"aws-ebs-csi-driver-node-qthms\" (UID: \"245f1c61-31ff-4104-8fa1-5d970dd40533\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qthms" Jul 21 00:40:54.996393 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.840177 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0b7ed92b-7e21-40b9-9ac6-f233a11168d0-run\") pod \"tuned-jwqsv\" (UID: \"0b7ed92b-7e21-40b9-9ac6-f233a11168d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jwqsv" Jul 21 00:40:54.996393 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.840205 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2f98\" (UniqueName: \"kubernetes.io/projected/71312c5c-a477-4500-996f-4c7cffd7fdc6-kube-api-access-m2f98\") pod \"ovnkube-node-bctst\" (UID: \"71312c5c-a477-4500-996f-4c7cffd7fdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bctst" Jul 21 00:40:54.996393 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.840263 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b7ed92b-7e21-40b9-9ac6-f233a11168d0-etc-kubernetes\") pod \"tuned-jwqsv\" (UID: \"0b7ed92b-7e21-40b9-9ac6-f233a11168d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jwqsv" Jul 21 00:40:54.996393 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.840278 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0b7ed92b-7e21-40b9-9ac6-f233a11168d0-run\") pod \"tuned-jwqsv\" (UID: \"0b7ed92b-7e21-40b9-9ac6-f233a11168d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jwqsv" Jul 21 00:40:54.996393 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.840310 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71312c5c-a477-4500-996f-4c7cffd7fdc6-var-lib-openvswitch\") pod \"ovnkube-node-bctst\" (UID: \"71312c5c-a477-4500-996f-4c7cffd7fdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bctst" Jul 21 00:40:54.996393 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.840302 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b7ed92b-7e21-40b9-9ac6-f233a11168d0-etc-kubernetes\") pod \"tuned-jwqsv\" (UID: \"0b7ed92b-7e21-40b9-9ac6-f233a11168d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jwqsv" Jul 21 00:40:54.996393 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.840346 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/71312c5c-a477-4500-996f-4c7cffd7fdc6-log-socket\") pod \"ovnkube-node-bctst\" (UID: \"71312c5c-a477-4500-996f-4c7cffd7fdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bctst" Jul 21 00:40:54.996393 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.840355 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71312c5c-a477-4500-996f-4c7cffd7fdc6-var-lib-openvswitch\") pod \"ovnkube-node-bctst\" (UID: \"71312c5c-a477-4500-996f-4c7cffd7fdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bctst" Jul 21 00:40:54.996393 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.840375 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/71312c5c-a477-4500-996f-4c7cffd7fdc6-log-socket\") pod \"ovnkube-node-bctst\" (UID: \"71312c5c-a477-4500-996f-4c7cffd7fdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bctst" Jul 21 00:40:54.996393 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.840393 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/71312c5c-a477-4500-996f-4c7cffd7fdc6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bctst\" (UID: \"71312c5c-a477-4500-996f-4c7cffd7fdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bctst" Jul 21 00:40:54.996393 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.840415 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9aeff916-a90a-49ae-9ac5-0e7a46d7f272-cnibin\") pod \"multus-krnqh\" (UID: \"9aeff916-a90a-49ae-9ac5-0e7a46d7f272\") " pod="openshift-multus/multus-krnqh" Jul 21 00:40:54.996393 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.840430 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9aeff916-a90a-49ae-9ac5-0e7a46d7f272-os-release\") pod \"multus-krnqh\" (UID: \"9aeff916-a90a-49ae-9ac5-0e7a46d7f272\") " pod="openshift-multus/multus-krnqh" Jul 21 00:40:54.997054 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.840437 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/71312c5c-a477-4500-996f-4c7cffd7fdc6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bctst\" (UID: \"71312c5c-a477-4500-996f-4c7cffd7fdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bctst" Jul 21 00:40:54.997054 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.840446 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/71312c5c-a477-4500-996f-4c7cffd7fdc6-systemd-units\") pod \"ovnkube-node-bctst\" (UID: \"71312c5c-a477-4500-996f-4c7cffd7fdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bctst" Jul 21 00:40:54.997054 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.840462 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0b7ed92b-7e21-40b9-9ac6-f233a11168d0-sys\") pod \"tuned-jwqsv\" (UID: \"0b7ed92b-7e21-40b9-9ac6-f233a11168d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jwqsv" Jul 21 00:40:54.997054 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.840478 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0b7ed92b-7e21-40b9-9ac6-f233a11168d0-host\") pod \"tuned-jwqsv\" (UID: \"0b7ed92b-7e21-40b9-9ac6-f233a11168d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jwqsv" Jul 21 00:40:54.997054 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.840484 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/71312c5c-a477-4500-996f-4c7cffd7fdc6-systemd-units\") pod \"ovnkube-node-bctst\" (UID: \"71312c5c-a477-4500-996f-4c7cffd7fdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bctst" Jul 21 00:40:54.997054 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.840492 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/71312c5c-a477-4500-996f-4c7cffd7fdc6-host-cni-bin\") pod \"ovnkube-node-bctst\" (UID: \"71312c5c-a477-4500-996f-4c7cffd7fdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bctst" Jul 21 00:40:54.997054 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.840508 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f3e26871-c996-46c1-adfb-c96c1869814b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6zg78\" (UID: \"f3e26871-c996-46c1-adfb-c96c1869814b\") " pod="openshift-multus/multus-additional-cni-plugins-6zg78" Jul 21 00:40:54.997054 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.840518 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0b7ed92b-7e21-40b9-9ac6-f233a11168d0-sys\") pod \"tuned-jwqsv\" (UID: \"0b7ed92b-7e21-40b9-9ac6-f233a11168d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jwqsv" Jul 21 00:40:54.997054 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.840524 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6fbd5477-ce7d-444d-938b-2c54aa3c040c-serviceca\") pod \"node-ca-2rxh4\" (UID: \"6fbd5477-ce7d-444d-938b-2c54aa3c040c\") " pod="openshift-image-registry/node-ca-2rxh4" Jul 21 00:40:54.997054 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.840548 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/71312c5c-a477-4500-996f-4c7cffd7fdc6-ovnkube-script-lib\") pod \"ovnkube-node-bctst\" (UID: \"71312c5c-a477-4500-996f-4c7cffd7fdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bctst" Jul 21 00:40:54.997054 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.840552 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0b7ed92b-7e21-40b9-9ac6-f233a11168d0-host\") pod \"tuned-jwqsv\" (UID: \"0b7ed92b-7e21-40b9-9ac6-f233a11168d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jwqsv" Jul 21 00:40:54.997054 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.840578 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/71312c5c-a477-4500-996f-4c7cffd7fdc6-host-cni-bin\") pod \"ovnkube-node-bctst\" (UID: \"71312c5c-a477-4500-996f-4c7cffd7fdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bctst" Jul 21 00:40:54.997054 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.840581 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fda1cd2e-3d5d-4dc1-94a1-1b8ebbb4a199-original-pull-secret\") pod \"global-pull-secret-syncer-dzcmw\" (UID: \"fda1cd2e-3d5d-4dc1-94a1-1b8ebbb4a199\") " pod="kube-system/global-pull-secret-syncer-dzcmw" Jul 21 00:40:54.997054 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.840624 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/df3aad65-414d-4dd0-9e23-c0c0e46e4f06-hosts-file\") pod \"node-resolver-858wv\" (UID: \"df3aad65-414d-4dd0-9e23-c0c0e46e4f06\") " pod="openshift-dns/node-resolver-858wv" Jul 21 00:40:54.997570 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.840644 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9aeff916-a90a-49ae-9ac5-0e7a46d7f272-etc-kubernetes\") pod \"multus-krnqh\" (UID: \"9aeff916-a90a-49ae-9ac5-0e7a46d7f272\") " pod="openshift-multus/multus-krnqh" Jul 21 00:40:54.997570 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.840679 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/245f1c61-31ff-4104-8fa1-5d970dd40533-sys-fs\") pod \"aws-ebs-csi-driver-node-qthms\" (UID: \"245f1c61-31ff-4104-8fa1-5d970dd40533\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qthms" Jul 21 00:40:54.997570 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.840700 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/71312c5c-a477-4500-996f-4c7cffd7fdc6-host-run-netns\") pod \"ovnkube-node-bctst\" (UID: \"71312c5c-a477-4500-996f-4c7cffd7fdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bctst" Jul 21 00:40:54.997570 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.840715 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9aeff916-a90a-49ae-9ac5-0e7a46d7f272-multus-daemon-config\") pod \"multus-krnqh\" (UID: \"9aeff916-a90a-49ae-9ac5-0e7a46d7f272\") " pod="openshift-multus/multus-krnqh" Jul 21 00:40:54.997570 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.840738 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/71312c5c-a477-4500-996f-4c7cffd7fdc6-host-run-netns\") pod \"ovnkube-node-bctst\" (UID: \"71312c5c-a477-4500-996f-4c7cffd7fdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bctst" Jul 21 00:40:54.997570 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.840741 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9aeff916-a90a-49ae-9ac5-0e7a46d7f272-host-run-multus-certs\") pod \"multus-krnqh\" (UID: \"9aeff916-a90a-49ae-9ac5-0e7a46d7f272\") " pod="openshift-multus/multus-krnqh" Jul 21 00:40:54.997570 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.840765 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e7629fc1-b058-4431-b0d3-1b465dee3b48-agent-certs\") pod \"konnectivity-agent-ntk2c\" (UID: \"e7629fc1-b058-4431-b0d3-1b465dee3b48\") " pod="kube-system/konnectivity-agent-ntk2c" Jul 21 00:40:54.997570 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.840801 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfsv5\" (UniqueName: \"kubernetes.io/projected/6cfb338e-19eb-45ee-a3de-ea7311469109-kube-api-access-mfsv5\") pod \"network-metrics-daemon-4nq5b\" (UID: \"6cfb338e-19eb-45ee-a3de-ea7311469109\") " pod="openshift-multus/network-metrics-daemon-4nq5b" Jul 21 00:40:54.997570 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.840836 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/70177b3d-5d28-4be4-a20d-63c49744b652-host-slash\") pod \"iptables-alerter-fj98d\" (UID: \"70177b3d-5d28-4be4-a20d-63c49744b652\") " pod="openshift-network-operator/iptables-alerter-fj98d" Jul 21 00:40:54.997570 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.840859 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0b7ed92b-7e21-40b9-9ac6-f233a11168d0-etc-tuned\") pod \"tuned-jwqsv\" (UID: \"0b7ed92b-7e21-40b9-9ac6-f233a11168d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jwqsv" Jul 21 00:40:54.997570 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.840877 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/71312c5c-a477-4500-996f-4c7cffd7fdc6-host-run-ovn-kubernetes\") pod \"ovnkube-node-bctst\" (UID: \"71312c5c-a477-4500-996f-4c7cffd7fdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bctst" Jul 21 00:40:54.997570 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.840893 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fda1cd2e-3d5d-4dc1-94a1-1b8ebbb4a199-kubelet-config\") pod \"global-pull-secret-syncer-dzcmw\" (UID: \"fda1cd2e-3d5d-4dc1-94a1-1b8ebbb4a199\") " pod="kube-system/global-pull-secret-syncer-dzcmw" Jul 21 00:40:54.997570 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.840908 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fda1cd2e-3d5d-4dc1-94a1-1b8ebbb4a199-dbus\") pod \"global-pull-secret-syncer-dzcmw\" (UID: \"fda1cd2e-3d5d-4dc1-94a1-1b8ebbb4a199\") " pod="kube-system/global-pull-secret-syncer-dzcmw" Jul 21 00:40:54.997570 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.840922 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lq8n8\" (UniqueName: \"kubernetes.io/projected/9aeff916-a90a-49ae-9ac5-0e7a46d7f272-kube-api-access-lq8n8\") pod \"multus-krnqh\" (UID: \"9aeff916-a90a-49ae-9ac5-0e7a46d7f272\") " pod="openshift-multus/multus-krnqh" Jul 21 00:40:54.998165 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.840940 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6fbd5477-ce7d-444d-938b-2c54aa3c040c-host\") pod \"node-ca-2rxh4\" (UID: \"6fbd5477-ce7d-444d-938b-2c54aa3c040c\") " pod="openshift-image-registry/node-ca-2rxh4" Jul 21 00:40:54.998165 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.840959 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/71312c5c-a477-4500-996f-4c7cffd7fdc6-host-run-ovn-kubernetes\") pod \"ovnkube-node-bctst\" (UID: \"71312c5c-a477-4500-996f-4c7cffd7fdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bctst" Jul 21 00:40:54.998165 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.840983 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/71312c5c-a477-4500-996f-4c7cffd7fdc6-host-slash\") pod \"ovnkube-node-bctst\" (UID: \"71312c5c-a477-4500-996f-4c7cffd7fdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bctst" Jul 21 00:40:54.998165 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.841012 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0b7ed92b-7e21-40b9-9ac6-f233a11168d0-etc-sysctl-d\") pod \"tuned-jwqsv\" (UID: \"0b7ed92b-7e21-40b9-9ac6-f233a11168d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jwqsv" Jul 21 00:40:54.998165 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.841022 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/71312c5c-a477-4500-996f-4c7cffd7fdc6-host-slash\") pod \"ovnkube-node-bctst\" (UID: \"71312c5c-a477-4500-996f-4c7cffd7fdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bctst" Jul 21 00:40:54.998165 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.841039 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0b7ed92b-7e21-40b9-9ac6-f233a11168d0-etc-sysctl-conf\") pod \"tuned-jwqsv\" (UID: \"0b7ed92b-7e21-40b9-9ac6-f233a11168d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jwqsv" Jul 21 00:40:54.998165 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.841062 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wb98\" (UniqueName: \"kubernetes.io/projected/df3aad65-414d-4dd0-9e23-c0c0e46e4f06-kube-api-access-6wb98\") pod \"node-resolver-858wv\" (UID: \"df3aad65-414d-4dd0-9e23-c0c0e46e4f06\") " pod="openshift-dns/node-resolver-858wv" Jul 21 00:40:54.998165 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.841080 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9aeff916-a90a-49ae-9ac5-0e7a46d7f272-hostroot\") pod \"multus-krnqh\" (UID: \"9aeff916-a90a-49ae-9ac5-0e7a46d7f272\") " pod="openshift-multus/multus-krnqh" Jul 21 00:40:54.998165 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.841089 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0b7ed92b-7e21-40b9-9ac6-f233a11168d0-etc-sysctl-d\") pod \"tuned-jwqsv\" (UID: \"0b7ed92b-7e21-40b9-9ac6-f233a11168d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jwqsv" Jul 21 00:40:54.998165 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.841097 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/245f1c61-31ff-4104-8fa1-5d970dd40533-socket-dir\") pod \"aws-ebs-csi-driver-node-qthms\" (UID: \"245f1c61-31ff-4104-8fa1-5d970dd40533\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qthms" Jul 21 00:40:54.998165 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.841125 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zd6rj\" (UniqueName: \"kubernetes.io/projected/245f1c61-31ff-4104-8fa1-5d970dd40533-kube-api-access-zd6rj\") pod \"aws-ebs-csi-driver-node-qthms\" (UID: \"245f1c61-31ff-4104-8fa1-5d970dd40533\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qthms" Jul 21 00:40:54.998165 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.841146 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0b7ed92b-7e21-40b9-9ac6-f233a11168d0-lib-modules\") pod \"tuned-jwqsv\" (UID: \"0b7ed92b-7e21-40b9-9ac6-f233a11168d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jwqsv" Jul 21 00:40:54.998165 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.841166 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b7ed92b-7e21-40b9-9ac6-f233a11168d0-tmp\") pod \"tuned-jwqsv\" (UID: \"0b7ed92b-7e21-40b9-9ac6-f233a11168d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jwqsv" Jul 21 00:40:54.998165 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.841146 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0b7ed92b-7e21-40b9-9ac6-f233a11168d0-etc-sysctl-conf\") pod \"tuned-jwqsv\" (UID: \"0b7ed92b-7e21-40b9-9ac6-f233a11168d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jwqsv" Jul 21 00:40:54.998712 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.841183 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch652\" (UniqueName: \"kubernetes.io/projected/70177b3d-5d28-4be4-a20d-63c49744b652-kube-api-access-ch652\") pod \"iptables-alerter-fj98d\" (UID: \"70177b3d-5d28-4be4-a20d-63c49744b652\") " pod="openshift-network-operator/iptables-alerter-fj98d" Jul 21 00:40:54.998712 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.841190 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/71312c5c-a477-4500-996f-4c7cffd7fdc6-ovnkube-script-lib\") pod \"ovnkube-node-bctst\" (UID: \"71312c5c-a477-4500-996f-4c7cffd7fdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bctst" Jul 21 00:40:54.998712 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.841203 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9aeff916-a90a-49ae-9ac5-0e7a46d7f272-system-cni-dir\") pod \"multus-krnqh\" (UID: \"9aeff916-a90a-49ae-9ac5-0e7a46d7f272\") " pod="openshift-multus/multus-krnqh" Jul 21 00:40:54.998712 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.841222 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9aeff916-a90a-49ae-9ac5-0e7a46d7f272-multus-socket-dir-parent\") pod \"multus-krnqh\" (UID: \"9aeff916-a90a-49ae-9ac5-0e7a46d7f272\") " pod="openshift-multus/multus-krnqh" Jul 21 00:40:54.998712 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.841245 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9aeff916-a90a-49ae-9ac5-0e7a46d7f272-host-run-k8s-cni-cncf-io\") pod \"multus-krnqh\" (UID: \"9aeff916-a90a-49ae-9ac5-0e7a46d7f272\") " pod="openshift-multus/multus-krnqh" Jul 21 00:40:54.998712 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.841216 2580 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 21 00:40:54.998712 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.841300 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0b7ed92b-7e21-40b9-9ac6-f233a11168d0-etc-systemd\") pod \"tuned-jwqsv\" (UID: \"0b7ed92b-7e21-40b9-9ac6-f233a11168d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jwqsv" Jul 21 00:40:54.998712 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.841310 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0b7ed92b-7e21-40b9-9ac6-f233a11168d0-lib-modules\") pod \"tuned-jwqsv\" (UID: \"0b7ed92b-7e21-40b9-9ac6-f233a11168d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jwqsv" Jul 21 00:40:54.998712 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.841343 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9aeff916-a90a-49ae-9ac5-0e7a46d7f272-host-var-lib-kubelet\") pod \"multus-krnqh\" (UID: \"9aeff916-a90a-49ae-9ac5-0e7a46d7f272\") " pod="openshift-multus/multus-krnqh" Jul 21 00:40:54.998712 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.841353 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0b7ed92b-7e21-40b9-9ac6-f233a11168d0-etc-systemd\") pod \"tuned-jwqsv\" (UID: \"0b7ed92b-7e21-40b9-9ac6-f233a11168d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jwqsv" Jul 21 00:40:54.998712 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.841360 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62qdg\" (UniqueName: \"kubernetes.io/projected/6fbd5477-ce7d-444d-938b-2c54aa3c040c-kube-api-access-62qdg\") pod \"node-ca-2rxh4\" (UID: \"6fbd5477-ce7d-444d-938b-2c54aa3c040c\") " pod="openshift-image-registry/node-ca-2rxh4" Jul 21 00:40:54.998712 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.841382 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0b7ed92b-7e21-40b9-9ac6-f233a11168d0-etc-modprobe-d\") pod \"tuned-jwqsv\" (UID: \"0b7ed92b-7e21-40b9-9ac6-f233a11168d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jwqsv" Jul 21 00:40:54.998712 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.841404 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/71312c5c-a477-4500-996f-4c7cffd7fdc6-env-overrides\") pod \"ovnkube-node-bctst\" (UID: \"71312c5c-a477-4500-996f-4c7cffd7fdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bctst" Jul 21 00:40:54.998712 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.841419 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f3e26871-c996-46c1-adfb-c96c1869814b-cnibin\") pod \"multus-additional-cni-plugins-6zg78\" (UID: \"f3e26871-c996-46c1-adfb-c96c1869814b\") " pod="openshift-multus/multus-additional-cni-plugins-6zg78" Jul 21 00:40:54.999266 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.841434 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wxxq\" (UniqueName: \"kubernetes.io/projected/e11587d4-f449-4b99-8355-787a52243ba4-kube-api-access-8wxxq\") pod \"network-check-target-9r2ff\" (UID: \"e11587d4-f449-4b99-8355-787a52243ba4\") " pod="openshift-network-diagnostics/network-check-target-9r2ff" Jul 21 00:40:54.999266 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.841451 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9aeff916-a90a-49ae-9ac5-0e7a46d7f272-multus-conf-dir\") pod \"multus-krnqh\" (UID: \"9aeff916-a90a-49ae-9ac5-0e7a46d7f272\") " pod="openshift-multus/multus-krnqh" Jul 21 00:40:54.999266 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.841472 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0b7ed92b-7e21-40b9-9ac6-f233a11168d0-etc-modprobe-d\") pod \"tuned-jwqsv\" (UID: \"0b7ed92b-7e21-40b9-9ac6-f233a11168d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jwqsv" Jul 21 00:40:54.999266 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.841487 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cfb338e-19eb-45ee-a3de-ea7311469109-metrics-certs\") pod \"network-metrics-daemon-4nq5b\" (UID: \"6cfb338e-19eb-45ee-a3de-ea7311469109\") " pod="openshift-multus/network-metrics-daemon-4nq5b" Jul 21 00:40:54.999266 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.841531 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/71312c5c-a477-4500-996f-4c7cffd7fdc6-run-systemd\") pod \"ovnkube-node-bctst\" (UID: \"71312c5c-a477-4500-996f-4c7cffd7fdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bctst" Jul 21 00:40:54.999266 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.841555 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kvss\" (UniqueName: \"kubernetes.io/projected/0b7ed92b-7e21-40b9-9ac6-f233a11168d0-kube-api-access-8kvss\") pod \"tuned-jwqsv\" (UID: \"0b7ed92b-7e21-40b9-9ac6-f233a11168d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jwqsv" Jul 21 00:40:54.999266 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.841583 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/71312c5c-a477-4500-996f-4c7cffd7fdc6-host-kubelet\") pod \"ovnkube-node-bctst\" (UID: \"71312c5c-a477-4500-996f-4c7cffd7fdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bctst" Jul 21 00:40:54.999266 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.841587 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/71312c5c-a477-4500-996f-4c7cffd7fdc6-run-systemd\") pod \"ovnkube-node-bctst\" (UID: \"71312c5c-a477-4500-996f-4c7cffd7fdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bctst" Jul 21 00:40:54.999266 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.841615 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/245f1c61-31ff-4104-8fa1-5d970dd40533-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qthms\" (UID: \"245f1c61-31ff-4104-8fa1-5d970dd40533\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qthms" Jul 21 00:40:54.999266 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.841627 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/71312c5c-a477-4500-996f-4c7cffd7fdc6-host-kubelet\") pod \"ovnkube-node-bctst\" (UID: \"71312c5c-a477-4500-996f-4c7cffd7fdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bctst" Jul 21 00:40:54.999266 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.841643 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71312c5c-a477-4500-996f-4c7cffd7fdc6-etc-openvswitch\") pod \"ovnkube-node-bctst\" (UID: \"71312c5c-a477-4500-996f-4c7cffd7fdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bctst" Jul 21 00:40:54.999266 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.841666 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/71312c5c-a477-4500-996f-4c7cffd7fdc6-run-ovn\") pod \"ovnkube-node-bctst\" (UID: \"71312c5c-a477-4500-996f-4c7cffd7fdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bctst" Jul 21 00:40:54.999266 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.841683 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/71312c5c-a477-4500-996f-4c7cffd7fdc6-host-cni-netd\") pod \"ovnkube-node-bctst\" (UID: \"71312c5c-a477-4500-996f-4c7cffd7fdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bctst" Jul 21 00:40:54.999266 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.841698 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/71312c5c-a477-4500-996f-4c7cffd7fdc6-ovnkube-config\") pod \"ovnkube-node-bctst\" (UID: \"71312c5c-a477-4500-996f-4c7cffd7fdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bctst" Jul 21 00:40:54.999776 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.841719 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f3e26871-c996-46c1-adfb-c96c1869814b-cni-binary-copy\") pod \"multus-additional-cni-plugins-6zg78\" (UID: \"f3e26871-c996-46c1-adfb-c96c1869814b\") " pod="openshift-multus/multus-additional-cni-plugins-6zg78" Jul 21 00:40:54.999776 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.841748 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/71312c5c-a477-4500-996f-4c7cffd7fdc6-host-cni-netd\") pod \"ovnkube-node-bctst\" (UID: \"71312c5c-a477-4500-996f-4c7cffd7fdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bctst" Jul 21 00:40:54.999776 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.841750 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9aeff916-a90a-49ae-9ac5-0e7a46d7f272-multus-cni-dir\") pod \"multus-krnqh\" (UID: \"9aeff916-a90a-49ae-9ac5-0e7a46d7f272\") " pod="openshift-multus/multus-krnqh" Jul 21 00:40:54.999776 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.841750 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71312c5c-a477-4500-996f-4c7cffd7fdc6-etc-openvswitch\") pod \"ovnkube-node-bctst\" (UID: \"71312c5c-a477-4500-996f-4c7cffd7fdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bctst" Jul 21 00:40:54.999776 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.841793 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/71312c5c-a477-4500-996f-4c7cffd7fdc6-run-ovn\") pod \"ovnkube-node-bctst\" (UID: \"71312c5c-a477-4500-996f-4c7cffd7fdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bctst" Jul 21 00:40:54.999776 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.841787 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9aeff916-a90a-49ae-9ac5-0e7a46d7f272-cni-binary-copy\") pod \"multus-krnqh\" (UID: \"9aeff916-a90a-49ae-9ac5-0e7a46d7f272\") " pod="openshift-multus/multus-krnqh" Jul 21 00:40:54.999776 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.841836 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71312c5c-a477-4500-996f-4c7cffd7fdc6-run-openvswitch\") pod \"ovnkube-node-bctst\" (UID: \"71312c5c-a477-4500-996f-4c7cffd7fdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bctst" Jul 21 00:40:54.999776 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.841862 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/71312c5c-a477-4500-996f-4c7cffd7fdc6-ovn-node-metrics-cert\") pod \"ovnkube-node-bctst\" (UID: \"71312c5c-a477-4500-996f-4c7cffd7fdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bctst" Jul 21 00:40:54.999776 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.841864 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/71312c5c-a477-4500-996f-4c7cffd7fdc6-env-overrides\") pod \"ovnkube-node-bctst\" (UID: \"71312c5c-a477-4500-996f-4c7cffd7fdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bctst" Jul 21 00:40:54.999776 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.841892 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f3e26871-c996-46c1-adfb-c96c1869814b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6zg78\" (UID: \"f3e26871-c996-46c1-adfb-c96c1869814b\") " pod="openshift-multus/multus-additional-cni-plugins-6zg78" Jul 21 00:40:54.999776 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.841899 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71312c5c-a477-4500-996f-4c7cffd7fdc6-run-openvswitch\") pod \"ovnkube-node-bctst\" (UID: \"71312c5c-a477-4500-996f-4c7cffd7fdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bctst" Jul 21 00:40:54.999776 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.841919 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9aeff916-a90a-49ae-9ac5-0e7a46d7f272-host-run-netns\") pod \"multus-krnqh\" (UID: \"9aeff916-a90a-49ae-9ac5-0e7a46d7f272\") " pod="openshift-multus/multus-krnqh" Jul 21 00:40:54.999776 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.842177 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/71312c5c-a477-4500-996f-4c7cffd7fdc6-ovnkube-config\") pod \"ovnkube-node-bctst\" (UID: \"71312c5c-a477-4500-996f-4c7cffd7fdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bctst" Jul 21 00:40:54.999776 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.844464 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0b7ed92b-7e21-40b9-9ac6-f233a11168d0-etc-tuned\") pod \"tuned-jwqsv\" (UID: \"0b7ed92b-7e21-40b9-9ac6-f233a11168d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jwqsv" Jul 21 00:40:55.000251 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.844526 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b7ed92b-7e21-40b9-9ac6-f233a11168d0-tmp\") pod \"tuned-jwqsv\" (UID: \"0b7ed92b-7e21-40b9-9ac6-f233a11168d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jwqsv" Jul 21 00:40:55.000251 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.844690 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/71312c5c-a477-4500-996f-4c7cffd7fdc6-ovn-node-metrics-cert\") pod \"ovnkube-node-bctst\" (UID: \"71312c5c-a477-4500-996f-4c7cffd7fdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bctst" Jul 21 00:40:55.000251 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:54.847369 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cf3a63f5e16fba7955768004cdb0f7c.slice/crio-bd690e61f074b75a79322c1ac747e548d9079e78c28b6ac6961d644dd9d6cfd2 WatchSource:0}: Error finding container bd690e61f074b75a79322c1ac747e548d9079e78c28b6ac6961d644dd9d6cfd2: Status 404 returned error can't find the container with id bd690e61f074b75a79322c1ac747e548d9079e78c28b6ac6961d644dd9d6cfd2 Jul 21 00:40:55.000251 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.852121 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kvss\" (UniqueName: \"kubernetes.io/projected/0b7ed92b-7e21-40b9-9ac6-f233a11168d0-kube-api-access-8kvss\") pod \"tuned-jwqsv\" (UID: \"0b7ed92b-7e21-40b9-9ac6-f233a11168d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jwqsv" Jul 21 00:40:55.000251 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.852504 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2f98\" (UniqueName: \"kubernetes.io/projected/71312c5c-a477-4500-996f-4c7cffd7fdc6-kube-api-access-m2f98\") pod \"ovnkube-node-bctst\" (UID: \"71312c5c-a477-4500-996f-4c7cffd7fdc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bctst" Jul 21 00:40:55.000251 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.855683 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 00:40:55.000251 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.911414 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-137.ec2.internal" event={"ID":"2cf3a63f5e16fba7955768004cdb0f7c","Type":"ContainerStarted","Data":"bd690e61f074b75a79322c1ac747e548d9079e78c28b6ac6961d644dd9d6cfd2"} Jul 21 00:40:55.000251 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.942823 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e7629fc1-b058-4431-b0d3-1b465dee3b48-agent-certs\") pod \"konnectivity-agent-ntk2c\" (UID: \"e7629fc1-b058-4431-b0d3-1b465dee3b48\") " pod="kube-system/konnectivity-agent-ntk2c" Jul 21 00:40:55.000251 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.942851 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfsv5\" (UniqueName: \"kubernetes.io/projected/6cfb338e-19eb-45ee-a3de-ea7311469109-kube-api-access-mfsv5\") pod \"network-metrics-daemon-4nq5b\" (UID: \"6cfb338e-19eb-45ee-a3de-ea7311469109\") " pod="openshift-multus/network-metrics-daemon-4nq5b" Jul 21 00:40:55.000251 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.942868 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/70177b3d-5d28-4be4-a20d-63c49744b652-host-slash\") pod \"iptables-alerter-fj98d\" (UID: \"70177b3d-5d28-4be4-a20d-63c49744b652\") " pod="openshift-network-operator/iptables-alerter-fj98d" Jul 21 00:40:55.000251 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.942915 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fda1cd2e-3d5d-4dc1-94a1-1b8ebbb4a199-kubelet-config\") pod \"global-pull-secret-syncer-dzcmw\" (UID: \"fda1cd2e-3d5d-4dc1-94a1-1b8ebbb4a199\") " pod="kube-system/global-pull-secret-syncer-dzcmw" Jul 21 00:40:55.000251 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.942937 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fda1cd2e-3d5d-4dc1-94a1-1b8ebbb4a199-dbus\") pod \"global-pull-secret-syncer-dzcmw\" (UID: \"fda1cd2e-3d5d-4dc1-94a1-1b8ebbb4a199\") " pod="kube-system/global-pull-secret-syncer-dzcmw" Jul 21 00:40:55.000251 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.942946 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/70177b3d-5d28-4be4-a20d-63c49744b652-host-slash\") pod \"iptables-alerter-fj98d\" (UID: \"70177b3d-5d28-4be4-a20d-63c49744b652\") " pod="openshift-network-operator/iptables-alerter-fj98d" Jul 21 00:40:55.000251 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.942955 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lq8n8\" (UniqueName: \"kubernetes.io/projected/9aeff916-a90a-49ae-9ac5-0e7a46d7f272-kube-api-access-lq8n8\") pod \"multus-krnqh\" (UID: \"9aeff916-a90a-49ae-9ac5-0e7a46d7f272\") " pod="openshift-multus/multus-krnqh" Jul 21 00:40:55.000251 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.943008 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fda1cd2e-3d5d-4dc1-94a1-1b8ebbb4a199-kubelet-config\") pod \"global-pull-secret-syncer-dzcmw\" (UID: \"fda1cd2e-3d5d-4dc1-94a1-1b8ebbb4a199\") " pod="kube-system/global-pull-secret-syncer-dzcmw" Jul 21 00:40:55.000740 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.943057 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6fbd5477-ce7d-444d-938b-2c54aa3c040c-host\") pod \"node-ca-2rxh4\" (UID: \"6fbd5477-ce7d-444d-938b-2c54aa3c040c\") " pod="openshift-image-registry/node-ca-2rxh4" Jul 21 00:40:55.000740 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.943084 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wb98\" (UniqueName: \"kubernetes.io/projected/df3aad65-414d-4dd0-9e23-c0c0e46e4f06-kube-api-access-6wb98\") pod \"node-resolver-858wv\" (UID: \"df3aad65-414d-4dd0-9e23-c0c0e46e4f06\") " pod="openshift-dns/node-resolver-858wv" Jul 21 00:40:55.000740 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.943101 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9aeff916-a90a-49ae-9ac5-0e7a46d7f272-hostroot\") pod \"multus-krnqh\" (UID: \"9aeff916-a90a-49ae-9ac5-0e7a46d7f272\") " pod="openshift-multus/multus-krnqh" Jul 21 00:40:55.000740 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.943119 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fda1cd2e-3d5d-4dc1-94a1-1b8ebbb4a199-dbus\") pod \"global-pull-secret-syncer-dzcmw\" (UID: \"fda1cd2e-3d5d-4dc1-94a1-1b8ebbb4a199\") " pod="kube-system/global-pull-secret-syncer-dzcmw" Jul 21 00:40:55.000740 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.943131 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/245f1c61-31ff-4104-8fa1-5d970dd40533-socket-dir\") pod \"aws-ebs-csi-driver-node-qthms\" (UID: \"245f1c61-31ff-4104-8fa1-5d970dd40533\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qthms" Jul 21 00:40:55.000740 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.943138 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6fbd5477-ce7d-444d-938b-2c54aa3c040c-host\") pod \"node-ca-2rxh4\" (UID: \"6fbd5477-ce7d-444d-938b-2c54aa3c040c\") " pod="openshift-image-registry/node-ca-2rxh4" Jul 21 00:40:55.000740 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.943158 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zd6rj\" (UniqueName: \"kubernetes.io/projected/245f1c61-31ff-4104-8fa1-5d970dd40533-kube-api-access-zd6rj\") pod \"aws-ebs-csi-driver-node-qthms\" (UID: \"245f1c61-31ff-4104-8fa1-5d970dd40533\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qthms" Jul 21 00:40:55.000740 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.943190 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch652\" (UniqueName: \"kubernetes.io/projected/70177b3d-5d28-4be4-a20d-63c49744b652-kube-api-access-ch652\") pod \"iptables-alerter-fj98d\" (UID: \"70177b3d-5d28-4be4-a20d-63c49744b652\") " pod="openshift-network-operator/iptables-alerter-fj98d" Jul 21 00:40:55.000740 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.943197 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9aeff916-a90a-49ae-9ac5-0e7a46d7f272-hostroot\") pod \"multus-krnqh\" (UID: \"9aeff916-a90a-49ae-9ac5-0e7a46d7f272\") " pod="openshift-multus/multus-krnqh" Jul 21 00:40:55.000740 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.943217 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9aeff916-a90a-49ae-9ac5-0e7a46d7f272-system-cni-dir\") pod \"multus-krnqh\" (UID: \"9aeff916-a90a-49ae-9ac5-0e7a46d7f272\") " pod="openshift-multus/multus-krnqh" Jul 21 00:40:55.000740 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.943236 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9aeff916-a90a-49ae-9ac5-0e7a46d7f272-multus-socket-dir-parent\") pod \"multus-krnqh\" (UID: \"9aeff916-a90a-49ae-9ac5-0e7a46d7f272\") " pod="openshift-multus/multus-krnqh" Jul 21 00:40:55.000740 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.943306 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9aeff916-a90a-49ae-9ac5-0e7a46d7f272-multus-socket-dir-parent\") pod \"multus-krnqh\" (UID: \"9aeff916-a90a-49ae-9ac5-0e7a46d7f272\") " pod="openshift-multus/multus-krnqh" Jul 21 00:40:55.000740 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.943325 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/245f1c61-31ff-4104-8fa1-5d970dd40533-socket-dir\") pod \"aws-ebs-csi-driver-node-qthms\" (UID: \"245f1c61-31ff-4104-8fa1-5d970dd40533\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qthms" Jul 21 00:40:55.000740 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.943330 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9aeff916-a90a-49ae-9ac5-0e7a46d7f272-system-cni-dir\") pod \"multus-krnqh\" (UID: \"9aeff916-a90a-49ae-9ac5-0e7a46d7f272\") " pod="openshift-multus/multus-krnqh" Jul 21 00:40:55.000740 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.943375 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9aeff916-a90a-49ae-9ac5-0e7a46d7f272-host-run-k8s-cni-cncf-io\") pod \"multus-krnqh\" (UID: \"9aeff916-a90a-49ae-9ac5-0e7a46d7f272\") " pod="openshift-multus/multus-krnqh" Jul 21 00:40:55.001242 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.943399 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9aeff916-a90a-49ae-9ac5-0e7a46d7f272-host-var-lib-kubelet\") pod \"multus-krnqh\" (UID: \"9aeff916-a90a-49ae-9ac5-0e7a46d7f272\") " pod="openshift-multus/multus-krnqh" Jul 21 00:40:55.001242 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.943420 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62qdg\" (UniqueName: \"kubernetes.io/projected/6fbd5477-ce7d-444d-938b-2c54aa3c040c-kube-api-access-62qdg\") pod \"node-ca-2rxh4\" (UID: \"6fbd5477-ce7d-444d-938b-2c54aa3c040c\") " pod="openshift-image-registry/node-ca-2rxh4" Jul 21 00:40:55.001242 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.943429 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9aeff916-a90a-49ae-9ac5-0e7a46d7f272-host-var-lib-kubelet\") pod \"multus-krnqh\" (UID: \"9aeff916-a90a-49ae-9ac5-0e7a46d7f272\") " pod="openshift-multus/multus-krnqh" Jul 21 00:40:55.001242 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.943448 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f3e26871-c996-46c1-adfb-c96c1869814b-cnibin\") pod \"multus-additional-cni-plugins-6zg78\" (UID: \"f3e26871-c996-46c1-adfb-c96c1869814b\") " pod="openshift-multus/multus-additional-cni-plugins-6zg78" Jul 21 00:40:55.001242 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.943466 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9aeff916-a90a-49ae-9ac5-0e7a46d7f272-host-run-k8s-cni-cncf-io\") pod \"multus-krnqh\" (UID: \"9aeff916-a90a-49ae-9ac5-0e7a46d7f272\") " pod="openshift-multus/multus-krnqh" Jul 21 00:40:55.001242 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.943471 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wxxq\" (UniqueName: \"kubernetes.io/projected/e11587d4-f449-4b99-8355-787a52243ba4-kube-api-access-8wxxq\") pod \"network-check-target-9r2ff\" (UID: \"e11587d4-f449-4b99-8355-787a52243ba4\") " pod="openshift-network-diagnostics/network-check-target-9r2ff" Jul 21 00:40:55.001242 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.943511 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9aeff916-a90a-49ae-9ac5-0e7a46d7f272-multus-conf-dir\") pod \"multus-krnqh\" (UID: \"9aeff916-a90a-49ae-9ac5-0e7a46d7f272\") " pod="openshift-multus/multus-krnqh" Jul 21 00:40:55.001242 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.943536 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cfb338e-19eb-45ee-a3de-ea7311469109-metrics-certs\") pod \"network-metrics-daemon-4nq5b\" (UID: \"6cfb338e-19eb-45ee-a3de-ea7311469109\") " pod="openshift-multus/network-metrics-daemon-4nq5b" Jul 21 00:40:55.001242 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.943563 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/245f1c61-31ff-4104-8fa1-5d970dd40533-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qthms\" (UID: \"245f1c61-31ff-4104-8fa1-5d970dd40533\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qthms" Jul 21 00:40:55.001242 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.943522 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f3e26871-c996-46c1-adfb-c96c1869814b-cnibin\") pod \"multus-additional-cni-plugins-6zg78\" (UID: \"f3e26871-c996-46c1-adfb-c96c1869814b\") " pod="openshift-multus/multus-additional-cni-plugins-6zg78" Jul 21 00:40:55.001242 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.943567 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9aeff916-a90a-49ae-9ac5-0e7a46d7f272-multus-conf-dir\") pod \"multus-krnqh\" (UID: \"9aeff916-a90a-49ae-9ac5-0e7a46d7f272\") " pod="openshift-multus/multus-krnqh" Jul 21 00:40:55.001242 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:40:54.943610 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 00:40:55.001242 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.943617 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f3e26871-c996-46c1-adfb-c96c1869814b-cni-binary-copy\") pod \"multus-additional-cni-plugins-6zg78\" (UID: \"f3e26871-c996-46c1-adfb-c96c1869814b\") " pod="openshift-multus/multus-additional-cni-plugins-6zg78" Jul 21 00:40:55.001242 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.943640 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9aeff916-a90a-49ae-9ac5-0e7a46d7f272-multus-cni-dir\") pod \"multus-krnqh\" (UID: \"9aeff916-a90a-49ae-9ac5-0e7a46d7f272\") " pod="openshift-multus/multus-krnqh" Jul 21 00:40:55.001242 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.943620 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/245f1c61-31ff-4104-8fa1-5d970dd40533-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qthms\" (UID: \"245f1c61-31ff-4104-8fa1-5d970dd40533\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qthms" Jul 21 00:40:55.001720 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.943661 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9aeff916-a90a-49ae-9ac5-0e7a46d7f272-cni-binary-copy\") pod \"multus-krnqh\" (UID: \"9aeff916-a90a-49ae-9ac5-0e7a46d7f272\") " pod="openshift-multus/multus-krnqh" Jul 21 00:40:55.001720 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:40:54.943690 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6cfb338e-19eb-45ee-a3de-ea7311469109-metrics-certs podName:6cfb338e-19eb-45ee-a3de-ea7311469109 nodeName:}" failed. No retries permitted until 2026-07-21 00:40:55.443667433 +0000 UTC m=+3.049776474 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6cfb338e-19eb-45ee-a3de-ea7311469109-metrics-certs") pod "network-metrics-daemon-4nq5b" (UID: "6cfb338e-19eb-45ee-a3de-ea7311469109") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 00:40:55.001720 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.943712 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9aeff916-a90a-49ae-9ac5-0e7a46d7f272-multus-cni-dir\") pod \"multus-krnqh\" (UID: \"9aeff916-a90a-49ae-9ac5-0e7a46d7f272\") " pod="openshift-multus/multus-krnqh" Jul 21 00:40:55.001720 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.943732 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f3e26871-c996-46c1-adfb-c96c1869814b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6zg78\" (UID: \"f3e26871-c996-46c1-adfb-c96c1869814b\") " pod="openshift-multus/multus-additional-cni-plugins-6zg78" Jul 21 00:40:55.001720 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.943762 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9aeff916-a90a-49ae-9ac5-0e7a46d7f272-host-run-netns\") pod \"multus-krnqh\" (UID: \"9aeff916-a90a-49ae-9ac5-0e7a46d7f272\") " pod="openshift-multus/multus-krnqh" Jul 21 00:40:55.001720 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.943791 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/245f1c61-31ff-4104-8fa1-5d970dd40533-etc-selinux\") pod \"aws-ebs-csi-driver-node-qthms\" (UID: \"245f1c61-31ff-4104-8fa1-5d970dd40533\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qthms" Jul 21 00:40:55.001720 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.943828 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f3e26871-c996-46c1-adfb-c96c1869814b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6zg78\" (UID: \"f3e26871-c996-46c1-adfb-c96c1869814b\") " pod="openshift-multus/multus-additional-cni-plugins-6zg78" Jul 21 00:40:55.001720 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.943850 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9aeff916-a90a-49ae-9ac5-0e7a46d7f272-host-run-netns\") pod \"multus-krnqh\" (UID: \"9aeff916-a90a-49ae-9ac5-0e7a46d7f272\") " pod="openshift-multus/multus-krnqh" Jul 21 00:40:55.001720 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.943863 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e7629fc1-b058-4431-b0d3-1b465dee3b48-konnectivity-ca\") pod \"konnectivity-agent-ntk2c\" (UID: \"e7629fc1-b058-4431-b0d3-1b465dee3b48\") " pod="kube-system/konnectivity-agent-ntk2c" Jul 21 00:40:55.001720 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.943892 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/70177b3d-5d28-4be4-a20d-63c49744b652-iptables-alerter-script\") pod \"iptables-alerter-fj98d\" (UID: \"70177b3d-5d28-4be4-a20d-63c49744b652\") " pod="openshift-network-operator/iptables-alerter-fj98d" Jul 21 00:40:55.001720 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.943934 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/245f1c61-31ff-4104-8fa1-5d970dd40533-etc-selinux\") pod \"aws-ebs-csi-driver-node-qthms\" (UID: \"245f1c61-31ff-4104-8fa1-5d970dd40533\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qthms" Jul 21 00:40:55.001720 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.943945 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f3e26871-c996-46c1-adfb-c96c1869814b-os-release\") pod \"multus-additional-cni-plugins-6zg78\" (UID: \"f3e26871-c996-46c1-adfb-c96c1869814b\") " pod="openshift-multus/multus-additional-cni-plugins-6zg78" Jul 21 00:40:55.001720 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.943972 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df3aad65-414d-4dd0-9e23-c0c0e46e4f06-tmp-dir\") pod \"node-resolver-858wv\" (UID: \"df3aad65-414d-4dd0-9e23-c0c0e46e4f06\") " pod="openshift-dns/node-resolver-858wv" Jul 21 00:40:55.002214 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.943998 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9aeff916-a90a-49ae-9ac5-0e7a46d7f272-host-var-lib-cni-multus\") pod \"multus-krnqh\" (UID: \"9aeff916-a90a-49ae-9ac5-0e7a46d7f272\") " pod="openshift-multus/multus-krnqh" Jul 21 00:40:55.002214 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.944025 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/245f1c61-31ff-4104-8fa1-5d970dd40533-registration-dir\") pod \"aws-ebs-csi-driver-node-qthms\" (UID: \"245f1c61-31ff-4104-8fa1-5d970dd40533\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qthms" Jul 21 00:40:55.002214 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.944051 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f3e26871-c996-46c1-adfb-c96c1869814b-system-cni-dir\") pod \"multus-additional-cni-plugins-6zg78\" (UID: \"f3e26871-c996-46c1-adfb-c96c1869814b\") " pod="openshift-multus/multus-additional-cni-plugins-6zg78" Jul 21 00:40:55.002214 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.944080 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtd5t\" (UniqueName: \"kubernetes.io/projected/f3e26871-c996-46c1-adfb-c96c1869814b-kube-api-access-jtd5t\") pod \"multus-additional-cni-plugins-6zg78\" (UID: \"f3e26871-c996-46c1-adfb-c96c1869814b\") " pod="openshift-multus/multus-additional-cni-plugins-6zg78" Jul 21 00:40:55.002214 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.944108 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9aeff916-a90a-49ae-9ac5-0e7a46d7f272-host-var-lib-cni-bin\") pod \"multus-krnqh\" (UID: \"9aeff916-a90a-49ae-9ac5-0e7a46d7f272\") " pod="openshift-multus/multus-krnqh" Jul 21 00:40:55.002214 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.944133 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/245f1c61-31ff-4104-8fa1-5d970dd40533-device-dir\") pod \"aws-ebs-csi-driver-node-qthms\" (UID: \"245f1c61-31ff-4104-8fa1-5d970dd40533\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qthms" Jul 21 00:40:55.002214 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.944164 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9aeff916-a90a-49ae-9ac5-0e7a46d7f272-cnibin\") pod \"multus-krnqh\" (UID: \"9aeff916-a90a-49ae-9ac5-0e7a46d7f272\") " pod="openshift-multus/multus-krnqh" Jul 21 00:40:55.002214 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.944189 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9aeff916-a90a-49ae-9ac5-0e7a46d7f272-os-release\") pod \"multus-krnqh\" (UID: \"9aeff916-a90a-49ae-9ac5-0e7a46d7f272\") " pod="openshift-multus/multus-krnqh" Jul 21 00:40:55.002214 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.944220 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f3e26871-c996-46c1-adfb-c96c1869814b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6zg78\" (UID: \"f3e26871-c996-46c1-adfb-c96c1869814b\") " pod="openshift-multus/multus-additional-cni-plugins-6zg78" Jul 21 00:40:55.002214 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.944244 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6fbd5477-ce7d-444d-938b-2c54aa3c040c-serviceca\") pod \"node-ca-2rxh4\" (UID: \"6fbd5477-ce7d-444d-938b-2c54aa3c040c\") " pod="openshift-image-registry/node-ca-2rxh4" Jul 21 00:40:55.002214 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.944264 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f3e26871-c996-46c1-adfb-c96c1869814b-cni-binary-copy\") pod \"multus-additional-cni-plugins-6zg78\" (UID: \"f3e26871-c996-46c1-adfb-c96c1869814b\") " pod="openshift-multus/multus-additional-cni-plugins-6zg78" Jul 21 00:40:55.002214 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.944296 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fda1cd2e-3d5d-4dc1-94a1-1b8ebbb4a199-original-pull-secret\") pod \"global-pull-secret-syncer-dzcmw\" (UID: \"fda1cd2e-3d5d-4dc1-94a1-1b8ebbb4a199\") " pod="kube-system/global-pull-secret-syncer-dzcmw" Jul 21 00:40:55.002214 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.944324 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/df3aad65-414d-4dd0-9e23-c0c0e46e4f06-hosts-file\") pod \"node-resolver-858wv\" (UID: \"df3aad65-414d-4dd0-9e23-c0c0e46e4f06\") " pod="openshift-dns/node-resolver-858wv" Jul 21 00:40:55.002214 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.944340 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9aeff916-a90a-49ae-9ac5-0e7a46d7f272-cni-binary-copy\") pod \"multus-krnqh\" (UID: \"9aeff916-a90a-49ae-9ac5-0e7a46d7f272\") " pod="openshift-multus/multus-krnqh" Jul 21 00:40:55.002674 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.944349 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9aeff916-a90a-49ae-9ac5-0e7a46d7f272-etc-kubernetes\") pod \"multus-krnqh\" (UID: \"9aeff916-a90a-49ae-9ac5-0e7a46d7f272\") " pod="openshift-multus/multus-krnqh" Jul 21 00:40:55.002674 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.944349 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f3e26871-c996-46c1-adfb-c96c1869814b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6zg78\" (UID: \"f3e26871-c996-46c1-adfb-c96c1869814b\") " pod="openshift-multus/multus-additional-cni-plugins-6zg78" Jul 21 00:40:55.002674 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.944395 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9aeff916-a90a-49ae-9ac5-0e7a46d7f272-host-var-lib-cni-multus\") pod \"multus-krnqh\" (UID: \"9aeff916-a90a-49ae-9ac5-0e7a46d7f272\") " pod="openshift-multus/multus-krnqh" Jul 21 00:40:55.002674 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.944418 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9aeff916-a90a-49ae-9ac5-0e7a46d7f272-host-var-lib-cni-bin\") pod \"multus-krnqh\" (UID: \"9aeff916-a90a-49ae-9ac5-0e7a46d7f272\") " pod="openshift-multus/multus-krnqh" Jul 21 00:40:55.002674 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.944429 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/245f1c61-31ff-4104-8fa1-5d970dd40533-sys-fs\") pod \"aws-ebs-csi-driver-node-qthms\" (UID: \"245f1c61-31ff-4104-8fa1-5d970dd40533\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qthms" Jul 21 00:40:55.002674 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.944435 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f3e26871-c996-46c1-adfb-c96c1869814b-os-release\") pod \"multus-additional-cni-plugins-6zg78\" (UID: \"f3e26871-c996-46c1-adfb-c96c1869814b\") " pod="openshift-multus/multus-additional-cni-plugins-6zg78" Jul 21 00:40:55.002674 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.944460 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9aeff916-a90a-49ae-9ac5-0e7a46d7f272-multus-daemon-config\") pod \"multus-krnqh\" (UID: \"9aeff916-a90a-49ae-9ac5-0e7a46d7f272\") " pod="openshift-multus/multus-krnqh" Jul 21 00:40:55.002674 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.944473 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/245f1c61-31ff-4104-8fa1-5d970dd40533-device-dir\") pod \"aws-ebs-csi-driver-node-qthms\" (UID: \"245f1c61-31ff-4104-8fa1-5d970dd40533\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qthms" Jul 21 00:40:55.002674 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.944469 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f3e26871-c996-46c1-adfb-c96c1869814b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6zg78\" (UID: \"f3e26871-c996-46c1-adfb-c96c1869814b\") " pod="openshift-multus/multus-additional-cni-plugins-6zg78" Jul 21 00:40:55.002674 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.944487 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9aeff916-a90a-49ae-9ac5-0e7a46d7f272-host-run-multus-certs\") pod \"multus-krnqh\" (UID: \"9aeff916-a90a-49ae-9ac5-0e7a46d7f272\") " pod="openshift-multus/multus-krnqh" Jul 21 00:40:55.002674 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.944527 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/245f1c61-31ff-4104-8fa1-5d970dd40533-registration-dir\") pod \"aws-ebs-csi-driver-node-qthms\" (UID: \"245f1c61-31ff-4104-8fa1-5d970dd40533\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qthms" Jul 21 00:40:55.002674 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.944579 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9aeff916-a90a-49ae-9ac5-0e7a46d7f272-etc-kubernetes\") pod \"multus-krnqh\" (UID: \"9aeff916-a90a-49ae-9ac5-0e7a46d7f272\") " pod="openshift-multus/multus-krnqh" Jul 21 00:40:55.002674 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:40:54.944587 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 00:40:55.002674 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.944588 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f3e26871-c996-46c1-adfb-c96c1869814b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6zg78\" (UID: \"f3e26871-c996-46c1-adfb-c96c1869814b\") " pod="openshift-multus/multus-additional-cni-plugins-6zg78" Jul 21 00:40:55.002674 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.944600 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/70177b3d-5d28-4be4-a20d-63c49744b652-iptables-alerter-script\") pod \"iptables-alerter-fj98d\" (UID: \"70177b3d-5d28-4be4-a20d-63c49744b652\") " pod="openshift-network-operator/iptables-alerter-fj98d" Jul 21 00:40:55.003180 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.944608 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e7629fc1-b058-4431-b0d3-1b465dee3b48-konnectivity-ca\") pod \"konnectivity-agent-ntk2c\" (UID: \"e7629fc1-b058-4431-b0d3-1b465dee3b48\") " pod="kube-system/konnectivity-agent-ntk2c" Jul 21 00:40:55.003180 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.944581 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/df3aad65-414d-4dd0-9e23-c0c0e46e4f06-hosts-file\") pod \"node-resolver-858wv\" (UID: \"df3aad65-414d-4dd0-9e23-c0c0e46e4f06\") " pod="openshift-dns/node-resolver-858wv" Jul 21 00:40:55.003180 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.944638 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f3e26871-c996-46c1-adfb-c96c1869814b-system-cni-dir\") pod \"multus-additional-cni-plugins-6zg78\" (UID: \"f3e26871-c996-46c1-adfb-c96c1869814b\") " pod="openshift-multus/multus-additional-cni-plugins-6zg78" Jul 21 00:40:55.003180 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.944643 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9aeff916-a90a-49ae-9ac5-0e7a46d7f272-host-run-multus-certs\") pod \"multus-krnqh\" (UID: \"9aeff916-a90a-49ae-9ac5-0e7a46d7f272\") " pod="openshift-multus/multus-krnqh" Jul 21 00:40:55.003180 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.944645 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9aeff916-a90a-49ae-9ac5-0e7a46d7f272-os-release\") pod \"multus-krnqh\" (UID: \"9aeff916-a90a-49ae-9ac5-0e7a46d7f272\") " pod="openshift-multus/multus-krnqh" Jul 21 00:40:55.003180 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:40:54.944686 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fda1cd2e-3d5d-4dc1-94a1-1b8ebbb4a199-original-pull-secret podName:fda1cd2e-3d5d-4dc1-94a1-1b8ebbb4a199 nodeName:}" failed. No retries permitted until 2026-07-21 00:40:55.444669527 +0000 UTC m=+3.050778561 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fda1cd2e-3d5d-4dc1-94a1-1b8ebbb4a199-original-pull-secret") pod "global-pull-secret-syncer-dzcmw" (UID: "fda1cd2e-3d5d-4dc1-94a1-1b8ebbb4a199") : object "kube-system"/"original-pull-secret" not registered Jul 21 00:40:55.003180 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.944688 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9aeff916-a90a-49ae-9ac5-0e7a46d7f272-cnibin\") pod \"multus-krnqh\" (UID: \"9aeff916-a90a-49ae-9ac5-0e7a46d7f272\") " pod="openshift-multus/multus-krnqh" Jul 21 00:40:55.003180 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.944684 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/245f1c61-31ff-4104-8fa1-5d970dd40533-sys-fs\") pod \"aws-ebs-csi-driver-node-qthms\" (UID: \"245f1c61-31ff-4104-8fa1-5d970dd40533\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qthms" Jul 21 00:40:55.003180 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.944883 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df3aad65-414d-4dd0-9e23-c0c0e46e4f06-tmp-dir\") pod \"node-resolver-858wv\" (UID: \"df3aad65-414d-4dd0-9e23-c0c0e46e4f06\") " pod="openshift-dns/node-resolver-858wv" Jul 21 00:40:55.003180 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.944952 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6fbd5477-ce7d-444d-938b-2c54aa3c040c-serviceca\") pod \"node-ca-2rxh4\" (UID: \"6fbd5477-ce7d-444d-938b-2c54aa3c040c\") " pod="openshift-image-registry/node-ca-2rxh4" Jul 21 00:40:55.003180 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.944978 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9aeff916-a90a-49ae-9ac5-0e7a46d7f272-multus-daemon-config\") pod \"multus-krnqh\" (UID: \"9aeff916-a90a-49ae-9ac5-0e7a46d7f272\") " pod="openshift-multus/multus-krnqh" Jul 21 00:40:55.003180 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.945549 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e7629fc1-b058-4431-b0d3-1b465dee3b48-agent-certs\") pod \"konnectivity-agent-ntk2c\" (UID: \"e7629fc1-b058-4431-b0d3-1b465dee3b48\") " pod="kube-system/konnectivity-agent-ntk2c" Jul 21 00:40:55.003180 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:40:54.951441 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 00:40:55.003180 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:40:54.951481 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 00:40:55.003180 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:40:54.951498 2580 projected.go:196] Error preparing data for projected volume kube-api-access-8wxxq for pod openshift-network-diagnostics/network-check-target-9r2ff: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 00:40:55.003756 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:40:54.951577 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e11587d4-f449-4b99-8355-787a52243ba4-kube-api-access-8wxxq podName:e11587d4-f449-4b99-8355-787a52243ba4 nodeName:}" failed. No retries permitted until 2026-07-21 00:40:55.451561757 +0000 UTC m=+3.057670781 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-8wxxq" (UniqueName: "kubernetes.io/projected/e11587d4-f449-4b99-8355-787a52243ba4-kube-api-access-8wxxq") pod "network-check-target-9r2ff" (UID: "e11587d4-f449-4b99-8355-787a52243ba4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 00:40:55.003756 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.953514 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lq8n8\" (UniqueName: \"kubernetes.io/projected/9aeff916-a90a-49ae-9ac5-0e7a46d7f272-kube-api-access-lq8n8\") pod \"multus-krnqh\" (UID: \"9aeff916-a90a-49ae-9ac5-0e7a46d7f272\") " pod="openshift-multus/multus-krnqh" Jul 21 00:40:55.003756 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.953606 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfsv5\" (UniqueName: \"kubernetes.io/projected/6cfb338e-19eb-45ee-a3de-ea7311469109-kube-api-access-mfsv5\") pod \"network-metrics-daemon-4nq5b\" (UID: \"6cfb338e-19eb-45ee-a3de-ea7311469109\") " pod="openshift-multus/network-metrics-daemon-4nq5b" Jul 21 00:40:55.003756 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.953816 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wb98\" (UniqueName: \"kubernetes.io/projected/df3aad65-414d-4dd0-9e23-c0c0e46e4f06-kube-api-access-6wb98\") pod \"node-resolver-858wv\" (UID: \"df3aad65-414d-4dd0-9e23-c0c0e46e4f06\") " pod="openshift-dns/node-resolver-858wv" Jul 21 00:40:55.003756 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.953842 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtd5t\" (UniqueName: \"kubernetes.io/projected/f3e26871-c996-46c1-adfb-c96c1869814b-kube-api-access-jtd5t\") pod \"multus-additional-cni-plugins-6zg78\" (UID: \"f3e26871-c996-46c1-adfb-c96c1869814b\") " pod="openshift-multus/multus-additional-cni-plugins-6zg78" Jul 21 00:40:55.003756 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.953858 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62qdg\" (UniqueName: \"kubernetes.io/projected/6fbd5477-ce7d-444d-938b-2c54aa3c040c-kube-api-access-62qdg\") pod \"node-ca-2rxh4\" (UID: \"6fbd5477-ce7d-444d-938b-2c54aa3c040c\") " pod="openshift-image-registry/node-ca-2rxh4" Jul 21 00:40:55.003756 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.954408 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zd6rj\" (UniqueName: \"kubernetes.io/projected/245f1c61-31ff-4104-8fa1-5d970dd40533-kube-api-access-zd6rj\") pod \"aws-ebs-csi-driver-node-qthms\" (UID: \"245f1c61-31ff-4104-8fa1-5d970dd40533\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qthms" Jul 21 00:40:55.003756 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:54.954859 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch652\" (UniqueName: \"kubernetes.io/projected/70177b3d-5d28-4be4-a20d-63c49744b652-kube-api-access-ch652\") pod \"iptables-alerter-fj98d\" (UID: \"70177b3d-5d28-4be4-a20d-63c49744b652\") " pod="openshift-network-operator/iptables-alerter-fj98d" Jul 21 00:40:55.242165 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:55.053054 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jwqsv" Jul 21 00:40:55.242165 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:55.063718 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b7ed92b_7e21_40b9_9ac6_f233a11168d0.slice/crio-92e982a3086dac97d8b7a647e13507321bb101d149c5f0e3505869d773c40e71 WatchSource:0}: Error finding container 92e982a3086dac97d8b7a647e13507321bb101d149c5f0e3505869d773c40e71: Status 404 returned error can't find the container with id 92e982a3086dac97d8b7a647e13507321bb101d149c5f0e3505869d773c40e71 Jul 21 00:40:55.242165 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:55.070635 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bctst" Jul 21 00:40:55.242165 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:55.079366 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71312c5c_a477_4500_996f_4c7cffd7fdc6.slice/crio-a1b4d3642b9c9526e9da7bef79019206a9e85f84000c57470d60be7568e4d515 WatchSource:0}: Error finding container a1b4d3642b9c9526e9da7bef79019206a9e85f84000c57470d60be7568e4d515: Status 404 returned error can't find the container with id a1b4d3642b9c9526e9da7bef79019206a9e85f84000c57470d60be7568e4d515 Jul 21 00:40:55.242165 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:55.090931 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qthms" Jul 21 00:40:55.242165 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:55.102915 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod245f1c61_31ff_4104_8fa1_5d970dd40533.slice/crio-5a514727cc61da6f003ca7411a880ab23de61806fbc06020510be6502d9bcda7 WatchSource:0}: Error finding container 5a514727cc61da6f003ca7411a880ab23de61806fbc06020510be6502d9bcda7: Status 404 returned error can't find the container with id 5a514727cc61da6f003ca7411a880ab23de61806fbc06020510be6502d9bcda7 Jul 21 00:40:55.242165 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:55.108803 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fj98d" Jul 21 00:40:55.242165 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:55.119498 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70177b3d_5d28_4be4_a20d_63c49744b652.slice/crio-5413f17fed7bb45f6fafb1bcf76305841418443470fd52df7461dd309b359d07 WatchSource:0}: Error finding container 5413f17fed7bb45f6fafb1bcf76305841418443470fd52df7461dd309b359d07: Status 404 returned error can't find the container with id 5413f17fed7bb45f6fafb1bcf76305841418443470fd52df7461dd309b359d07 Jul 21 00:40:55.242165 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:55.127669 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-krnqh" Jul 21 00:40:55.242165 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:55.134929 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6zg78" Jul 21 00:40:55.242165 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:55.138972 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9aeff916_a90a_49ae_9ac5_0e7a46d7f272.slice/crio-3a402723bde1712657bb1b3b721dfedd080a428f23d21ac282d8611dd75b71c8 WatchSource:0}: Error finding container 3a402723bde1712657bb1b3b721dfedd080a428f23d21ac282d8611dd75b71c8: Status 404 returned error can't find the container with id 3a402723bde1712657bb1b3b721dfedd080a428f23d21ac282d8611dd75b71c8 Jul 21 00:40:55.242165 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:55.144462 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3e26871_c996_46c1_adfb_c96c1869814b.slice/crio-b7649acfa4490f78d8bc02f9f3f56a58b60f57e9a93bd8d85fc20df67e649007 WatchSource:0}: Error finding container b7649acfa4490f78d8bc02f9f3f56a58b60f57e9a93bd8d85fc20df67e649007: Status 404 returned error can't find the container with id b7649acfa4490f78d8bc02f9f3f56a58b60f57e9a93bd8d85fc20df67e649007 Jul 21 00:40:55.242165 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:55.162053 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2rxh4" Jul 21 00:40:55.242165 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:55.171077 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fbd5477_ce7d_444d_938b_2c54aa3c040c.slice/crio-7f354b54986963ec41f2a270abbf0a275e40da4246f180503477a098af43c8cc WatchSource:0}: Error finding container 7f354b54986963ec41f2a270abbf0a275e40da4246f180503477a098af43c8cc: Status 404 returned error can't find the container with id 7f354b54986963ec41f2a270abbf0a275e40da4246f180503477a098af43c8cc Jul 21 00:40:55.242165 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:55.171405 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-858wv" Jul 21 00:40:55.242165 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:55.172955 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ntk2c" Jul 21 00:40:55.243494 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:55.180605 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf3aad65_414d_4dd0_9e23_c0c0e46e4f06.slice/crio-be5f7527b066a97b0a88dd0ebb6f8d599b213728b477a0fc1dfd544ed8269b4e WatchSource:0}: Error finding container be5f7527b066a97b0a88dd0ebb6f8d599b213728b477a0fc1dfd544ed8269b4e: Status 404 returned error can't find the container with id be5f7527b066a97b0a88dd0ebb6f8d599b213728b477a0fc1dfd544ed8269b4e Jul 21 00:40:55.243494 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:55.183468 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7629fc1_b058_4431_b0d3_1b465dee3b48.slice/crio-dfa209e9b532d17276fdbc3d95293fbf2b8d11c3da7954539b5017658890bf2b WatchSource:0}: Error finding container dfa209e9b532d17276fdbc3d95293fbf2b8d11c3da7954539b5017658890bf2b: Status 404 returned error can't find the container with id dfa209e9b532d17276fdbc3d95293fbf2b8d11c3da7954539b5017658890bf2b Jul 21 00:40:55.486173 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:55.261462 2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 00:40:55.486173 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:40:55.368507 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf449d6f92faa6a51a14c54944141985e.slice/crio-6fccfdc040e60c9c26168e5a7ec06636eb3b5449c1c08ca03b84e68bf1dc39d5 WatchSource:0}: Error finding container 6fccfdc040e60c9c26168e5a7ec06636eb3b5449c1c08ca03b84e68bf1dc39d5: Status 404 returned error can't find the container with id 6fccfdc040e60c9c26168e5a7ec06636eb3b5449c1c08ca03b84e68bf1dc39d5 Jul 21 00:40:55.486173 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:55.446662 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cfb338e-19eb-45ee-a3de-ea7311469109-metrics-certs\") pod \"network-metrics-daemon-4nq5b\" (UID: \"6cfb338e-19eb-45ee-a3de-ea7311469109\") " pod="openshift-multus/network-metrics-daemon-4nq5b" Jul 21 00:40:55.486173 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:55.446714 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fda1cd2e-3d5d-4dc1-94a1-1b8ebbb4a199-original-pull-secret\") pod \"global-pull-secret-syncer-dzcmw\" (UID: \"fda1cd2e-3d5d-4dc1-94a1-1b8ebbb4a199\") " pod="kube-system/global-pull-secret-syncer-dzcmw" Jul 21 00:40:55.486173 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:40:55.446815 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 00:40:55.486173 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:40:55.446817 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 00:40:55.486173 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:40:55.446868 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fda1cd2e-3d5d-4dc1-94a1-1b8ebbb4a199-original-pull-secret podName:fda1cd2e-3d5d-4dc1-94a1-1b8ebbb4a199 nodeName:}" failed. No retries permitted until 2026-07-21 00:40:56.446853404 +0000 UTC m=+4.052962424 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fda1cd2e-3d5d-4dc1-94a1-1b8ebbb4a199-original-pull-secret") pod "global-pull-secret-syncer-dzcmw" (UID: "fda1cd2e-3d5d-4dc1-94a1-1b8ebbb4a199") : object "kube-system"/"original-pull-secret" not registered Jul 21 00:40:55.486173 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:40:55.446884 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6cfb338e-19eb-45ee-a3de-ea7311469109-metrics-certs podName:6cfb338e-19eb-45ee-a3de-ea7311469109 nodeName:}" failed. No retries permitted until 2026-07-21 00:40:56.446876425 +0000 UTC m=+4.052985445 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6cfb338e-19eb-45ee-a3de-ea7311469109-metrics-certs") pod "network-metrics-daemon-4nq5b" (UID: "6cfb338e-19eb-45ee-a3de-ea7311469109") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 00:40:55.622737 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:55.547779 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wxxq\" (UniqueName: \"kubernetes.io/projected/e11587d4-f449-4b99-8355-787a52243ba4-kube-api-access-8wxxq\") pod \"network-check-target-9r2ff\" (UID: \"e11587d4-f449-4b99-8355-787a52243ba4\") " pod="openshift-network-diagnostics/network-check-target-9r2ff" Jul 21 00:40:55.622737 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:40:55.547922 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 00:40:55.622737 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:40:55.547944 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 00:40:55.622737 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:40:55.547957 2580 projected.go:196] Error preparing data for projected volume kube-api-access-8wxxq for pod openshift-network-diagnostics/network-check-target-9r2ff: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 00:40:55.622737 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:40:55.548019 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e11587d4-f449-4b99-8355-787a52243ba4-kube-api-access-8wxxq podName:e11587d4-f449-4b99-8355-787a52243ba4 nodeName:}" failed. No retries permitted until 2026-07-21 00:40:56.54799978 +0000 UTC m=+4.154108819 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-8wxxq" (UniqueName: "kubernetes.io/projected/e11587d4-f449-4b99-8355-787a52243ba4-kube-api-access-8wxxq") pod "network-check-target-9r2ff" (UID: "e11587d4-f449-4b99-8355-787a52243ba4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 00:40:55.881837 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:55.796477 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-20 00:35:55 +0000 UTC" deadline="2028-04-04 13:29:42.882645012 +0000 UTC" Jul 21 00:40:55.881837 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:55.796539 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14964h48m47.086110617s" Jul 21 00:40:56.003206 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:55.917411 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2rxh4" event={"ID":"6fbd5477-ce7d-444d-938b-2c54aa3c040c","Type":"ContainerStarted","Data":"7f354b54986963ec41f2a270abbf0a275e40da4246f180503477a098af43c8cc"} Jul 21 00:40:56.003206 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:55.919165 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fj98d" event={"ID":"70177b3d-5d28-4be4-a20d-63c49744b652","Type":"ContainerStarted","Data":"5413f17fed7bb45f6fafb1bcf76305841418443470fd52df7461dd309b359d07"} Jul 21 00:40:56.003206 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:55.922199 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ntk2c" event={"ID":"e7629fc1-b058-4431-b0d3-1b465dee3b48","Type":"ContainerStarted","Data":"dfa209e9b532d17276fdbc3d95293fbf2b8d11c3da7954539b5017658890bf2b"} Jul 21 00:40:56.003206 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:55.926935 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-858wv" event={"ID":"df3aad65-414d-4dd0-9e23-c0c0e46e4f06","Type":"ContainerStarted","Data":"be5f7527b066a97b0a88dd0ebb6f8d599b213728b477a0fc1dfd544ed8269b4e"} Jul 21 00:40:56.003206 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:55.929543 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6zg78" event={"ID":"f3e26871-c996-46c1-adfb-c96c1869814b","Type":"ContainerStarted","Data":"b7649acfa4490f78d8bc02f9f3f56a58b60f57e9a93bd8d85fc20df67e649007"} Jul 21 00:40:56.003206 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:55.931321 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-krnqh" event={"ID":"9aeff916-a90a-49ae-9ac5-0e7a46d7f272","Type":"ContainerStarted","Data":"3a402723bde1712657bb1b3b721dfedd080a428f23d21ac282d8611dd75b71c8"} Jul 21 00:40:56.003206 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:55.934585 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qthms" event={"ID":"245f1c61-31ff-4104-8fa1-5d970dd40533","Type":"ContainerStarted","Data":"5a514727cc61da6f003ca7411a880ab23de61806fbc06020510be6502d9bcda7"} Jul 21 00:40:56.003206 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:55.936504 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bctst" event={"ID":"71312c5c-a477-4500-996f-4c7cffd7fdc6","Type":"ContainerStarted","Data":"a1b4d3642b9c9526e9da7bef79019206a9e85f84000c57470d60be7568e4d515"} Jul 21 00:40:56.003206 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:55.938197 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jwqsv" event={"ID":"0b7ed92b-7e21-40b9-9ac6-f233a11168d0","Type":"ContainerStarted","Data":"92e982a3086dac97d8b7a647e13507321bb101d149c5f0e3505869d773c40e71"} Jul 21 00:40:56.003206 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:55.942226 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-137.ec2.internal" event={"ID":"f449d6f92faa6a51a14c54944141985e","Type":"ContainerStarted","Data":"6fccfdc040e60c9c26168e5a7ec06636eb3b5449c1c08ca03b84e68bf1dc39d5"} Jul 21 00:40:56.494813 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:56.457867 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cfb338e-19eb-45ee-a3de-ea7311469109-metrics-certs\") pod \"network-metrics-daemon-4nq5b\" (UID: \"6cfb338e-19eb-45ee-a3de-ea7311469109\") " pod="openshift-multus/network-metrics-daemon-4nq5b" Jul 21 00:40:56.494813 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:56.457935 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fda1cd2e-3d5d-4dc1-94a1-1b8ebbb4a199-original-pull-secret\") pod \"global-pull-secret-syncer-dzcmw\" (UID: \"fda1cd2e-3d5d-4dc1-94a1-1b8ebbb4a199\") " pod="kube-system/global-pull-secret-syncer-dzcmw" Jul 21 00:40:56.494813 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:40:56.458056 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 00:40:56.494813 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:40:56.458112 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fda1cd2e-3d5d-4dc1-94a1-1b8ebbb4a199-original-pull-secret podName:fda1cd2e-3d5d-4dc1-94a1-1b8ebbb4a199 nodeName:}" failed. No retries permitted until 2026-07-21 00:40:58.458093836 +0000 UTC m=+6.064202859 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fda1cd2e-3d5d-4dc1-94a1-1b8ebbb4a199-original-pull-secret") pod "global-pull-secret-syncer-dzcmw" (UID: "fda1cd2e-3d5d-4dc1-94a1-1b8ebbb4a199") : object "kube-system"/"original-pull-secret" not registered Jul 21 00:40:56.494813 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:40:56.458552 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 00:40:56.494813 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:40:56.458605 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6cfb338e-19eb-45ee-a3de-ea7311469109-metrics-certs podName:6cfb338e-19eb-45ee-a3de-ea7311469109 nodeName:}" failed. No retries permitted until 2026-07-21 00:40:58.458588377 +0000 UTC m=+6.064697400 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6cfb338e-19eb-45ee-a3de-ea7311469109-metrics-certs") pod "network-metrics-daemon-4nq5b" (UID: "6cfb338e-19eb-45ee-a3de-ea7311469109") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 00:40:56.558883 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:56.558292 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wxxq\" (UniqueName: \"kubernetes.io/projected/e11587d4-f449-4b99-8355-787a52243ba4-kube-api-access-8wxxq\") pod \"network-check-target-9r2ff\" (UID: \"e11587d4-f449-4b99-8355-787a52243ba4\") " pod="openshift-network-diagnostics/network-check-target-9r2ff" Jul 21 00:40:56.558883 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:40:56.558469 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 00:40:56.558883 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:40:56.558488 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 00:40:56.558883 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:40:56.558501 2580 projected.go:196] Error preparing data for projected volume kube-api-access-8wxxq for pod openshift-network-diagnostics/network-check-target-9r2ff: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 00:40:56.558883 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:40:56.558557 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e11587d4-f449-4b99-8355-787a52243ba4-kube-api-access-8wxxq podName:e11587d4-f449-4b99-8355-787a52243ba4 nodeName:}" failed. No retries permitted until 2026-07-21 00:40:58.55853917 +0000 UTC m=+6.164648195 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-8wxxq" (UniqueName: "kubernetes.io/projected/e11587d4-f449-4b99-8355-787a52243ba4-kube-api-access-8wxxq") pod "network-check-target-9r2ff" (UID: "e11587d4-f449-4b99-8355-787a52243ba4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 00:40:56.799825 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:56.799708 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-20 00:35:55 +0000 UTC" deadline="2028-05-05 15:29:46.773441646 +0000 UTC" Jul 21 00:40:56.799825 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:56.799746 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15710h48m49.973701044s" Jul 21 00:40:56.919032 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:56.919000 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4nq5b" Jul 21 00:40:56.919204 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:40:56.919156 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4nq5b" podUID="6cfb338e-19eb-45ee-a3de-ea7311469109" Jul 21 00:40:56.919649 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:56.919626 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzcmw" Jul 21 00:40:56.919762 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:40:56.919727 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dzcmw" podUID="fda1cd2e-3d5d-4dc1-94a1-1b8ebbb4a199" Jul 21 00:40:56.919820 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:56.919793 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9r2ff" Jul 21 00:40:56.919874 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:40:56.919863 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9r2ff" podUID="e11587d4-f449-4b99-8355-787a52243ba4" Jul 21 00:40:58.477133 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:58.476785 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cfb338e-19eb-45ee-a3de-ea7311469109-metrics-certs\") pod \"network-metrics-daemon-4nq5b\" (UID: \"6cfb338e-19eb-45ee-a3de-ea7311469109\") " pod="openshift-multus/network-metrics-daemon-4nq5b" Jul 21 00:40:58.477133 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:58.476857 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fda1cd2e-3d5d-4dc1-94a1-1b8ebbb4a199-original-pull-secret\") pod \"global-pull-secret-syncer-dzcmw\" (UID: \"fda1cd2e-3d5d-4dc1-94a1-1b8ebbb4a199\") " pod="kube-system/global-pull-secret-syncer-dzcmw" Jul 21 00:40:58.477133 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:40:58.476985 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 00:40:58.477133 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:40:58.477022 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 00:40:58.477133 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:40:58.477062 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6cfb338e-19eb-45ee-a3de-ea7311469109-metrics-certs podName:6cfb338e-19eb-45ee-a3de-ea7311469109 nodeName:}" failed. No retries permitted until 2026-07-21 00:41:02.477039945 +0000 UTC m=+10.083148983 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6cfb338e-19eb-45ee-a3de-ea7311469109-metrics-certs") pod "network-metrics-daemon-4nq5b" (UID: "6cfb338e-19eb-45ee-a3de-ea7311469109") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 00:40:58.477133 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:40:58.477112 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fda1cd2e-3d5d-4dc1-94a1-1b8ebbb4a199-original-pull-secret podName:fda1cd2e-3d5d-4dc1-94a1-1b8ebbb4a199 nodeName:}" failed. No retries permitted until 2026-07-21 00:41:02.477095423 +0000 UTC m=+10.083204451 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fda1cd2e-3d5d-4dc1-94a1-1b8ebbb4a199-original-pull-secret") pod "global-pull-secret-syncer-dzcmw" (UID: "fda1cd2e-3d5d-4dc1-94a1-1b8ebbb4a199") : object "kube-system"/"original-pull-secret" not registered Jul 21 00:40:58.579068 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:58.578307 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wxxq\" (UniqueName: \"kubernetes.io/projected/e11587d4-f449-4b99-8355-787a52243ba4-kube-api-access-8wxxq\") pod \"network-check-target-9r2ff\" (UID: \"e11587d4-f449-4b99-8355-787a52243ba4\") " pod="openshift-network-diagnostics/network-check-target-9r2ff" Jul 21 00:40:58.579068 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:40:58.578542 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 00:40:58.579068 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:40:58.578566 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 00:40:58.579068 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:40:58.578581 2580 projected.go:196] Error preparing data for projected volume kube-api-access-8wxxq for pod openshift-network-diagnostics/network-check-target-9r2ff: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 00:40:58.579068 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:40:58.578647 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e11587d4-f449-4b99-8355-787a52243ba4-kube-api-access-8wxxq podName:e11587d4-f449-4b99-8355-787a52243ba4 nodeName:}" failed. No retries permitted until 2026-07-21 00:41:02.578627434 +0000 UTC m=+10.184736466 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-8wxxq" (UniqueName: "kubernetes.io/projected/e11587d4-f449-4b99-8355-787a52243ba4-kube-api-access-8wxxq") pod "network-check-target-9r2ff" (UID: "e11587d4-f449-4b99-8355-787a52243ba4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 00:40:58.906959 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:58.906165 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzcmw" Jul 21 00:40:58.906959 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:40:58.906331 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dzcmw" podUID="fda1cd2e-3d5d-4dc1-94a1-1b8ebbb4a199" Jul 21 00:40:58.906959 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:58.906749 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9r2ff" Jul 21 00:40:58.906959 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:40:58.906853 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9r2ff" podUID="e11587d4-f449-4b99-8355-787a52243ba4" Jul 21 00:40:58.907291 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:40:58.906984 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4nq5b" Jul 21 00:40:58.907291 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:40:58.907076 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4nq5b" podUID="6cfb338e-19eb-45ee-a3de-ea7311469109" Jul 21 00:41:00.907811 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:00.907774 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzcmw" Jul 21 00:41:00.908323 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:00.907905 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dzcmw" podUID="fda1cd2e-3d5d-4dc1-94a1-1b8ebbb4a199" Jul 21 00:41:00.908423 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:00.908342 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4nq5b" Jul 21 00:41:00.908488 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:00.908342 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9r2ff" Jul 21 00:41:00.908545 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:00.908486 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4nq5b" podUID="6cfb338e-19eb-45ee-a3de-ea7311469109" Jul 21 00:41:00.908545 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:00.908513 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9r2ff" podUID="e11587d4-f449-4b99-8355-787a52243ba4" Jul 21 00:41:02.515260 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:02.515209 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cfb338e-19eb-45ee-a3de-ea7311469109-metrics-certs\") pod \"network-metrics-daemon-4nq5b\" (UID: \"6cfb338e-19eb-45ee-a3de-ea7311469109\") " pod="openshift-multus/network-metrics-daemon-4nq5b" Jul 21 00:41:02.515705 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:02.515308 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fda1cd2e-3d5d-4dc1-94a1-1b8ebbb4a199-original-pull-secret\") pod \"global-pull-secret-syncer-dzcmw\" (UID: \"fda1cd2e-3d5d-4dc1-94a1-1b8ebbb4a199\") " pod="kube-system/global-pull-secret-syncer-dzcmw" Jul 21 00:41:02.515705 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:02.515434 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 00:41:02.515705 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:02.515492 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fda1cd2e-3d5d-4dc1-94a1-1b8ebbb4a199-original-pull-secret podName:fda1cd2e-3d5d-4dc1-94a1-1b8ebbb4a199 nodeName:}" failed. No retries permitted until 2026-07-21 00:41:10.515474147 +0000 UTC m=+18.121583172 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fda1cd2e-3d5d-4dc1-94a1-1b8ebbb4a199-original-pull-secret") pod "global-pull-secret-syncer-dzcmw" (UID: "fda1cd2e-3d5d-4dc1-94a1-1b8ebbb4a199") : object "kube-system"/"original-pull-secret" not registered Jul 21 00:41:02.515705 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:02.515525 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 00:41:02.515705 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:02.515593 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6cfb338e-19eb-45ee-a3de-ea7311469109-metrics-certs podName:6cfb338e-19eb-45ee-a3de-ea7311469109 nodeName:}" failed. No retries permitted until 2026-07-21 00:41:10.515575101 +0000 UTC m=+18.121684128 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6cfb338e-19eb-45ee-a3de-ea7311469109-metrics-certs") pod "network-metrics-daemon-4nq5b" (UID: "6cfb338e-19eb-45ee-a3de-ea7311469109") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 00:41:02.616079 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:02.616048 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wxxq\" (UniqueName: \"kubernetes.io/projected/e11587d4-f449-4b99-8355-787a52243ba4-kube-api-access-8wxxq\") pod \"network-check-target-9r2ff\" (UID: \"e11587d4-f449-4b99-8355-787a52243ba4\") " pod="openshift-network-diagnostics/network-check-target-9r2ff" Jul 21 00:41:02.616268 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:02.616249 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 00:41:02.616326 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:02.616275 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 00:41:02.616326 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:02.616290 2580 projected.go:196] Error preparing data for projected volume kube-api-access-8wxxq for pod openshift-network-diagnostics/network-check-target-9r2ff: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 00:41:02.616413 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:02.616354 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e11587d4-f449-4b99-8355-787a52243ba4-kube-api-access-8wxxq podName:e11587d4-f449-4b99-8355-787a52243ba4 nodeName:}" failed. No retries permitted until 2026-07-21 00:41:10.61633522 +0000 UTC m=+18.222444243 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-8wxxq" (UniqueName: "kubernetes.io/projected/e11587d4-f449-4b99-8355-787a52243ba4-kube-api-access-8wxxq") pod "network-check-target-9r2ff" (UID: "e11587d4-f449-4b99-8355-787a52243ba4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 00:41:02.908470 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:02.908396 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4nq5b" Jul 21 00:41:02.908613 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:02.908494 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzcmw" Jul 21 00:41:02.908613 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:02.908502 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4nq5b" podUID="6cfb338e-19eb-45ee-a3de-ea7311469109" Jul 21 00:41:02.908613 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:02.908596 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dzcmw" podUID="fda1cd2e-3d5d-4dc1-94a1-1b8ebbb4a199" Jul 21 00:41:02.908769 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:02.908627 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9r2ff" Jul 21 00:41:02.908769 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:02.908697 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9r2ff" podUID="e11587d4-f449-4b99-8355-787a52243ba4" Jul 21 00:41:04.910265 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:04.910165 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4nq5b" Jul 21 00:41:04.910696 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:04.910316 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4nq5b" podUID="6cfb338e-19eb-45ee-a3de-ea7311469109" Jul 21 00:41:04.910696 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:04.910458 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzcmw" Jul 21 00:41:04.910696 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:04.910568 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dzcmw" podUID="fda1cd2e-3d5d-4dc1-94a1-1b8ebbb4a199" Jul 21 00:41:04.910696 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:04.910603 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9r2ff" Jul 21 00:41:04.910696 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:04.910672 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9r2ff" podUID="e11587d4-f449-4b99-8355-787a52243ba4" Jul 21 00:41:06.906281 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:06.906059 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzcmw" Jul 21 00:41:06.906281 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:06.906059 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4nq5b" Jul 21 00:41:06.906281 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:06.906059 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9r2ff" Jul 21 00:41:06.906281 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:06.906212 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dzcmw" podUID="fda1cd2e-3d5d-4dc1-94a1-1b8ebbb4a199" Jul 21 00:41:06.906699 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:06.906368 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4nq5b" podUID="6cfb338e-19eb-45ee-a3de-ea7311469109" Jul 21 00:41:06.906699 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:06.906464 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9r2ff" podUID="e11587d4-f449-4b99-8355-787a52243ba4" Jul 21 00:41:08.906272 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:08.906242 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4nq5b" Jul 21 00:41:08.906272 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:08.906241 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9r2ff" Jul 21 00:41:08.906762 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:08.906406 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzcmw" Jul 21 00:41:08.906762 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:08.906405 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4nq5b" podUID="6cfb338e-19eb-45ee-a3de-ea7311469109" Jul 21 00:41:08.906762 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:08.906509 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9r2ff" podUID="e11587d4-f449-4b99-8355-787a52243ba4" Jul 21 00:41:08.906762 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:08.906598 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dzcmw" podUID="fda1cd2e-3d5d-4dc1-94a1-1b8ebbb4a199" Jul 21 00:41:10.578270 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:10.578223 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cfb338e-19eb-45ee-a3de-ea7311469109-metrics-certs\") pod \"network-metrics-daemon-4nq5b\" (UID: \"6cfb338e-19eb-45ee-a3de-ea7311469109\") " pod="openshift-multus/network-metrics-daemon-4nq5b" Jul 21 00:41:10.578622 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:10.578296 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fda1cd2e-3d5d-4dc1-94a1-1b8ebbb4a199-original-pull-secret\") pod \"global-pull-secret-syncer-dzcmw\" (UID: \"fda1cd2e-3d5d-4dc1-94a1-1b8ebbb4a199\") " pod="kube-system/global-pull-secret-syncer-dzcmw" Jul 21 00:41:10.578622 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:10.578369 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 00:41:10.578622 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:10.578429 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6cfb338e-19eb-45ee-a3de-ea7311469109-metrics-certs podName:6cfb338e-19eb-45ee-a3de-ea7311469109 nodeName:}" failed. No retries permitted until 2026-07-21 00:41:26.578414499 +0000 UTC m=+34.184523518 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6cfb338e-19eb-45ee-a3de-ea7311469109-metrics-certs") pod "network-metrics-daemon-4nq5b" (UID: "6cfb338e-19eb-45ee-a3de-ea7311469109") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 00:41:10.578622 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:10.578376 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 00:41:10.578622 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:10.578465 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fda1cd2e-3d5d-4dc1-94a1-1b8ebbb4a199-original-pull-secret podName:fda1cd2e-3d5d-4dc1-94a1-1b8ebbb4a199 nodeName:}" failed. No retries permitted until 2026-07-21 00:41:26.578458665 +0000 UTC m=+34.184567687 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fda1cd2e-3d5d-4dc1-94a1-1b8ebbb4a199-original-pull-secret") pod "global-pull-secret-syncer-dzcmw" (UID: "fda1cd2e-3d5d-4dc1-94a1-1b8ebbb4a199") : object "kube-system"/"original-pull-secret" not registered Jul 21 00:41:10.678602 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:10.678574 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wxxq\" (UniqueName: \"kubernetes.io/projected/e11587d4-f449-4b99-8355-787a52243ba4-kube-api-access-8wxxq\") pod \"network-check-target-9r2ff\" (UID: \"e11587d4-f449-4b99-8355-787a52243ba4\") " pod="openshift-network-diagnostics/network-check-target-9r2ff" Jul 21 00:41:10.678736 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:10.678697 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 00:41:10.678736 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:10.678712 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 00:41:10.678736 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:10.678720 2580 projected.go:196] Error preparing data for projected volume kube-api-access-8wxxq for pod openshift-network-diagnostics/network-check-target-9r2ff: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 00:41:10.678825 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:10.678763 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e11587d4-f449-4b99-8355-787a52243ba4-kube-api-access-8wxxq podName:e11587d4-f449-4b99-8355-787a52243ba4 nodeName:}" failed. No retries permitted until 2026-07-21 00:41:26.678750806 +0000 UTC m=+34.284859825 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-8wxxq" (UniqueName: "kubernetes.io/projected/e11587d4-f449-4b99-8355-787a52243ba4-kube-api-access-8wxxq") pod "network-check-target-9r2ff" (UID: "e11587d4-f449-4b99-8355-787a52243ba4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 00:41:10.906259 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:10.906155 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4nq5b" Jul 21 00:41:10.906438 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:10.906155 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzcmw" Jul 21 00:41:10.906438 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:10.906330 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4nq5b" podUID="6cfb338e-19eb-45ee-a3de-ea7311469109" Jul 21 00:41:10.906438 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:10.906350 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9r2ff" Jul 21 00:41:10.906588 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:10.906458 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dzcmw" podUID="fda1cd2e-3d5d-4dc1-94a1-1b8ebbb4a199" Jul 21 00:41:10.906588 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:10.906534 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9r2ff" podUID="e11587d4-f449-4b99-8355-787a52243ba4" Jul 21 00:41:12.906800 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:12.906759 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4nq5b" Jul 21 00:41:12.907251 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:12.906868 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4nq5b" podUID="6cfb338e-19eb-45ee-a3de-ea7311469109" Jul 21 00:41:12.907251 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:12.906913 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzcmw" Jul 21 00:41:12.907251 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:12.907040 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dzcmw" podUID="fda1cd2e-3d5d-4dc1-94a1-1b8ebbb4a199" Jul 21 00:41:12.907251 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:12.907055 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9r2ff" Jul 21 00:41:12.907251 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:12.907141 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9r2ff" podUID="e11587d4-f449-4b99-8355-787a52243ba4" Jul 21 00:41:14.906128 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:14.906091 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9r2ff" Jul 21 00:41:14.906128 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:14.906114 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzcmw" Jul 21 00:41:14.906635 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:14.906091 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4nq5b" Jul 21 00:41:14.906635 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:14.906213 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9r2ff" podUID="e11587d4-f449-4b99-8355-787a52243ba4" Jul 21 00:41:14.906635 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:14.906287 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4nq5b" podUID="6cfb338e-19eb-45ee-a3de-ea7311469109" Jul 21 00:41:14.906635 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:14.906352 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dzcmw" podUID="fda1cd2e-3d5d-4dc1-94a1-1b8ebbb4a199" Jul 21 00:41:16.005372 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:16.004994 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-krnqh" event={"ID":"9aeff916-a90a-49ae-9ac5-0e7a46d7f272","Type":"ContainerStarted","Data":"b19b72ba153e9f0e6bea41a70b552988a1cd246178c140cedfaa83f75c30090b"} Jul 21 00:41:16.007800 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:16.007779 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bctst" event={"ID":"71312c5c-a477-4500-996f-4c7cffd7fdc6","Type":"ContainerStarted","Data":"73952197ea2b6b93476568eabd8b56c74a7d090bffcc9d4f867555a143da2609"} Jul 21 00:41:16.007896 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:16.007835 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bctst" event={"ID":"71312c5c-a477-4500-996f-4c7cffd7fdc6","Type":"ContainerStarted","Data":"a658c31712f699c9e090a56e3ce2c0921dca04191c74be481d225fcf3b1b2adf"} Jul 21 00:41:16.007896 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:16.007863 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bctst" event={"ID":"71312c5c-a477-4500-996f-4c7cffd7fdc6","Type":"ContainerStarted","Data":"2b67f104eb317980565fc2729419c7d0961121e7eb3e98916132d3e6e87fc59a"} Jul 21 00:41:16.007896 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:16.007889 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bctst" event={"ID":"71312c5c-a477-4500-996f-4c7cffd7fdc6","Type":"ContainerStarted","Data":"65e356b3b9e4d11d7b0cbf2f63da6eacafc8691841fc9c8b619daf0b6bdd6ad2"} Jul 21 00:41:16.008045 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:16.007916 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bctst" event={"ID":"71312c5c-a477-4500-996f-4c7cffd7fdc6","Type":"ContainerStarted","Data":"aaac84b8dab3969cf426aa204f60d77574d0b5c597a6cb39fe2eb6b18fd1539a"} Jul 21 00:41:16.009198 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:16.009170 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jwqsv" event={"ID":"0b7ed92b-7e21-40b9-9ac6-f233a11168d0","Type":"ContainerStarted","Data":"23cf179e7bc30922840d610ba09076c51adc0725da6ecfc4f6a3d42d3a4dcd0d"} Jul 21 00:41:16.010659 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:16.010637 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-137.ec2.internal" event={"ID":"f449d6f92faa6a51a14c54944141985e","Type":"ContainerStarted","Data":"50ef98826926387ddea259758af427d21202810eb9e269c2ed9276272da5ff3b"} Jul 21 00:41:16.021520 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:16.021474 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-krnqh" podStartSLOduration=2.961572176 podStartE2EDuration="23.0214594s" totalImagesPullingTime="20.059887224s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 00:40:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:41:16.021060325 +0000 UTC m=+23.627169367" watchObservedRunningTime="2026-07-21 00:41:16.0214594 +0000 UTC m=+23.627568440" Jul 21 00:41:16.034689 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:16.034658 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-137.ec2.internal" podStartSLOduration=22.034646311 podStartE2EDuration="22.034646311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 00:40:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:41:16.034640136 +0000 UTC m=+23.640749176" watchObservedRunningTime="2026-07-21 00:41:16.034646311 +0000 UTC m=+23.640755351" Jul 21 00:41:16.049890 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:16.049856 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-jwqsv" podStartSLOduration=2.91580082 podStartE2EDuration="23.049845298s" totalImagesPullingTime="20.134044478s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 00:40:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:41:16.049633443 +0000 UTC m=+23.655742482" watchObservedRunningTime="2026-07-21 00:41:16.049845298 +0000 UTC m=+23.655954337" Jul 21 00:41:16.906313 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:16.906220 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4nq5b" Jul 21 00:41:16.906552 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:16.906349 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4nq5b" podUID="6cfb338e-19eb-45ee-a3de-ea7311469109" Jul 21 00:41:16.906552 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:16.906364 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9r2ff" Jul 21 00:41:16.906552 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:16.906220 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzcmw" Jul 21 00:41:16.906552 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:16.906447 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9r2ff" podUID="e11587d4-f449-4b99-8355-787a52243ba4" Jul 21 00:41:16.906552 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:16.906496 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dzcmw" podUID="fda1cd2e-3d5d-4dc1-94a1-1b8ebbb4a199" Jul 21 00:41:17.016781 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:17.016573 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bctst" event={"ID":"71312c5c-a477-4500-996f-4c7cffd7fdc6","Type":"ContainerStarted","Data":"38992df022df3d4654a011dfc015137753a92436665456631941f1d7066e3a16"} Jul 21 00:41:17.018288 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:17.018260 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2rxh4" event={"ID":"6fbd5477-ce7d-444d-938b-2c54aa3c040c","Type":"ContainerStarted","Data":"26072bf991a82c54e4e184fe3e03382ce9928d72dbf8e5ec388b3f98504ce12f"} Jul 21 00:41:17.019930 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:17.019904 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fj98d" event={"ID":"70177b3d-5d28-4be4-a20d-63c49744b652","Type":"ContainerStarted","Data":"4fac896f6059a94e39b2e866c8a4efe2cedb98472942945ddecba18027ebf4d3"} Jul 21 00:41:17.021470 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:17.021431 2580 generic.go:356] "Generic (PLEG): container finished" podID="2cf3a63f5e16fba7955768004cdb0f7c" containerID="511a5bc03d069be2a8f9af87dd69f559100b8ea3598e8f6ffad8a65049328945" exitCode=0 Jul 21 00:41:17.021586 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:17.021482 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-137.ec2.internal" event={"ID":"2cf3a63f5e16fba7955768004cdb0f7c","Type":"ContainerDied","Data":"511a5bc03d069be2a8f9af87dd69f559100b8ea3598e8f6ffad8a65049328945"} Jul 21 00:41:17.022918 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:17.022894 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ntk2c" event={"ID":"e7629fc1-b058-4431-b0d3-1b465dee3b48","Type":"ContainerStarted","Data":"b316d668b84c7afbd54934883e1a19f90f711ad9446b2ba62963d23bccb674fe"} Jul 21 00:41:17.024370 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:17.024342 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-858wv" event={"ID":"df3aad65-414d-4dd0-9e23-c0c0e46e4f06","Type":"ContainerStarted","Data":"706cc45fce7486fd07ba67da1356e6c3cc62e13b790f444a782f1c0511b2730f"} Jul 21 00:41:17.025926 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:17.025908 2580 generic.go:356] "Generic (PLEG): container finished" podID="f3e26871-c996-46c1-adfb-c96c1869814b" containerID="a345e58a4a9b1f8a71e1b2b23ef39cc85eefcefd7180c2c86ad53b587675fed2" exitCode=0 Jul 21 00:41:17.026028 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:17.026002 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6zg78" event={"ID":"f3e26871-c996-46c1-adfb-c96c1869814b","Type":"ContainerDied","Data":"a345e58a4a9b1f8a71e1b2b23ef39cc85eefcefd7180c2c86ad53b587675fed2"} Jul 21 00:41:17.027631 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:17.027611 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qthms" event={"ID":"245f1c61-31ff-4104-8fa1-5d970dd40533","Type":"ContainerStarted","Data":"ed01adf37f1cde26bcb52dbd51f94d2d971037db2e316b51b028fe88c2299856"} Jul 21 00:41:17.051454 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:17.051417 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-2rxh4" podStartSLOduration=4.057008417 podStartE2EDuration="24.051401262s" totalImagesPullingTime="19.994392845s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 00:40:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:41:17.049805821 +0000 UTC m=+24.655914861" watchObservedRunningTime="2026-07-21 00:41:17.051401262 +0000 UTC m=+24.657510300" Jul 21 00:41:17.099993 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:17.099939 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-fj98d" podStartSLOduration=4.023835473 podStartE2EDuration="24.099919346s" totalImagesPullingTime="20.076083873s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 00:40:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:41:17.099361101 +0000 UTC m=+24.705470142" watchObservedRunningTime="2026-07-21 00:41:17.099919346 +0000 UTC m=+24.706028388" Jul 21 00:41:17.138150 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:17.138096 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-858wv" podStartSLOduration=4.15330928 podStartE2EDuration="24.138077175s" totalImagesPullingTime="19.984767895s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 00:40:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:41:17.137314288 +0000 UTC m=+24.743423328" watchObservedRunningTime="2026-07-21 00:41:17.138077175 +0000 UTC m=+24.744186217" Jul 21 00:41:17.150801 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:17.150735 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-ntk2c" podStartSLOduration=4.138863914 podStartE2EDuration="24.150717158s" totalImagesPullingTime="20.011853244s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:40:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:41:17.150479609 +0000 UTC m=+24.756588651" watchObservedRunningTime="2026-07-21 00:41:17.150717158 +0000 UTC m=+24.756826196" Jul 21 00:41:17.686428 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:17.686404 2580 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 21 00:41:17.821735 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:17.821597 2580 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-21T00:41:17.68642501Z","UUID":"8c8fe9dd-4a67-44ce-b2a1-31f6c06100db","Handler":null,"Name":"","Endpoint":""} Jul 21 00:41:17.823100 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:17.823083 2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 21 00:41:17.823210 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:17.823118 2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 21 00:41:18.032791 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:18.032750 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qthms" event={"ID":"245f1c61-31ff-4104-8fa1-5d970dd40533","Type":"ContainerStarted","Data":"486f84ccb8daf77f30cc08b0a6b0a8f1fa26a2c5bbf4ce765752f55a8491cff4"} Jul 21 00:41:18.034647 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:18.034619 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-137.ec2.internal" event={"ID":"2cf3a63f5e16fba7955768004cdb0f7c","Type":"ContainerStarted","Data":"14530638576d9afe60391ab9cf47259b4a477e967b9104f81bf1b3f7a2fc1f99"} Jul 21 00:41:18.049884 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:18.049843 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-137.ec2.internal" podStartSLOduration=24.049831133 podStartE2EDuration="24.049831133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 00:40:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:41:18.049733063 +0000 UTC m=+25.655842104" watchObservedRunningTime="2026-07-21 00:41:18.049831133 +0000 UTC m=+25.655940172" Jul 21 00:41:18.392517 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:18.392488 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-ntk2c" Jul 21 00:41:18.393103 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:18.393087 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-ntk2c" Jul 21 00:41:18.906219 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:18.906185 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9r2ff" Jul 21 00:41:18.906395 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:18.906195 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4nq5b" Jul 21 00:41:18.906395 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:18.906228 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzcmw" Jul 21 00:41:18.906395 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:18.906317 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9r2ff" podUID="e11587d4-f449-4b99-8355-787a52243ba4" Jul 21 00:41:18.906506 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:18.906431 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dzcmw" podUID="fda1cd2e-3d5d-4dc1-94a1-1b8ebbb4a199" Jul 21 00:41:18.906506 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:18.906499 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4nq5b" podUID="6cfb338e-19eb-45ee-a3de-ea7311469109" Jul 21 00:41:19.042139 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:19.042097 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bctst" event={"ID":"71312c5c-a477-4500-996f-4c7cffd7fdc6","Type":"ContainerStarted","Data":"7f04df2a5e6043629f377ab529c110c51c4219d09796902b4c5a500d4d39f5d2"} Jul 21 00:41:20.046812 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:20.046740 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qthms" event={"ID":"245f1c61-31ff-4104-8fa1-5d970dd40533","Type":"ContainerStarted","Data":"b1bb2818bcc5f7f3c6edc309f1c96024af1ed4835948c8c219dd788343a6e26a"} Jul 21 00:41:20.046812 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:20.046778 2580 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 21 00:41:20.066773 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:20.066731 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qthms" podStartSLOduration=4.270820219 podStartE2EDuration="27.0667204s" totalImagesPullingTime="22.795900181s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 00:40:53 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:41:20.066331958 +0000 UTC m=+27.672440999" watchObservedRunningTime="2026-07-21 00:41:20.0667204 +0000 UTC m=+27.672829439" Jul 21 00:41:20.906916 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:20.906679 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4nq5b" Jul 21 00:41:20.907085 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:20.906685 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzcmw" Jul 21 00:41:20.907085 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:20.907029 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4nq5b" podUID="6cfb338e-19eb-45ee-a3de-ea7311469109" Jul 21 00:41:20.907085 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:20.906712 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9r2ff" Jul 21 00:41:20.907085 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:20.907071 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dzcmw" podUID="fda1cd2e-3d5d-4dc1-94a1-1b8ebbb4a199" Jul 21 00:41:20.907212 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:20.907143 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9r2ff" podUID="e11587d4-f449-4b99-8355-787a52243ba4" Jul 21 00:41:21.053554 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:21.053517 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bctst" event={"ID":"71312c5c-a477-4500-996f-4c7cffd7fdc6","Type":"ContainerStarted","Data":"8792f8a51d5c5b1752d8a43b7888be62164bceb2ea8e8fee8cc028118a5fa17c"} Jul 21 00:41:21.078892 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:21.078849 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-bctst" podStartSLOduration=7.6856089359999995 podStartE2EDuration="28.0788317s" totalImagesPullingTime="20.393222764s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 00:40:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:41:21.077673776 +0000 UTC m=+28.683782816" watchObservedRunningTime="2026-07-21 00:41:21.0788317 +0000 UTC m=+28.684940733" Jul 21 00:41:22.056363 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:22.056327 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bctst" Jul 21 00:41:22.056957 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:22.056458 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bctst" Jul 21 00:41:22.056957 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:22.056497 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bctst" Jul 21 00:41:22.072177 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:22.072154 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bctst" Jul 21 00:41:22.072331 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:22.072289 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bctst" Jul 21 00:41:22.519900 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:22.519823 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9r2ff"] Jul 21 00:41:22.520104 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:22.519955 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9r2ff" Jul 21 00:41:22.520104 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:22.520064 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9r2ff" podUID="e11587d4-f449-4b99-8355-787a52243ba4" Jul 21 00:41:22.522693 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:22.522667 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4nq5b"] Jul 21 00:41:22.522852 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:22.522777 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4nq5b" Jul 21 00:41:22.522928 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:22.522879 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4nq5b" podUID="6cfb338e-19eb-45ee-a3de-ea7311469109" Jul 21 00:41:22.523926 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:22.523908 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dzcmw"] Jul 21 00:41:22.524033 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:22.523996 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzcmw" Jul 21 00:41:22.524093 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:22.524068 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dzcmw" podUID="fda1cd2e-3d5d-4dc1-94a1-1b8ebbb4a199" Jul 21 00:41:23.769307 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:23.769111 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-ntk2c" Jul 21 00:41:23.769756 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:23.769393 2580 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 21 00:41:23.769756 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:23.769728 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-ntk2c" Jul 21 00:41:23.905812 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:23.905780 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4nq5b" Jul 21 00:41:23.905995 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:23.905913 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4nq5b" podUID="6cfb338e-19eb-45ee-a3de-ea7311469109" Jul 21 00:41:23.905995 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:23.905951 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzcmw" Jul 21 00:41:23.906097 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:23.906019 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dzcmw" podUID="fda1cd2e-3d5d-4dc1-94a1-1b8ebbb4a199" Jul 21 00:41:24.061595 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:24.061520 2580 generic.go:356] "Generic (PLEG): container finished" podID="f3e26871-c996-46c1-adfb-c96c1869814b" containerID="0a29eb5ff8a2f8e00fe5b325a3b611b95bf012ec1a07b49b05c1b4e7eab9bee6" exitCode=0 Jul 21 00:41:24.061747 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:24.061589 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6zg78" event={"ID":"f3e26871-c996-46c1-adfb-c96c1869814b","Type":"ContainerDied","Data":"0a29eb5ff8a2f8e00fe5b325a3b611b95bf012ec1a07b49b05c1b4e7eab9bee6"} Jul 21 00:41:24.906118 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:24.906090 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9r2ff" Jul 21 00:41:24.906474 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:24.906261 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9r2ff" podUID="e11587d4-f449-4b99-8355-787a52243ba4" Jul 21 00:41:25.066086 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:25.066056 2580 generic.go:356] "Generic (PLEG): container finished" podID="f3e26871-c996-46c1-adfb-c96c1869814b" containerID="d9eb42d98bf29939f0193b69b95c79d4403de6d7c356d241710e5f8eb2afd179" exitCode=0 Jul 21 00:41:25.066275 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:25.066117 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6zg78" event={"ID":"f3e26871-c996-46c1-adfb-c96c1869814b","Type":"ContainerDied","Data":"d9eb42d98bf29939f0193b69b95c79d4403de6d7c356d241710e5f8eb2afd179"} Jul 21 00:41:25.906391 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:25.906363 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4nq5b" Jul 21 00:41:25.906391 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:25.906377 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzcmw" Jul 21 00:41:25.906839 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:25.906469 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4nq5b" podUID="6cfb338e-19eb-45ee-a3de-ea7311469109" Jul 21 00:41:25.906839 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:25.906619 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dzcmw" podUID="fda1cd2e-3d5d-4dc1-94a1-1b8ebbb4a199" Jul 21 00:41:26.602951 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:26.602918 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cfb338e-19eb-45ee-a3de-ea7311469109-metrics-certs\") pod \"network-metrics-daemon-4nq5b\" (UID: \"6cfb338e-19eb-45ee-a3de-ea7311469109\") " pod="openshift-multus/network-metrics-daemon-4nq5b" Jul 21 00:41:26.603101 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:26.602961 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fda1cd2e-3d5d-4dc1-94a1-1b8ebbb4a199-original-pull-secret\") pod \"global-pull-secret-syncer-dzcmw\" (UID: \"fda1cd2e-3d5d-4dc1-94a1-1b8ebbb4a199\") " pod="kube-system/global-pull-secret-syncer-dzcmw" Jul 21 00:41:26.603101 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:26.603041 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 00:41:26.603101 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:26.603064 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 00:41:26.603101 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:26.603084 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fda1cd2e-3d5d-4dc1-94a1-1b8ebbb4a199-original-pull-secret podName:fda1cd2e-3d5d-4dc1-94a1-1b8ebbb4a199 nodeName:}" failed. No retries permitted until 2026-07-21 00:41:58.603071692 +0000 UTC m=+66.209180711 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fda1cd2e-3d5d-4dc1-94a1-1b8ebbb4a199-original-pull-secret") pod "global-pull-secret-syncer-dzcmw" (UID: "fda1cd2e-3d5d-4dc1-94a1-1b8ebbb4a199") : object "kube-system"/"original-pull-secret" not registered Jul 21 00:41:26.603284 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:26.603113 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6cfb338e-19eb-45ee-a3de-ea7311469109-metrics-certs podName:6cfb338e-19eb-45ee-a3de-ea7311469109 nodeName:}" failed. No retries permitted until 2026-07-21 00:41:58.603100757 +0000 UTC m=+66.209209779 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6cfb338e-19eb-45ee-a3de-ea7311469109-metrics-certs") pod "network-metrics-daemon-4nq5b" (UID: "6cfb338e-19eb-45ee-a3de-ea7311469109") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 00:41:26.703792 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:26.703713 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wxxq\" (UniqueName: \"kubernetes.io/projected/e11587d4-f449-4b99-8355-787a52243ba4-kube-api-access-8wxxq\") pod \"network-check-target-9r2ff\" (UID: \"e11587d4-f449-4b99-8355-787a52243ba4\") " pod="openshift-network-diagnostics/network-check-target-9r2ff" Jul 21 00:41:26.704027 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:26.703851 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 00:41:26.704027 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:26.703870 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 00:41:26.704027 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:26.703884 2580 projected.go:196] Error preparing data for projected volume kube-api-access-8wxxq for pod openshift-network-diagnostics/network-check-target-9r2ff: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 00:41:26.704027 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:26.703942 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e11587d4-f449-4b99-8355-787a52243ba4-kube-api-access-8wxxq podName:e11587d4-f449-4b99-8355-787a52243ba4 nodeName:}" failed. No retries permitted until 2026-07-21 00:41:58.703925039 +0000 UTC m=+66.310034060 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-8wxxq" (UniqueName: "kubernetes.io/projected/e11587d4-f449-4b99-8355-787a52243ba4-kube-api-access-8wxxq") pod "network-check-target-9r2ff" (UID: "e11587d4-f449-4b99-8355-787a52243ba4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 00:41:26.906008 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:26.905984 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9r2ff" Jul 21 00:41:26.906166 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:26.906076 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9r2ff" podUID="e11587d4-f449-4b99-8355-787a52243ba4" Jul 21 00:41:27.072300 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:27.072205 2580 generic.go:356] "Generic (PLEG): container finished" podID="f3e26871-c996-46c1-adfb-c96c1869814b" containerID="a2aee4e4a750335123a4814ff2efe4048f83c35c7a00a518d243cc2e031058f2" exitCode=0 Jul 21 00:41:27.072300 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:27.072264 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6zg78" event={"ID":"f3e26871-c996-46c1-adfb-c96c1869814b","Type":"ContainerDied","Data":"a2aee4e4a750335123a4814ff2efe4048f83c35c7a00a518d243cc2e031058f2"} Jul 21 00:41:27.906754 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:27.906581 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4nq5b" Jul 21 00:41:27.906895 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:27.906595 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzcmw" Jul 21 00:41:27.906895 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:27.906857 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4nq5b" podUID="6cfb338e-19eb-45ee-a3de-ea7311469109" Jul 21 00:41:27.907010 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:27.906901 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dzcmw" podUID="fda1cd2e-3d5d-4dc1-94a1-1b8ebbb4a199" Jul 21 00:41:28.179831 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.179768 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-137.ec2.internal" event="NodeReady" Jul 21 00:41:28.180164 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.179884 2580 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 21 00:41:28.214620 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.214304 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-mss59"] Jul 21 00:41:28.217765 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.217744 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-tdb28"] Jul 21 00:41:28.217979 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.217881 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-mss59" Jul 21 00:41:28.220543 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.220520 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 21 00:41:28.220881 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.220857 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-hrpgb\"" Jul 21 00:41:28.221389 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.221367 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 21 00:41:28.222128 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.222098 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-rb4ph"] Jul 21 00:41:28.222247 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.222207 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-tdb28" Jul 21 00:41:28.224773 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.224745 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-555d9d7885-9rqvh"] Jul 21 00:41:28.224967 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.224937 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-rb4ph" Jul 21 00:41:28.225387 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.225119 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-vprxn\"" Jul 21 00:41:28.225387 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.225285 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 21 00:41:28.225731 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.225713 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 21 00:41:28.225731 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.225733 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 21 00:41:28.225968 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.225903 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 21 00:41:28.227686 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.227668 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-twlx9"] Jul 21 00:41:28.227991 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.227969 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-9rqvh" Jul 21 00:41:28.228166 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.228144 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 21 00:41:28.228732 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.228711 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-jhzrf\"" Jul 21 00:41:28.229005 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.228985 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 21 00:41:28.229073 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.228985 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 21 00:41:28.229597 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.229578 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 21 00:41:28.233873 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.231942 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 21 00:41:28.233873 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.232017 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 21 00:41:28.233873 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.232600 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 21 00:41:28.233873 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.232626 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 21 00:41:28.234289 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.234147 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-b8c9w"] Jul 21 00:41:28.235550 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.234293 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-twlx9" Jul 21 00:41:28.235550 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.234415 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-hjs4w\"" Jul 21 00:41:28.243744 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.241297 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-r8lcd"] Jul 21 00:41:28.243744 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.241875 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 21 00:41:28.243888 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.243821 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-rntns\"" Jul 21 00:41:28.244018 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.243995 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 21 00:41:28.244121 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.244057 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-b8c9w" Jul 21 00:41:28.244301 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.244282 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 21 00:41:28.246080 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.244923 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 21 00:41:28.246782 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.246762 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-69fd648678-dm5nn"] Jul 21 00:41:28.247994 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.247974 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-gfgjp\"" Jul 21 00:41:28.248162 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.248147 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 21 00:41:28.248396 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.248148 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-r8lcd" Jul 21 00:41:28.249724 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.249705 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 21 00:41:28.249911 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.249769 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 21 00:41:28.250476 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.250453 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 21 00:41:28.250569 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.250462 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 21 00:41:28.250735 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.250716 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-bdd745b78-7cccc"] Jul 21 00:41:28.251864 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.250835 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-69fd648678-dm5nn" Jul 21 00:41:28.253275 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.252574 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 21 00:41:28.253275 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.252845 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 21 00:41:28.253275 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.252902 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-v9zt8\"" Jul 21 00:41:28.254318 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.254296 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 21 00:41:28.254408 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.254360 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-mss59"] Jul 21 00:41:28.254408 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.254391 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-rb4ph"] Jul 21 00:41:28.254507 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.254411 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-tdb28"] Jul 21 00:41:28.254507 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.254430 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-9rqvh"] Jul 21 00:41:28.254507 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.254454 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-2vh4t"] Jul 21 00:41:28.254507 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.254500 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 21 00:41:28.254701 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.254513 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 21 00:41:28.254789 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.254741 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-bdd745b78-7cccc" Jul 21 00:41:28.255137 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.255120 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 21 00:41:28.255263 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.255216 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 21 00:41:28.255413 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.255395 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 21 00:41:28.255550 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.255527 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 21 00:41:28.255749 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.255731 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 21 00:41:28.256058 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.256040 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-rpj4j\"" Jul 21 00:41:28.258335 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.258078 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-9r7r2\"" Jul 21 00:41:28.258708 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.258691 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 21 00:41:28.259099 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.259075 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-sx8h8"] Jul 21 00:41:28.259373 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.259359 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 21 00:41:28.259794 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.259774 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 21 00:41:28.260225 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.260199 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2vh4t" Jul 21 00:41:28.263029 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.263008 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-69fd648678-dm5nn"] Jul 21 00:41:28.263125 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.263052 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-lsf2j"] Jul 21 00:41:28.266184 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.265416 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-sx8h8" Jul 21 00:41:28.266501 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.266466 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 21 00:41:28.268306 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.267274 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-nlszd"] Jul 21 00:41:28.270077 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.268356 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 21 00:41:28.270077 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.268690 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-lsf2j" Jul 21 00:41:28.270077 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.269978 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 21 00:41:28.272048 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.271576 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-ql26v\"" Jul 21 00:41:28.272302 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.272283 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 21 00:41:28.272722 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.272701 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 21 00:41:28.273633 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.273616 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 21 00:41:28.274086 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.273649 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-bdd745b78-7cccc"] Jul 21 00:41:28.274202 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.274180 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-r8lcd"] Jul 21 00:41:28.274326 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.274227 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-twlx9"] Jul 21 00:41:28.274326 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.274271 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2vh4t"] Jul 21 00:41:28.274326 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.274274 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-mcwsk\"" Jul 21 00:41:28.274326 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.274290 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nlszd"] Jul 21 00:41:28.274326 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.274313 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-b8c9w"] Jul 21 00:41:28.274326 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.274329 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-lsf2j"] Jul 21 00:41:28.274608 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.274347 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-sx8h8"] Jul 21 00:41:28.274608 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.273770 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-pxt2s\"" Jul 21 00:41:28.274608 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.273713 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nlszd" Jul 21 00:41:28.275949 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.275933 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 21 00:41:28.276672 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.276655 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 21 00:41:28.277436 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.277417 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lf5vb\"" Jul 21 00:41:28.277711 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.277695 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 21 00:41:28.277915 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.277898 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 21 00:41:28.313990 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.313972 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/826d0b75-8b78-4bf5-9a54-d71162f8e0da-tmp\") pod \"insights-operator-555d9d7885-9rqvh\" (UID: \"826d0b75-8b78-4bf5-9a54-d71162f8e0da\") " pod="openshift-insights/insights-operator-555d9d7885-9rqvh" Jul 21 00:41:28.314100 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.313999 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2ec824c1-400c-4ddb-ad88-bc5a37215aa8-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-twlx9\" (UID: \"2ec824c1-400c-4ddb-ad88-bc5a37215aa8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-twlx9" Jul 21 00:41:28.314100 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.314018 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3ee1dbf0-b4ff-4871-9432-284eb25f97b1-trusted-ca\") pod \"console-operator-565b7b569d-rb4ph\" (UID: \"3ee1dbf0-b4ff-4871-9432-284eb25f97b1\") " pod="openshift-console-operator/console-operator-565b7b569d-rb4ph" Jul 21 00:41:28.314100 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.314054 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/826d0b75-8b78-4bf5-9a54-d71162f8e0da-snapshots\") pod \"insights-operator-555d9d7885-9rqvh\" (UID: \"826d0b75-8b78-4bf5-9a54-d71162f8e0da\") " pod="openshift-insights/insights-operator-555d9d7885-9rqvh" Jul 21 00:41:28.314100 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.314069 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f7fa2eb6-5645-47f1-8199-1dd64ea512dc-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-tdb28\" (UID: \"f7fa2eb6-5645-47f1-8199-1dd64ea512dc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-tdb28" Jul 21 00:41:28.314342 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.314140 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6zdv\" (UniqueName: \"kubernetes.io/projected/f7fa2eb6-5645-47f1-8199-1dd64ea512dc-kube-api-access-z6zdv\") pod \"kube-storage-version-migrator-operator-7658d9b749-tdb28\" (UID: \"f7fa2eb6-5645-47f1-8199-1dd64ea512dc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-tdb28" Jul 21 00:41:28.314342 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.314172 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk9zj\" (UniqueName: \"kubernetes.io/projected/3ee1dbf0-b4ff-4871-9432-284eb25f97b1-kube-api-access-xk9zj\") pod \"console-operator-565b7b569d-rb4ph\" (UID: \"3ee1dbf0-b4ff-4871-9432-284eb25f97b1\") " pod="openshift-console-operator/console-operator-565b7b569d-rb4ph" Jul 21 00:41:28.314342 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.314208 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/826d0b75-8b78-4bf5-9a54-d71162f8e0da-serving-cert\") pod \"insights-operator-555d9d7885-9rqvh\" (UID: \"826d0b75-8b78-4bf5-9a54-d71162f8e0da\") " pod="openshift-insights/insights-operator-555d9d7885-9rqvh" Jul 21 00:41:28.314342 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.314243 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwvzs\" (UniqueName: \"kubernetes.io/projected/826d0b75-8b78-4bf5-9a54-d71162f8e0da-kube-api-access-zwvzs\") pod \"insights-operator-555d9d7885-9rqvh\" (UID: \"826d0b75-8b78-4bf5-9a54-d71162f8e0da\") " pod="openshift-insights/insights-operator-555d9d7885-9rqvh" Jul 21 00:41:28.314342 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.314268 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3ee1dbf0-b4ff-4871-9432-284eb25f97b1-serving-cert\") pod \"console-operator-565b7b569d-rb4ph\" (UID: \"3ee1dbf0-b4ff-4871-9432-284eb25f97b1\") " pod="openshift-console-operator/console-operator-565b7b569d-rb4ph" Jul 21 00:41:28.314342 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.314297 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8379df88-86a0-4d28-95fc-a102fff737fd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-b8c9w\" (UID: \"8379df88-86a0-4d28-95fc-a102fff737fd\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-b8c9w" Jul 21 00:41:28.314342 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.314327 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlfgw\" (UniqueName: \"kubernetes.io/projected/8379df88-86a0-4d28-95fc-a102fff737fd-kube-api-access-tlfgw\") pod \"cluster-monitoring-operator-76cf8966f8-b8c9w\" (UID: \"8379df88-86a0-4d28-95fc-a102fff737fd\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-b8c9w" Jul 21 00:41:28.314670 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.314353 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/8379df88-86a0-4d28-95fc-a102fff737fd-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-b8c9w\" (UID: \"8379df88-86a0-4d28-95fc-a102fff737fd\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-b8c9w" Jul 21 00:41:28.314670 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.314371 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6hj4\" (UniqueName: \"kubernetes.io/projected/2ec824c1-400c-4ddb-ad88-bc5a37215aa8-kube-api-access-c6hj4\") pod \"cluster-samples-operator-79b7c55f87-twlx9\" (UID: \"2ec824c1-400c-4ddb-ad88-bc5a37215aa8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-twlx9" Jul 21 00:41:28.314670 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.314386 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/826d0b75-8b78-4bf5-9a54-d71162f8e0da-service-ca-bundle\") pod \"insights-operator-555d9d7885-9rqvh\" (UID: \"826d0b75-8b78-4bf5-9a54-d71162f8e0da\") " pod="openshift-insights/insights-operator-555d9d7885-9rqvh" Jul 21 00:41:28.314670 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.314433 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3ee1dbf0-b4ff-4871-9432-284eb25f97b1-config\") pod \"console-operator-565b7b569d-rb4ph\" (UID: \"3ee1dbf0-b4ff-4871-9432-284eb25f97b1\") " pod="openshift-console-operator/console-operator-565b7b569d-rb4ph" Jul 21 00:41:28.314670 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.314482 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8sq8\" (UniqueName: \"kubernetes.io/projected/f43e318f-e1aa-4f27-971b-615e797691d9-kube-api-access-z8sq8\") pod \"volume-data-source-validator-78cf4c6f6-mss59\" (UID: \"f43e318f-e1aa-4f27-971b-615e797691d9\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-mss59" Jul 21 00:41:28.314670 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.314502 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/826d0b75-8b78-4bf5-9a54-d71162f8e0da-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-9rqvh\" (UID: \"826d0b75-8b78-4bf5-9a54-d71162f8e0da\") " pod="openshift-insights/insights-operator-555d9d7885-9rqvh" Jul 21 00:41:28.314670 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.314521 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f7fa2eb6-5645-47f1-8199-1dd64ea512dc-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-tdb28\" (UID: \"f7fa2eb6-5645-47f1-8199-1dd64ea512dc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-tdb28" Jul 21 00:41:28.415759 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.415728 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f7fa2eb6-5645-47f1-8199-1dd64ea512dc-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-tdb28\" (UID: \"f7fa2eb6-5645-47f1-8199-1dd64ea512dc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-tdb28" Jul 21 00:41:28.415759 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.415760 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2d480ddf-1712-4f08-a5ef-98d3627df334-config\") pod \"service-ca-operator-5bcbc94c78-r8lcd\" (UID: \"2d480ddf-1712-4f08-a5ef-98d3627df334\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-r8lcd" Jul 21 00:41:28.415994 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.415780 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk9zj\" (UniqueName: \"kubernetes.io/projected/3ee1dbf0-b4ff-4871-9432-284eb25f97b1-kube-api-access-xk9zj\") pod \"console-operator-565b7b569d-rb4ph\" (UID: \"3ee1dbf0-b4ff-4871-9432-284eb25f97b1\") " pod="openshift-console-operator/console-operator-565b7b569d-rb4ph" Jul 21 00:41:28.415994 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.415805 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/826d0b75-8b78-4bf5-9a54-d71162f8e0da-serving-cert\") pod \"insights-operator-555d9d7885-9rqvh\" (UID: \"826d0b75-8b78-4bf5-9a54-d71162f8e0da\") " pod="openshift-insights/insights-operator-555d9d7885-9rqvh" Jul 21 00:41:28.415994 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.415825 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6543206b-bf43-43c5-930d-c3c8834b0f2c-trusted-ca\") pod \"image-registry-bdd745b78-7cccc\" (UID: \"6543206b-bf43-43c5-930d-c3c8834b0f2c\") " pod="openshift-image-registry/image-registry-bdd745b78-7cccc" Jul 21 00:41:28.415994 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.415842 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/40960433-ccd7-4aa8-ab03-2cde39f80875-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-sx8h8\" (UID: \"40960433-ccd7-4aa8-ab03-2cde39f80875\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-sx8h8" Jul 21 00:41:28.415994 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.415857 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6543206b-bf43-43c5-930d-c3c8834b0f2c-bound-sa-token\") pod \"image-registry-bdd745b78-7cccc\" (UID: \"6543206b-bf43-43c5-930d-c3c8834b0f2c\") " pod="openshift-image-registry/image-registry-bdd745b78-7cccc" Jul 21 00:41:28.415994 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.415878 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6543206b-bf43-43c5-930d-c3c8834b0f2c-image-registry-private-configuration\") pod \"image-registry-bdd745b78-7cccc\" (UID: \"6543206b-bf43-43c5-930d-c3c8834b0f2c\") " pod="openshift-image-registry/image-registry-bdd745b78-7cccc" Jul 21 00:41:28.415994 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.415895 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49cc45d2-5087-447a-9123-3885c7bc03a2-tmp-dir\") pod \"dns-default-nlszd\" (UID: \"49cc45d2-5087-447a-9123-3885c7bc03a2\") " pod="openshift-dns/dns-default-nlszd" Jul 21 00:41:28.415994 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.415943 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/8379df88-86a0-4d28-95fc-a102fff737fd-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-b8c9w\" (UID: \"8379df88-86a0-4d28-95fc-a102fff737fd\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-b8c9w" Jul 21 00:41:28.416354 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.416005 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9970c160-3cf7-4d90-b892-1a337f23a065-service-ca-bundle\") pod \"router-default-69fd648678-dm5nn\" (UID: \"9970c160-3cf7-4d90-b892-1a337f23a065\") " pod="openshift-ingress/router-default-69fd648678-dm5nn" Jul 21 00:41:28.416354 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.416032 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9970c160-3cf7-4d90-b892-1a337f23a065-metrics-certs\") pod \"router-default-69fd648678-dm5nn\" (UID: \"9970c160-3cf7-4d90-b892-1a337f23a065\") " pod="openshift-ingress/router-default-69fd648678-dm5nn" Jul 21 00:41:28.416354 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.416065 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49cc45d2-5087-447a-9123-3885c7bc03a2-metrics-tls\") pod \"dns-default-nlszd\" (UID: \"49cc45d2-5087-447a-9123-3885c7bc03a2\") " pod="openshift-dns/dns-default-nlszd" Jul 21 00:41:28.416354 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.416093 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/826d0b75-8b78-4bf5-9a54-d71162f8e0da-tmp\") pod \"insights-operator-555d9d7885-9rqvh\" (UID: \"826d0b75-8b78-4bf5-9a54-d71162f8e0da\") " pod="openshift-insights/insights-operator-555d9d7885-9rqvh" Jul 21 00:41:28.416354 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.416169 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2d480ddf-1712-4f08-a5ef-98d3627df334-serving-cert\") pod \"service-ca-operator-5bcbc94c78-r8lcd\" (UID: \"2d480ddf-1712-4f08-a5ef-98d3627df334\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-r8lcd" Jul 21 00:41:28.416354 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.416217 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8sq8\" (UniqueName: \"kubernetes.io/projected/f43e318f-e1aa-4f27-971b-615e797691d9-kube-api-access-z8sq8\") pod \"volume-data-source-validator-78cf4c6f6-mss59\" (UID: \"f43e318f-e1aa-4f27-971b-615e797691d9\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-mss59" Jul 21 00:41:28.416354 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.416267 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bd5l\" (UniqueName: \"kubernetes.io/projected/c2e92c7d-ed96-4cba-96c5-e1032d141bd4-kube-api-access-7bd5l\") pod \"network-check-source-7bc5b6bf85-lsf2j\" (UID: \"c2e92c7d-ed96-4cba-96c5-e1032d141bd4\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-lsf2j" Jul 21 00:41:28.416354 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.416301 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/826d0b75-8b78-4bf5-9a54-d71162f8e0da-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-9rqvh\" (UID: \"826d0b75-8b78-4bf5-9a54-d71162f8e0da\") " pod="openshift-insights/insights-operator-555d9d7885-9rqvh" Jul 21 00:41:28.416354 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.416323 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f7fa2eb6-5645-47f1-8199-1dd64ea512dc-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-tdb28\" (UID: \"f7fa2eb6-5645-47f1-8199-1dd64ea512dc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-tdb28" Jul 21 00:41:28.416354 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.416327 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f7fa2eb6-5645-47f1-8199-1dd64ea512dc-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-tdb28\" (UID: \"f7fa2eb6-5645-47f1-8199-1dd64ea512dc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-tdb28" Jul 21 00:41:28.416354 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.416352 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3ee1dbf0-b4ff-4871-9432-284eb25f97b1-trusted-ca\") pod \"console-operator-565b7b569d-rb4ph\" (UID: \"3ee1dbf0-b4ff-4871-9432-284eb25f97b1\") " pod="openshift-console-operator/console-operator-565b7b569d-rb4ph" Jul 21 00:41:28.416830 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.416380 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2bj4\" (UniqueName: \"kubernetes.io/projected/6543206b-bf43-43c5-930d-c3c8834b0f2c-kube-api-access-k2bj4\") pod \"image-registry-bdd745b78-7cccc\" (UID: \"6543206b-bf43-43c5-930d-c3c8834b0f2c\") " pod="openshift-image-registry/image-registry-bdd745b78-7cccc" Jul 21 00:41:28.416830 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.416403 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a625da2a-a501-4fa3-a8c0-21ac34950978-cert\") pod \"ingress-canary-2vh4t\" (UID: \"a625da2a-a501-4fa3-a8c0-21ac34950978\") " pod="openshift-ingress-canary/ingress-canary-2vh4t" Jul 21 00:41:28.416830 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.416414 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/826d0b75-8b78-4bf5-9a54-d71162f8e0da-tmp\") pod \"insights-operator-555d9d7885-9rqvh\" (UID: \"826d0b75-8b78-4bf5-9a54-d71162f8e0da\") " pod="openshift-insights/insights-operator-555d9d7885-9rqvh" Jul 21 00:41:28.416830 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.416432 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/826d0b75-8b78-4bf5-9a54-d71162f8e0da-snapshots\") pod \"insights-operator-555d9d7885-9rqvh\" (UID: \"826d0b75-8b78-4bf5-9a54-d71162f8e0da\") " pod="openshift-insights/insights-operator-555d9d7885-9rqvh" Jul 21 00:41:28.416830 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.416458 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/49cc45d2-5087-447a-9123-3885c7bc03a2-config-volume\") pod \"dns-default-nlszd\" (UID: \"49cc45d2-5087-447a-9123-3885c7bc03a2\") " pod="openshift-dns/dns-default-nlszd" Jul 21 00:41:28.416830 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.416491 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6543206b-bf43-43c5-930d-c3c8834b0f2c-installation-pull-secrets\") pod \"image-registry-bdd745b78-7cccc\" (UID: \"6543206b-bf43-43c5-930d-c3c8834b0f2c\") " pod="openshift-image-registry/image-registry-bdd745b78-7cccc" Jul 21 00:41:28.416830 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.416567 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6zdv\" (UniqueName: \"kubernetes.io/projected/f7fa2eb6-5645-47f1-8199-1dd64ea512dc-kube-api-access-z6zdv\") pod \"kube-storage-version-migrator-operator-7658d9b749-tdb28\" (UID: \"f7fa2eb6-5645-47f1-8199-1dd64ea512dc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-tdb28" Jul 21 00:41:28.416830 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.416595 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6543206b-bf43-43c5-930d-c3c8834b0f2c-registry-tls\") pod \"image-registry-bdd745b78-7cccc\" (UID: \"6543206b-bf43-43c5-930d-c3c8834b0f2c\") " pod="openshift-image-registry/image-registry-bdd745b78-7cccc" Jul 21 00:41:28.416830 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.416627 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwvzs\" (UniqueName: \"kubernetes.io/projected/826d0b75-8b78-4bf5-9a54-d71162f8e0da-kube-api-access-zwvzs\") pod \"insights-operator-555d9d7885-9rqvh\" (UID: \"826d0b75-8b78-4bf5-9a54-d71162f8e0da\") " pod="openshift-insights/insights-operator-555d9d7885-9rqvh" Jul 21 00:41:28.416830 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.416658 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3ee1dbf0-b4ff-4871-9432-284eb25f97b1-serving-cert\") pod \"console-operator-565b7b569d-rb4ph\" (UID: \"3ee1dbf0-b4ff-4871-9432-284eb25f97b1\") " pod="openshift-console-operator/console-operator-565b7b569d-rb4ph" Jul 21 00:41:28.416830 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.416684 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqbq6\" (UniqueName: \"kubernetes.io/projected/2d480ddf-1712-4f08-a5ef-98d3627df334-kube-api-access-bqbq6\") pod \"service-ca-operator-5bcbc94c78-r8lcd\" (UID: \"2d480ddf-1712-4f08-a5ef-98d3627df334\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-r8lcd" Jul 21 00:41:28.416830 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.416710 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/9970c160-3cf7-4d90-b892-1a337f23a065-stats-auth\") pod \"router-default-69fd648678-dm5nn\" (UID: \"9970c160-3cf7-4d90-b892-1a337f23a065\") " pod="openshift-ingress/router-default-69fd648678-dm5nn" Jul 21 00:41:28.416830 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.416733 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8379df88-86a0-4d28-95fc-a102fff737fd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-b8c9w\" (UID: \"8379df88-86a0-4d28-95fc-a102fff737fd\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-b8c9w" Jul 21 00:41:28.417481 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.416754 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/40960433-ccd7-4aa8-ab03-2cde39f80875-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-sx8h8\" (UID: \"40960433-ccd7-4aa8-ab03-2cde39f80875\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-sx8h8" Jul 21 00:41:28.417481 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.416775 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnfxm\" (UniqueName: \"kubernetes.io/projected/49cc45d2-5087-447a-9123-3885c7bc03a2-kube-api-access-dnfxm\") pod \"dns-default-nlszd\" (UID: \"49cc45d2-5087-447a-9123-3885c7bc03a2\") " pod="openshift-dns/dns-default-nlszd" Jul 21 00:41:28.417481 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.416777 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/8379df88-86a0-4d28-95fc-a102fff737fd-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-b8c9w\" (UID: \"8379df88-86a0-4d28-95fc-a102fff737fd\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-b8c9w" Jul 21 00:41:28.417481 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.416795 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6543206b-bf43-43c5-930d-c3c8834b0f2c-ca-trust-extracted\") pod \"image-registry-bdd745b78-7cccc\" (UID: \"6543206b-bf43-43c5-930d-c3c8834b0f2c\") " pod="openshift-image-registry/image-registry-bdd745b78-7cccc" Jul 21 00:41:28.417481 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.416822 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlfgw\" (UniqueName: \"kubernetes.io/projected/8379df88-86a0-4d28-95fc-a102fff737fd-kube-api-access-tlfgw\") pod \"cluster-monitoring-operator-76cf8966f8-b8c9w\" (UID: \"8379df88-86a0-4d28-95fc-a102fff737fd\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-b8c9w" Jul 21 00:41:28.417481 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.416842 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/9970c160-3cf7-4d90-b892-1a337f23a065-default-certificate\") pod \"router-default-69fd648678-dm5nn\" (UID: \"9970c160-3cf7-4d90-b892-1a337f23a065\") " pod="openshift-ingress/router-default-69fd648678-dm5nn" Jul 21 00:41:28.417481 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.416866 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6hj4\" (UniqueName: \"kubernetes.io/projected/2ec824c1-400c-4ddb-ad88-bc5a37215aa8-kube-api-access-c6hj4\") pod \"cluster-samples-operator-79b7c55f87-twlx9\" (UID: \"2ec824c1-400c-4ddb-ad88-bc5a37215aa8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-twlx9" Jul 21 00:41:28.417481 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.416887 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtf9t\" (UniqueName: \"kubernetes.io/projected/9970c160-3cf7-4d90-b892-1a337f23a065-kube-api-access-vtf9t\") pod \"router-default-69fd648678-dm5nn\" (UID: \"9970c160-3cf7-4d90-b892-1a337f23a065\") " pod="openshift-ingress/router-default-69fd648678-dm5nn" Jul 21 00:41:28.417481 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.416910 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/826d0b75-8b78-4bf5-9a54-d71162f8e0da-service-ca-bundle\") pod \"insights-operator-555d9d7885-9rqvh\" (UID: \"826d0b75-8b78-4bf5-9a54-d71162f8e0da\") " pod="openshift-insights/insights-operator-555d9d7885-9rqvh" Jul 21 00:41:28.417481 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.416961 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3ee1dbf0-b4ff-4871-9432-284eb25f97b1-config\") pod \"console-operator-565b7b569d-rb4ph\" (UID: \"3ee1dbf0-b4ff-4871-9432-284eb25f97b1\") " pod="openshift-console-operator/console-operator-565b7b569d-rb4ph" Jul 21 00:41:28.417481 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.417003 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2ec824c1-400c-4ddb-ad88-bc5a37215aa8-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-twlx9\" (UID: \"2ec824c1-400c-4ddb-ad88-bc5a37215aa8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-twlx9" Jul 21 00:41:28.417481 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.417021 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/826d0b75-8b78-4bf5-9a54-d71162f8e0da-snapshots\") pod \"insights-operator-555d9d7885-9rqvh\" (UID: \"826d0b75-8b78-4bf5-9a54-d71162f8e0da\") " pod="openshift-insights/insights-operator-555d9d7885-9rqvh" Jul 21 00:41:28.417481 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.417025 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6543206b-bf43-43c5-930d-c3c8834b0f2c-registry-certificates\") pod \"image-registry-bdd745b78-7cccc\" (UID: \"6543206b-bf43-43c5-930d-c3c8834b0f2c\") " pod="openshift-image-registry/image-registry-bdd745b78-7cccc" Jul 21 00:41:28.417974 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.417070 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/826d0b75-8b78-4bf5-9a54-d71162f8e0da-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-9rqvh\" (UID: \"826d0b75-8b78-4bf5-9a54-d71162f8e0da\") " pod="openshift-insights/insights-operator-555d9d7885-9rqvh" Jul 21 00:41:28.417974 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.417092 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzz7x\" (UniqueName: \"kubernetes.io/projected/a625da2a-a501-4fa3-a8c0-21ac34950978-kube-api-access-tzz7x\") pod \"ingress-canary-2vh4t\" (UID: \"a625da2a-a501-4fa3-a8c0-21ac34950978\") " pod="openshift-ingress-canary/ingress-canary-2vh4t" Jul 21 00:41:28.417974 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:28.417115 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 21 00:41:28.417974 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:28.417198 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 21 00:41:28.417974 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:28.417266 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8379df88-86a0-4d28-95fc-a102fff737fd-cluster-monitoring-operator-tls podName:8379df88-86a0-4d28-95fc-a102fff737fd nodeName:}" failed. No retries permitted until 2026-07-21 00:41:28.91722635 +0000 UTC m=+36.523335381 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8379df88-86a0-4d28-95fc-a102fff737fd-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-b8c9w" (UID: "8379df88-86a0-4d28-95fc-a102fff737fd") : secret "cluster-monitoring-operator-tls" not found Jul 21 00:41:28.417974 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:28.417285 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ec824c1-400c-4ddb-ad88-bc5a37215aa8-samples-operator-tls podName:2ec824c1-400c-4ddb-ad88-bc5a37215aa8 nodeName:}" failed. No retries permitted until 2026-07-21 00:41:28.917277602 +0000 UTC m=+36.523386622 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2ec824c1-400c-4ddb-ad88-bc5a37215aa8-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-twlx9" (UID: "2ec824c1-400c-4ddb-ad88-bc5a37215aa8") : secret "samples-operator-tls" not found Jul 21 00:41:28.417974 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.417337 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3ee1dbf0-b4ff-4871-9432-284eb25f97b1-trusted-ca\") pod \"console-operator-565b7b569d-rb4ph\" (UID: \"3ee1dbf0-b4ff-4871-9432-284eb25f97b1\") " pod="openshift-console-operator/console-operator-565b7b569d-rb4ph" Jul 21 00:41:28.417974 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.417719 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/826d0b75-8b78-4bf5-9a54-d71162f8e0da-service-ca-bundle\") pod \"insights-operator-555d9d7885-9rqvh\" (UID: \"826d0b75-8b78-4bf5-9a54-d71162f8e0da\") " pod="openshift-insights/insights-operator-555d9d7885-9rqvh" Jul 21 00:41:28.417974 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.417760 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3ee1dbf0-b4ff-4871-9432-284eb25f97b1-config\") pod \"console-operator-565b7b569d-rb4ph\" (UID: \"3ee1dbf0-b4ff-4871-9432-284eb25f97b1\") " pod="openshift-console-operator/console-operator-565b7b569d-rb4ph" Jul 21 00:41:28.420293 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.420272 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f7fa2eb6-5645-47f1-8199-1dd64ea512dc-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-tdb28\" (UID: \"f7fa2eb6-5645-47f1-8199-1dd64ea512dc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-tdb28" Jul 21 00:41:28.420403 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.420348 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3ee1dbf0-b4ff-4871-9432-284eb25f97b1-serving-cert\") pod \"console-operator-565b7b569d-rb4ph\" (UID: \"3ee1dbf0-b4ff-4871-9432-284eb25f97b1\") " pod="openshift-console-operator/console-operator-565b7b569d-rb4ph" Jul 21 00:41:28.420403 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.420386 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/826d0b75-8b78-4bf5-9a54-d71162f8e0da-serving-cert\") pod \"insights-operator-555d9d7885-9rqvh\" (UID: \"826d0b75-8b78-4bf5-9a54-d71162f8e0da\") " pod="openshift-insights/insights-operator-555d9d7885-9rqvh" Jul 21 00:41:28.428721 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.428697 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk9zj\" (UniqueName: \"kubernetes.io/projected/3ee1dbf0-b4ff-4871-9432-284eb25f97b1-kube-api-access-xk9zj\") pod \"console-operator-565b7b569d-rb4ph\" (UID: \"3ee1dbf0-b4ff-4871-9432-284eb25f97b1\") " pod="openshift-console-operator/console-operator-565b7b569d-rb4ph" Jul 21 00:41:28.429629 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.429603 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwvzs\" (UniqueName: \"kubernetes.io/projected/826d0b75-8b78-4bf5-9a54-d71162f8e0da-kube-api-access-zwvzs\") pod \"insights-operator-555d9d7885-9rqvh\" (UID: \"826d0b75-8b78-4bf5-9a54-d71162f8e0da\") " pod="openshift-insights/insights-operator-555d9d7885-9rqvh" Jul 21 00:41:28.429725 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.429647 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6zdv\" (UniqueName: \"kubernetes.io/projected/f7fa2eb6-5645-47f1-8199-1dd64ea512dc-kube-api-access-z6zdv\") pod \"kube-storage-version-migrator-operator-7658d9b749-tdb28\" (UID: \"f7fa2eb6-5645-47f1-8199-1dd64ea512dc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-tdb28" Jul 21 00:41:28.430114 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.430058 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlfgw\" (UniqueName: \"kubernetes.io/projected/8379df88-86a0-4d28-95fc-a102fff737fd-kube-api-access-tlfgw\") pod \"cluster-monitoring-operator-76cf8966f8-b8c9w\" (UID: \"8379df88-86a0-4d28-95fc-a102fff737fd\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-b8c9w" Jul 21 00:41:28.430622 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.430597 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8sq8\" (UniqueName: \"kubernetes.io/projected/f43e318f-e1aa-4f27-971b-615e797691d9-kube-api-access-z8sq8\") pod \"volume-data-source-validator-78cf4c6f6-mss59\" (UID: \"f43e318f-e1aa-4f27-971b-615e797691d9\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-mss59" Jul 21 00:41:28.430700 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.430673 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6hj4\" (UniqueName: \"kubernetes.io/projected/2ec824c1-400c-4ddb-ad88-bc5a37215aa8-kube-api-access-c6hj4\") pod \"cluster-samples-operator-79b7c55f87-twlx9\" (UID: \"2ec824c1-400c-4ddb-ad88-bc5a37215aa8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-twlx9" Jul 21 00:41:28.518423 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.518387 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6543206b-bf43-43c5-930d-c3c8834b0f2c-registry-certificates\") pod \"image-registry-bdd745b78-7cccc\" (UID: \"6543206b-bf43-43c5-930d-c3c8834b0f2c\") " pod="openshift-image-registry/image-registry-bdd745b78-7cccc" Jul 21 00:41:28.518569 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.518436 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzz7x\" (UniqueName: \"kubernetes.io/projected/a625da2a-a501-4fa3-a8c0-21ac34950978-kube-api-access-tzz7x\") pod \"ingress-canary-2vh4t\" (UID: \"a625da2a-a501-4fa3-a8c0-21ac34950978\") " pod="openshift-ingress-canary/ingress-canary-2vh4t" Jul 21 00:41:28.518569 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.518474 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2d480ddf-1712-4f08-a5ef-98d3627df334-config\") pod \"service-ca-operator-5bcbc94c78-r8lcd\" (UID: \"2d480ddf-1712-4f08-a5ef-98d3627df334\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-r8lcd" Jul 21 00:41:28.518569 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.518505 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6543206b-bf43-43c5-930d-c3c8834b0f2c-trusted-ca\") pod \"image-registry-bdd745b78-7cccc\" (UID: \"6543206b-bf43-43c5-930d-c3c8834b0f2c\") " pod="openshift-image-registry/image-registry-bdd745b78-7cccc" Jul 21 00:41:28.518999 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.518969 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/40960433-ccd7-4aa8-ab03-2cde39f80875-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-sx8h8\" (UID: \"40960433-ccd7-4aa8-ab03-2cde39f80875\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-sx8h8" Jul 21 00:41:28.519107 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.519032 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6543206b-bf43-43c5-930d-c3c8834b0f2c-bound-sa-token\") pod \"image-registry-bdd745b78-7cccc\" (UID: \"6543206b-bf43-43c5-930d-c3c8834b0f2c\") " pod="openshift-image-registry/image-registry-bdd745b78-7cccc" Jul 21 00:41:28.519107 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.519068 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6543206b-bf43-43c5-930d-c3c8834b0f2c-image-registry-private-configuration\") pod \"image-registry-bdd745b78-7cccc\" (UID: \"6543206b-bf43-43c5-930d-c3c8834b0f2c\") " pod="openshift-image-registry/image-registry-bdd745b78-7cccc" Jul 21 00:41:28.519107 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.519093 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49cc45d2-5087-447a-9123-3885c7bc03a2-tmp-dir\") pod \"dns-default-nlszd\" (UID: \"49cc45d2-5087-447a-9123-3885c7bc03a2\") " pod="openshift-dns/dns-default-nlszd" Jul 21 00:41:28.519298 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.519131 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9970c160-3cf7-4d90-b892-1a337f23a065-service-ca-bundle\") pod \"router-default-69fd648678-dm5nn\" (UID: \"9970c160-3cf7-4d90-b892-1a337f23a065\") " pod="openshift-ingress/router-default-69fd648678-dm5nn" Jul 21 00:41:28.519298 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.519163 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9970c160-3cf7-4d90-b892-1a337f23a065-metrics-certs\") pod \"router-default-69fd648678-dm5nn\" (UID: \"9970c160-3cf7-4d90-b892-1a337f23a065\") " pod="openshift-ingress/router-default-69fd648678-dm5nn" Jul 21 00:41:28.519298 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.519198 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49cc45d2-5087-447a-9123-3885c7bc03a2-metrics-tls\") pod \"dns-default-nlszd\" (UID: \"49cc45d2-5087-447a-9123-3885c7bc03a2\") " pod="openshift-dns/dns-default-nlszd" Jul 21 00:41:28.519298 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.519225 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2d480ddf-1712-4f08-a5ef-98d3627df334-serving-cert\") pod \"service-ca-operator-5bcbc94c78-r8lcd\" (UID: \"2d480ddf-1712-4f08-a5ef-98d3627df334\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-r8lcd" Jul 21 00:41:28.519298 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.519282 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bd5l\" (UniqueName: \"kubernetes.io/projected/c2e92c7d-ed96-4cba-96c5-e1032d141bd4-kube-api-access-7bd5l\") pod \"network-check-source-7bc5b6bf85-lsf2j\" (UID: \"c2e92c7d-ed96-4cba-96c5-e1032d141bd4\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-lsf2j" Jul 21 00:41:28.519538 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.519378 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2d480ddf-1712-4f08-a5ef-98d3627df334-config\") pod \"service-ca-operator-5bcbc94c78-r8lcd\" (UID: \"2d480ddf-1712-4f08-a5ef-98d3627df334\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-r8lcd" Jul 21 00:41:28.519538 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.519402 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6543206b-bf43-43c5-930d-c3c8834b0f2c-registry-certificates\") pod \"image-registry-bdd745b78-7cccc\" (UID: \"6543206b-bf43-43c5-930d-c3c8834b0f2c\") " pod="openshift-image-registry/image-registry-bdd745b78-7cccc" Jul 21 00:41:28.519762 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:28.519740 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 00:41:28.519846 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:28.519833 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49cc45d2-5087-447a-9123-3885c7bc03a2-metrics-tls podName:49cc45d2-5087-447a-9123-3885c7bc03a2 nodeName:}" failed. No retries permitted until 2026-07-21 00:41:29.01980515 +0000 UTC m=+36.625914184 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/49cc45d2-5087-447a-9123-3885c7bc03a2-metrics-tls") pod "dns-default-nlszd" (UID: "49cc45d2-5087-447a-9123-3885c7bc03a2") : secret "dns-default-metrics-tls" not found Jul 21 00:41:28.519926 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.519881 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2bj4\" (UniqueName: \"kubernetes.io/projected/6543206b-bf43-43c5-930d-c3c8834b0f2c-kube-api-access-k2bj4\") pod \"image-registry-bdd745b78-7cccc\" (UID: \"6543206b-bf43-43c5-930d-c3c8834b0f2c\") " pod="openshift-image-registry/image-registry-bdd745b78-7cccc" Jul 21 00:41:28.519926 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:28.519909 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 21 00:41:28.520031 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:28.520001 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9970c160-3cf7-4d90-b892-1a337f23a065-metrics-certs podName:9970c160-3cf7-4d90-b892-1a337f23a065 nodeName:}" failed. No retries permitted until 2026-07-21 00:41:29.01997714 +0000 UTC m=+36.626086162 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9970c160-3cf7-4d90-b892-1a337f23a065-metrics-certs") pod "router-default-69fd648678-dm5nn" (UID: "9970c160-3cf7-4d90-b892-1a337f23a065") : secret "router-metrics-certs-default" not found Jul 21 00:41:28.520031 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.519827 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49cc45d2-5087-447a-9123-3885c7bc03a2-tmp-dir\") pod \"dns-default-nlszd\" (UID: \"49cc45d2-5087-447a-9123-3885c7bc03a2\") " pod="openshift-dns/dns-default-nlszd" Jul 21 00:41:28.520136 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.520053 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a625da2a-a501-4fa3-a8c0-21ac34950978-cert\") pod \"ingress-canary-2vh4t\" (UID: \"a625da2a-a501-4fa3-a8c0-21ac34950978\") " pod="openshift-ingress-canary/ingress-canary-2vh4t" Jul 21 00:41:28.520136 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.520091 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/49cc45d2-5087-447a-9123-3885c7bc03a2-config-volume\") pod \"dns-default-nlszd\" (UID: \"49cc45d2-5087-447a-9123-3885c7bc03a2\") " pod="openshift-dns/dns-default-nlszd" Jul 21 00:41:28.520227 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.520163 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6543206b-bf43-43c5-930d-c3c8834b0f2c-installation-pull-secrets\") pod \"image-registry-bdd745b78-7cccc\" (UID: \"6543206b-bf43-43c5-930d-c3c8834b0f2c\") " pod="openshift-image-registry/image-registry-bdd745b78-7cccc" Jul 21 00:41:28.520299 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.520227 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6543206b-bf43-43c5-930d-c3c8834b0f2c-registry-tls\") pod \"image-registry-bdd745b78-7cccc\" (UID: \"6543206b-bf43-43c5-930d-c3c8834b0f2c\") " pod="openshift-image-registry/image-registry-bdd745b78-7cccc" Jul 21 00:41:28.520358 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.520294 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqbq6\" (UniqueName: \"kubernetes.io/projected/2d480ddf-1712-4f08-a5ef-98d3627df334-kube-api-access-bqbq6\") pod \"service-ca-operator-5bcbc94c78-r8lcd\" (UID: \"2d480ddf-1712-4f08-a5ef-98d3627df334\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-r8lcd" Jul 21 00:41:28.520358 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.520327 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/9970c160-3cf7-4d90-b892-1a337f23a065-stats-auth\") pod \"router-default-69fd648678-dm5nn\" (UID: \"9970c160-3cf7-4d90-b892-1a337f23a065\") " pod="openshift-ingress/router-default-69fd648678-dm5nn" Jul 21 00:41:28.520459 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.520382 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/40960433-ccd7-4aa8-ab03-2cde39f80875-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-sx8h8\" (UID: \"40960433-ccd7-4aa8-ab03-2cde39f80875\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-sx8h8" Jul 21 00:41:28.520459 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.520418 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnfxm\" (UniqueName: \"kubernetes.io/projected/49cc45d2-5087-447a-9123-3885c7bc03a2-kube-api-access-dnfxm\") pod \"dns-default-nlszd\" (UID: \"49cc45d2-5087-447a-9123-3885c7bc03a2\") " pod="openshift-dns/dns-default-nlszd" Jul 21 00:41:28.520459 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.520448 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6543206b-bf43-43c5-930d-c3c8834b0f2c-ca-trust-extracted\") pod \"image-registry-bdd745b78-7cccc\" (UID: \"6543206b-bf43-43c5-930d-c3c8834b0f2c\") " pod="openshift-image-registry/image-registry-bdd745b78-7cccc" Jul 21 00:41:28.520613 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.520489 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/9970c160-3cf7-4d90-b892-1a337f23a065-default-certificate\") pod \"router-default-69fd648678-dm5nn\" (UID: \"9970c160-3cf7-4d90-b892-1a337f23a065\") " pod="openshift-ingress/router-default-69fd648678-dm5nn" Jul 21 00:41:28.520613 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.520534 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtf9t\" (UniqueName: \"kubernetes.io/projected/9970c160-3cf7-4d90-b892-1a337f23a065-kube-api-access-vtf9t\") pod \"router-default-69fd648678-dm5nn\" (UID: \"9970c160-3cf7-4d90-b892-1a337f23a065\") " pod="openshift-ingress/router-default-69fd648678-dm5nn" Jul 21 00:41:28.520941 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:28.520923 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9970c160-3cf7-4d90-b892-1a337f23a065-service-ca-bundle podName:9970c160-3cf7-4d90-b892-1a337f23a065 nodeName:}" failed. No retries permitted until 2026-07-21 00:41:29.020907225 +0000 UTC m=+36.627016263 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9970c160-3cf7-4d90-b892-1a337f23a065-service-ca-bundle") pod "router-default-69fd648678-dm5nn" (UID: "9970c160-3cf7-4d90-b892-1a337f23a065") : configmap references non-existent config key: service-ca.crt Jul 21 00:41:28.521038 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:28.521001 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 21 00:41:28.521095 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:28.521044 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40960433-ccd7-4aa8-ab03-2cde39f80875-networking-console-plugin-cert podName:40960433-ccd7-4aa8-ab03-2cde39f80875 nodeName:}" failed. No retries permitted until 2026-07-21 00:41:29.021033066 +0000 UTC m=+36.627142096 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/40960433-ccd7-4aa8-ab03-2cde39f80875-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-sx8h8" (UID: "40960433-ccd7-4aa8-ab03-2cde39f80875") : secret "networking-console-plugin-cert" not found Jul 21 00:41:28.521409 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.521381 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/49cc45d2-5087-447a-9123-3885c7bc03a2-config-volume\") pod \"dns-default-nlszd\" (UID: \"49cc45d2-5087-447a-9123-3885c7bc03a2\") " pod="openshift-dns/dns-default-nlszd" Jul 21 00:41:28.521489 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.521421 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6543206b-bf43-43c5-930d-c3c8834b0f2c-trusted-ca\") pod \"image-registry-bdd745b78-7cccc\" (UID: \"6543206b-bf43-43c5-930d-c3c8834b0f2c\") " pod="openshift-image-registry/image-registry-bdd745b78-7cccc" Jul 21 00:41:28.521810 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.521786 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6543206b-bf43-43c5-930d-c3c8834b0f2c-ca-trust-extracted\") pod \"image-registry-bdd745b78-7cccc\" (UID: \"6543206b-bf43-43c5-930d-c3c8834b0f2c\") " pod="openshift-image-registry/image-registry-bdd745b78-7cccc" Jul 21 00:41:28.522538 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.522313 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2d480ddf-1712-4f08-a5ef-98d3627df334-serving-cert\") pod \"service-ca-operator-5bcbc94c78-r8lcd\" (UID: \"2d480ddf-1712-4f08-a5ef-98d3627df334\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-r8lcd" Jul 21 00:41:28.524965 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:28.522733 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 21 00:41:28.524965 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:28.522752 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bdd745b78-7cccc: secret "image-registry-tls" not found Jul 21 00:41:28.524965 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.522779 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/40960433-ccd7-4aa8-ab03-2cde39f80875-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-sx8h8\" (UID: \"40960433-ccd7-4aa8-ab03-2cde39f80875\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-sx8h8" Jul 21 00:41:28.524965 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:28.522802 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6543206b-bf43-43c5-930d-c3c8834b0f2c-registry-tls podName:6543206b-bf43-43c5-930d-c3c8834b0f2c nodeName:}" failed. No retries permitted until 2026-07-21 00:41:29.022788594 +0000 UTC m=+36.628897612 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6543206b-bf43-43c5-930d-c3c8834b0f2c-registry-tls") pod "image-registry-bdd745b78-7cccc" (UID: "6543206b-bf43-43c5-930d-c3c8834b0f2c") : secret "image-registry-tls" not found Jul 21 00:41:28.524965 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.524091 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/9970c160-3cf7-4d90-b892-1a337f23a065-default-certificate\") pod \"router-default-69fd648678-dm5nn\" (UID: \"9970c160-3cf7-4d90-b892-1a337f23a065\") " pod="openshift-ingress/router-default-69fd648678-dm5nn" Jul 21 00:41:28.525470 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.525450 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/9970c160-3cf7-4d90-b892-1a337f23a065-stats-auth\") pod \"router-default-69fd648678-dm5nn\" (UID: \"9970c160-3cf7-4d90-b892-1a337f23a065\") " pod="openshift-ingress/router-default-69fd648678-dm5nn" Jul 21 00:41:28.526147 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.526114 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6543206b-bf43-43c5-930d-c3c8834b0f2c-installation-pull-secrets\") pod \"image-registry-bdd745b78-7cccc\" (UID: \"6543206b-bf43-43c5-930d-c3c8834b0f2c\") " pod="openshift-image-registry/image-registry-bdd745b78-7cccc" Jul 21 00:41:28.526496 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.526445 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a625da2a-a501-4fa3-a8c0-21ac34950978-cert\") pod \"ingress-canary-2vh4t\" (UID: \"a625da2a-a501-4fa3-a8c0-21ac34950978\") " pod="openshift-ingress-canary/ingress-canary-2vh4t" Jul 21 00:41:28.527181 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.527073 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6543206b-bf43-43c5-930d-c3c8834b0f2c-image-registry-private-configuration\") pod \"image-registry-bdd745b78-7cccc\" (UID: \"6543206b-bf43-43c5-930d-c3c8834b0f2c\") " pod="openshift-image-registry/image-registry-bdd745b78-7cccc" Jul 21 00:41:28.528966 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.528928 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6543206b-bf43-43c5-930d-c3c8834b0f2c-bound-sa-token\") pod \"image-registry-bdd745b78-7cccc\" (UID: \"6543206b-bf43-43c5-930d-c3c8834b0f2c\") " pod="openshift-image-registry/image-registry-bdd745b78-7cccc" Jul 21 00:41:28.529626 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.529598 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzz7x\" (UniqueName: \"kubernetes.io/projected/a625da2a-a501-4fa3-a8c0-21ac34950978-kube-api-access-tzz7x\") pod \"ingress-canary-2vh4t\" (UID: \"a625da2a-a501-4fa3-a8c0-21ac34950978\") " pod="openshift-ingress-canary/ingress-canary-2vh4t" Jul 21 00:41:28.529905 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.529784 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bd5l\" (UniqueName: \"kubernetes.io/projected/c2e92c7d-ed96-4cba-96c5-e1032d141bd4-kube-api-access-7bd5l\") pod \"network-check-source-7bc5b6bf85-lsf2j\" (UID: \"c2e92c7d-ed96-4cba-96c5-e1032d141bd4\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-lsf2j" Jul 21 00:41:28.530896 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.530852 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnfxm\" (UniqueName: \"kubernetes.io/projected/49cc45d2-5087-447a-9123-3885c7bc03a2-kube-api-access-dnfxm\") pod \"dns-default-nlszd\" (UID: \"49cc45d2-5087-447a-9123-3885c7bc03a2\") " pod="openshift-dns/dns-default-nlszd" Jul 21 00:41:28.531013 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.530993 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqbq6\" (UniqueName: \"kubernetes.io/projected/2d480ddf-1712-4f08-a5ef-98d3627df334-kube-api-access-bqbq6\") pod \"service-ca-operator-5bcbc94c78-r8lcd\" (UID: \"2d480ddf-1712-4f08-a5ef-98d3627df334\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-r8lcd" Jul 21 00:41:28.531340 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.531315 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtf9t\" (UniqueName: \"kubernetes.io/projected/9970c160-3cf7-4d90-b892-1a337f23a065-kube-api-access-vtf9t\") pod \"router-default-69fd648678-dm5nn\" (UID: \"9970c160-3cf7-4d90-b892-1a337f23a065\") " pod="openshift-ingress/router-default-69fd648678-dm5nn" Jul 21 00:41:28.531921 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.531900 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2bj4\" (UniqueName: \"kubernetes.io/projected/6543206b-bf43-43c5-930d-c3c8834b0f2c-kube-api-access-k2bj4\") pod \"image-registry-bdd745b78-7cccc\" (UID: \"6543206b-bf43-43c5-930d-c3c8834b0f2c\") " pod="openshift-image-registry/image-registry-bdd745b78-7cccc" Jul 21 00:41:28.531996 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.531974 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-mss59" Jul 21 00:41:28.545256 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.545226 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-tdb28" Jul 21 00:41:28.556166 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.556143 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-rb4ph" Jul 21 00:41:28.566845 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.566801 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-9rqvh" Jul 21 00:41:28.591076 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.590906 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-r8lcd" Jul 21 00:41:28.610580 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.610061 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2vh4t" Jul 21 00:41:28.623334 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.622754 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-lsf2j" Jul 21 00:41:28.755693 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.753789 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-mss59"] Jul 21 00:41:28.775200 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.774860 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-tdb28"] Jul 21 00:41:28.801781 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.800886 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-rb4ph"] Jul 21 00:41:28.803510 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:41:28.803447 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ee1dbf0_b4ff_4871_9432_284eb25f97b1.slice/crio-c60980099e219f453f6dec46a6ce8ecefd867b101c9cff4a781f0cb058cd0d88 WatchSource:0}: Error finding container c60980099e219f453f6dec46a6ce8ecefd867b101c9cff4a781f0cb058cd0d88: Status 404 returned error can't find the container with id c60980099e219f453f6dec46a6ce8ecefd867b101c9cff4a781f0cb058cd0d88 Jul 21 00:41:28.814249 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.814147 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-9rqvh"] Jul 21 00:41:28.818680 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:41:28.818651 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod826d0b75_8b78_4bf5_9a54_d71162f8e0da.slice/crio-ba9a779627c5d513fcf135a310ef1acc3005ae452a91e3368922bd539c2e1da0 WatchSource:0}: Error finding container ba9a779627c5d513fcf135a310ef1acc3005ae452a91e3368922bd539c2e1da0: Status 404 returned error can't find the container with id ba9a779627c5d513fcf135a310ef1acc3005ae452a91e3368922bd539c2e1da0 Jul 21 00:41:28.828718 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.828642 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-r8lcd"] Jul 21 00:41:28.831994 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:41:28.831930 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d480ddf_1712_4f08_a5ef_98d3627df334.slice/crio-2e79eeb872942a0494bc42ad21f599be785f9ce6e673f6727426c76c064e525a WatchSource:0}: Error finding container 2e79eeb872942a0494bc42ad21f599be785f9ce6e673f6727426c76c064e525a: Status 404 returned error can't find the container with id 2e79eeb872942a0494bc42ad21f599be785f9ce6e673f6727426c76c064e525a Jul 21 00:41:28.857139 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.857111 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2vh4t"] Jul 21 00:41:28.859583 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.859508 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-lsf2j"] Jul 21 00:41:28.862993 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:41:28.862959 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda625da2a_a501_4fa3_a8c0_21ac34950978.slice/crio-d5831f1026bd71d66ada82ac1b138895e57e1ca2e2df3ceca3b1acb3b747c9e6 WatchSource:0}: Error finding container d5831f1026bd71d66ada82ac1b138895e57e1ca2e2df3ceca3b1acb3b747c9e6: Status 404 returned error can't find the container with id d5831f1026bd71d66ada82ac1b138895e57e1ca2e2df3ceca3b1acb3b747c9e6 Jul 21 00:41:28.864115 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:41:28.864091 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2e92c7d_ed96_4cba_96c5_e1032d141bd4.slice/crio-e825b4da65c4c327d8b6357b3be0ad503984b60f2d9262cbcb155847fa81e32e WatchSource:0}: Error finding container e825b4da65c4c327d8b6357b3be0ad503984b60f2d9262cbcb155847fa81e32e: Status 404 returned error can't find the container with id e825b4da65c4c327d8b6357b3be0ad503984b60f2d9262cbcb155847fa81e32e Jul 21 00:41:28.906519 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.906476 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9r2ff" Jul 21 00:41:28.909644 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.909620 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-nqmsz\"" Jul 21 00:41:28.924569 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.924542 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2ec824c1-400c-4ddb-ad88-bc5a37215aa8-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-twlx9\" (UID: \"2ec824c1-400c-4ddb-ad88-bc5a37215aa8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-twlx9" Jul 21 00:41:28.924709 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:28.924689 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 21 00:41:28.924780 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:28.924769 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ec824c1-400c-4ddb-ad88-bc5a37215aa8-samples-operator-tls podName:2ec824c1-400c-4ddb-ad88-bc5a37215aa8 nodeName:}" failed. No retries permitted until 2026-07-21 00:41:29.924748957 +0000 UTC m=+37.530857986 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2ec824c1-400c-4ddb-ad88-bc5a37215aa8-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-twlx9" (UID: "2ec824c1-400c-4ddb-ad88-bc5a37215aa8") : secret "samples-operator-tls" not found Jul 21 00:41:28.924853 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:28.924836 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8379df88-86a0-4d28-95fc-a102fff737fd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-b8c9w\" (UID: \"8379df88-86a0-4d28-95fc-a102fff737fd\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-b8c9w" Jul 21 00:41:28.924986 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:28.924966 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 21 00:41:28.925073 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:28.925026 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8379df88-86a0-4d28-95fc-a102fff737fd-cluster-monitoring-operator-tls podName:8379df88-86a0-4d28-95fc-a102fff737fd nodeName:}" failed. No retries permitted until 2026-07-21 00:41:29.925012547 +0000 UTC m=+37.531121566 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8379df88-86a0-4d28-95fc-a102fff737fd-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-b8c9w" (UID: "8379df88-86a0-4d28-95fc-a102fff737fd") : secret "cluster-monitoring-operator-tls" not found Jul 21 00:41:29.025688 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:29.025605 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/40960433-ccd7-4aa8-ab03-2cde39f80875-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-sx8h8\" (UID: \"40960433-ccd7-4aa8-ab03-2cde39f80875\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-sx8h8" Jul 21 00:41:29.025688 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:29.025638 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9970c160-3cf7-4d90-b892-1a337f23a065-service-ca-bundle\") pod \"router-default-69fd648678-dm5nn\" (UID: \"9970c160-3cf7-4d90-b892-1a337f23a065\") " pod="openshift-ingress/router-default-69fd648678-dm5nn" Jul 21 00:41:29.025688 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:29.025655 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9970c160-3cf7-4d90-b892-1a337f23a065-metrics-certs\") pod \"router-default-69fd648678-dm5nn\" (UID: \"9970c160-3cf7-4d90-b892-1a337f23a065\") " pod="openshift-ingress/router-default-69fd648678-dm5nn" Jul 21 00:41:29.025688 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:29.025674 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49cc45d2-5087-447a-9123-3885c7bc03a2-metrics-tls\") pod \"dns-default-nlszd\" (UID: \"49cc45d2-5087-447a-9123-3885c7bc03a2\") " pod="openshift-dns/dns-default-nlszd" Jul 21 00:41:29.025930 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:29.025719 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6543206b-bf43-43c5-930d-c3c8834b0f2c-registry-tls\") pod \"image-registry-bdd745b78-7cccc\" (UID: \"6543206b-bf43-43c5-930d-c3c8834b0f2c\") " pod="openshift-image-registry/image-registry-bdd745b78-7cccc" Jul 21 00:41:29.025930 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:29.025739 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 21 00:41:29.025930 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:29.025761 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 21 00:41:29.025930 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:29.025786 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9970c160-3cf7-4d90-b892-1a337f23a065-service-ca-bundle podName:9970c160-3cf7-4d90-b892-1a337f23a065 nodeName:}" failed. No retries permitted until 2026-07-21 00:41:30.025765884 +0000 UTC m=+37.631874903 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9970c160-3cf7-4d90-b892-1a337f23a065-service-ca-bundle") pod "router-default-69fd648678-dm5nn" (UID: "9970c160-3cf7-4d90-b892-1a337f23a065") : configmap references non-existent config key: service-ca.crt Jul 21 00:41:29.025930 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:29.025807 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40960433-ccd7-4aa8-ab03-2cde39f80875-networking-console-plugin-cert podName:40960433-ccd7-4aa8-ab03-2cde39f80875 nodeName:}" failed. No retries permitted until 2026-07-21 00:41:30.025800047 +0000 UTC m=+37.631909065 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/40960433-ccd7-4aa8-ab03-2cde39f80875-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-sx8h8" (UID: "40960433-ccd7-4aa8-ab03-2cde39f80875") : secret "networking-console-plugin-cert" not found Jul 21 00:41:29.025930 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:29.025817 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 00:41:29.025930 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:29.025837 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 21 00:41:29.025930 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:29.025849 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bdd745b78-7cccc: secret "image-registry-tls" not found Jul 21 00:41:29.025930 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:29.025860 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9970c160-3cf7-4d90-b892-1a337f23a065-metrics-certs podName:9970c160-3cf7-4d90-b892-1a337f23a065 nodeName:}" failed. No retries permitted until 2026-07-21 00:41:30.025811547 +0000 UTC m=+37.631920565 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9970c160-3cf7-4d90-b892-1a337f23a065-metrics-certs") pod "router-default-69fd648678-dm5nn" (UID: "9970c160-3cf7-4d90-b892-1a337f23a065") : secret "router-metrics-certs-default" not found Jul 21 00:41:29.025930 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:29.025913 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49cc45d2-5087-447a-9123-3885c7bc03a2-metrics-tls podName:49cc45d2-5087-447a-9123-3885c7bc03a2 nodeName:}" failed. No retries permitted until 2026-07-21 00:41:30.02587903 +0000 UTC m=+37.631988049 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/49cc45d2-5087-447a-9123-3885c7bc03a2-metrics-tls") pod "dns-default-nlszd" (UID: "49cc45d2-5087-447a-9123-3885c7bc03a2") : secret "dns-default-metrics-tls" not found Jul 21 00:41:29.025930 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:29.025927 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6543206b-bf43-43c5-930d-c3c8834b0f2c-registry-tls podName:6543206b-bf43-43c5-930d-c3c8834b0f2c nodeName:}" failed. No retries permitted until 2026-07-21 00:41:30.025921341 +0000 UTC m=+37.632030359 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6543206b-bf43-43c5-930d-c3c8834b0f2c-registry-tls") pod "image-registry-bdd745b78-7cccc" (UID: "6543206b-bf43-43c5-930d-c3c8834b0f2c") : secret "image-registry-tls" not found Jul 21 00:41:29.078939 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:29.078902 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-9rqvh" event={"ID":"826d0b75-8b78-4bf5-9a54-d71162f8e0da","Type":"ContainerStarted","Data":"ba9a779627c5d513fcf135a310ef1acc3005ae452a91e3368922bd539c2e1da0"} Jul 21 00:41:29.080158 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:29.080131 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-tdb28" event={"ID":"f7fa2eb6-5645-47f1-8199-1dd64ea512dc","Type":"ContainerStarted","Data":"1829df53b346fe18be756368f448a37ab195c6cf9bb8f9be7ee7a500208b9f00"} Jul 21 00:41:29.081384 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:29.081360 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2vh4t" event={"ID":"a625da2a-a501-4fa3-a8c0-21ac34950978","Type":"ContainerStarted","Data":"d5831f1026bd71d66ada82ac1b138895e57e1ca2e2df3ceca3b1acb3b747c9e6"} Jul 21 00:41:29.082549 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:29.082520 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-rb4ph" event={"ID":"3ee1dbf0-b4ff-4871-9432-284eb25f97b1","Type":"ContainerStarted","Data":"c60980099e219f453f6dec46a6ce8ecefd867b101c9cff4a781f0cb058cd0d88"} Jul 21 00:41:29.083695 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:29.083673 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-lsf2j" event={"ID":"c2e92c7d-ed96-4cba-96c5-e1032d141bd4","Type":"ContainerStarted","Data":"e825b4da65c4c327d8b6357b3be0ad503984b60f2d9262cbcb155847fa81e32e"} Jul 21 00:41:29.084758 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:29.084731 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-r8lcd" event={"ID":"2d480ddf-1712-4f08-a5ef-98d3627df334","Type":"ContainerStarted","Data":"2e79eeb872942a0494bc42ad21f599be785f9ce6e673f6727426c76c064e525a"} Jul 21 00:41:29.085939 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:29.085919 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-mss59" event={"ID":"f43e318f-e1aa-4f27-971b-615e797691d9","Type":"ContainerStarted","Data":"beaf8124bb02bafab617bdc10d3e4d9ab8e6e32ad365fff8ed47b9a61e33c815"} Jul 21 00:41:29.907636 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:29.906570 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzcmw" Jul 21 00:41:29.907636 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:29.907009 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4nq5b" Jul 21 00:41:29.910661 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:29.909986 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wcfmk\"" Jul 21 00:41:29.910661 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:29.910228 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 21 00:41:29.910661 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:29.910456 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 21 00:41:29.935425 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:29.935392 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8379df88-86a0-4d28-95fc-a102fff737fd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-b8c9w\" (UID: \"8379df88-86a0-4d28-95fc-a102fff737fd\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-b8c9w" Jul 21 00:41:29.935538 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:29.935473 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2ec824c1-400c-4ddb-ad88-bc5a37215aa8-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-twlx9\" (UID: \"2ec824c1-400c-4ddb-ad88-bc5a37215aa8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-twlx9" Jul 21 00:41:29.935642 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:29.935625 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 21 00:41:29.935714 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:29.935685 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ec824c1-400c-4ddb-ad88-bc5a37215aa8-samples-operator-tls podName:2ec824c1-400c-4ddb-ad88-bc5a37215aa8 nodeName:}" failed. No retries permitted until 2026-07-21 00:41:31.935668885 +0000 UTC m=+39.541777920 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2ec824c1-400c-4ddb-ad88-bc5a37215aa8-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-twlx9" (UID: "2ec824c1-400c-4ddb-ad88-bc5a37215aa8") : secret "samples-operator-tls" not found Jul 21 00:41:29.935782 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:29.935750 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 21 00:41:29.935832 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:29.935782 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8379df88-86a0-4d28-95fc-a102fff737fd-cluster-monitoring-operator-tls podName:8379df88-86a0-4d28-95fc-a102fff737fd nodeName:}" failed. No retries permitted until 2026-07-21 00:41:31.935772091 +0000 UTC m=+39.541881114 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8379df88-86a0-4d28-95fc-a102fff737fd-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-b8c9w" (UID: "8379df88-86a0-4d28-95fc-a102fff737fd") : secret "cluster-monitoring-operator-tls" not found Jul 21 00:41:30.036615 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:30.036582 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6543206b-bf43-43c5-930d-c3c8834b0f2c-registry-tls\") pod \"image-registry-bdd745b78-7cccc\" (UID: \"6543206b-bf43-43c5-930d-c3c8834b0f2c\") " pod="openshift-image-registry/image-registry-bdd745b78-7cccc" Jul 21 00:41:30.036794 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:30.036728 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/40960433-ccd7-4aa8-ab03-2cde39f80875-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-sx8h8\" (UID: \"40960433-ccd7-4aa8-ab03-2cde39f80875\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-sx8h8" Jul 21 00:41:30.036794 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:30.036762 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9970c160-3cf7-4d90-b892-1a337f23a065-service-ca-bundle\") pod \"router-default-69fd648678-dm5nn\" (UID: \"9970c160-3cf7-4d90-b892-1a337f23a065\") " pod="openshift-ingress/router-default-69fd648678-dm5nn" Jul 21 00:41:30.036794 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:30.036788 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9970c160-3cf7-4d90-b892-1a337f23a065-metrics-certs\") pod \"router-default-69fd648678-dm5nn\" (UID: \"9970c160-3cf7-4d90-b892-1a337f23a065\") " pod="openshift-ingress/router-default-69fd648678-dm5nn" Jul 21 00:41:30.037017 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:30.036818 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49cc45d2-5087-447a-9123-3885c7bc03a2-metrics-tls\") pod \"dns-default-nlszd\" (UID: \"49cc45d2-5087-447a-9123-3885c7bc03a2\") " pod="openshift-dns/dns-default-nlszd" Jul 21 00:41:30.037813 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:30.037348 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9970c160-3cf7-4d90-b892-1a337f23a065-service-ca-bundle podName:9970c160-3cf7-4d90-b892-1a337f23a065 nodeName:}" failed. No retries permitted until 2026-07-21 00:41:32.037325854 +0000 UTC m=+39.643434890 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9970c160-3cf7-4d90-b892-1a337f23a065-service-ca-bundle") pod "router-default-69fd648678-dm5nn" (UID: "9970c160-3cf7-4d90-b892-1a337f23a065") : configmap references non-existent config key: service-ca.crt Jul 21 00:41:30.037813 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:30.037468 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 21 00:41:30.037813 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:30.037530 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9970c160-3cf7-4d90-b892-1a337f23a065-metrics-certs podName:9970c160-3cf7-4d90-b892-1a337f23a065 nodeName:}" failed. No retries permitted until 2026-07-21 00:41:32.037513042 +0000 UTC m=+39.643622078 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9970c160-3cf7-4d90-b892-1a337f23a065-metrics-certs") pod "router-default-69fd648678-dm5nn" (UID: "9970c160-3cf7-4d90-b892-1a337f23a065") : secret "router-metrics-certs-default" not found Jul 21 00:41:30.037813 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:30.037601 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 00:41:30.037813 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:30.037635 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49cc45d2-5087-447a-9123-3885c7bc03a2-metrics-tls podName:49cc45d2-5087-447a-9123-3885c7bc03a2 nodeName:}" failed. No retries permitted until 2026-07-21 00:41:32.037624529 +0000 UTC m=+39.643733560 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/49cc45d2-5087-447a-9123-3885c7bc03a2-metrics-tls") pod "dns-default-nlszd" (UID: "49cc45d2-5087-447a-9123-3885c7bc03a2") : secret "dns-default-metrics-tls" not found Jul 21 00:41:30.037813 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:30.037691 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 21 00:41:30.037813 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:30.037738 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40960433-ccd7-4aa8-ab03-2cde39f80875-networking-console-plugin-cert podName:40960433-ccd7-4aa8-ab03-2cde39f80875 nodeName:}" failed. No retries permitted until 2026-07-21 00:41:32.037727306 +0000 UTC m=+39.643836338 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/40960433-ccd7-4aa8-ab03-2cde39f80875-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-sx8h8" (UID: "40960433-ccd7-4aa8-ab03-2cde39f80875") : secret "networking-console-plugin-cert" not found Jul 21 00:41:30.038275 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:30.037847 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 21 00:41:30.038275 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:30.037863 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bdd745b78-7cccc: secret "image-registry-tls" not found Jul 21 00:41:30.038275 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:30.037911 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6543206b-bf43-43c5-930d-c3c8834b0f2c-registry-tls podName:6543206b-bf43-43c5-930d-c3c8834b0f2c nodeName:}" failed. No retries permitted until 2026-07-21 00:41:32.037895762 +0000 UTC m=+39.644004785 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6543206b-bf43-43c5-930d-c3c8834b0f2c-registry-tls") pod "image-registry-bdd745b78-7cccc" (UID: "6543206b-bf43-43c5-930d-c3c8834b0f2c") : secret "image-registry-tls" not found Jul 21 00:41:31.956603 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:31.956554 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8379df88-86a0-4d28-95fc-a102fff737fd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-b8c9w\" (UID: \"8379df88-86a0-4d28-95fc-a102fff737fd\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-b8c9w" Jul 21 00:41:31.957075 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:31.956652 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2ec824c1-400c-4ddb-ad88-bc5a37215aa8-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-twlx9\" (UID: \"2ec824c1-400c-4ddb-ad88-bc5a37215aa8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-twlx9" Jul 21 00:41:31.957075 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:31.956830 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 21 00:41:31.957075 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:31.956898 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ec824c1-400c-4ddb-ad88-bc5a37215aa8-samples-operator-tls podName:2ec824c1-400c-4ddb-ad88-bc5a37215aa8 nodeName:}" failed. No retries permitted until 2026-07-21 00:41:35.956878119 +0000 UTC m=+43.562987158 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2ec824c1-400c-4ddb-ad88-bc5a37215aa8-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-twlx9" (UID: "2ec824c1-400c-4ddb-ad88-bc5a37215aa8") : secret "samples-operator-tls" not found Jul 21 00:41:31.957075 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:31.956971 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 21 00:41:31.957075 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:31.957010 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8379df88-86a0-4d28-95fc-a102fff737fd-cluster-monitoring-operator-tls podName:8379df88-86a0-4d28-95fc-a102fff737fd nodeName:}" failed. No retries permitted until 2026-07-21 00:41:35.956998216 +0000 UTC m=+43.563107235 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8379df88-86a0-4d28-95fc-a102fff737fd-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-b8c9w" (UID: "8379df88-86a0-4d28-95fc-a102fff737fd") : secret "cluster-monitoring-operator-tls" not found Jul 21 00:41:32.057572 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:32.057540 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/40960433-ccd7-4aa8-ab03-2cde39f80875-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-sx8h8\" (UID: \"40960433-ccd7-4aa8-ab03-2cde39f80875\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-sx8h8" Jul 21 00:41:32.057745 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:32.057582 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9970c160-3cf7-4d90-b892-1a337f23a065-service-ca-bundle\") pod \"router-default-69fd648678-dm5nn\" (UID: \"9970c160-3cf7-4d90-b892-1a337f23a065\") " pod="openshift-ingress/router-default-69fd648678-dm5nn" Jul 21 00:41:32.057745 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:32.057602 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9970c160-3cf7-4d90-b892-1a337f23a065-metrics-certs\") pod \"router-default-69fd648678-dm5nn\" (UID: \"9970c160-3cf7-4d90-b892-1a337f23a065\") " pod="openshift-ingress/router-default-69fd648678-dm5nn" Jul 21 00:41:32.057745 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:32.057621 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49cc45d2-5087-447a-9123-3885c7bc03a2-metrics-tls\") pod \"dns-default-nlszd\" (UID: \"49cc45d2-5087-447a-9123-3885c7bc03a2\") " pod="openshift-dns/dns-default-nlszd" Jul 21 00:41:32.057745 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:32.057664 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6543206b-bf43-43c5-930d-c3c8834b0f2c-registry-tls\") pod \"image-registry-bdd745b78-7cccc\" (UID: \"6543206b-bf43-43c5-930d-c3c8834b0f2c\") " pod="openshift-image-registry/image-registry-bdd745b78-7cccc" Jul 21 00:41:32.057745 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:32.057714 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 21 00:41:32.057745 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:32.057734 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9970c160-3cf7-4d90-b892-1a337f23a065-service-ca-bundle podName:9970c160-3cf7-4d90-b892-1a337f23a065 nodeName:}" failed. No retries permitted until 2026-07-21 00:41:36.057712015 +0000 UTC m=+43.663821050 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9970c160-3cf7-4d90-b892-1a337f23a065-service-ca-bundle") pod "router-default-69fd648678-dm5nn" (UID: "9970c160-3cf7-4d90-b892-1a337f23a065") : configmap references non-existent config key: service-ca.crt Jul 21 00:41:32.058034 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:32.057735 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 21 00:41:32.058034 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:32.057776 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40960433-ccd7-4aa8-ab03-2cde39f80875-networking-console-plugin-cert podName:40960433-ccd7-4aa8-ab03-2cde39f80875 nodeName:}" failed. No retries permitted until 2026-07-21 00:41:36.057759328 +0000 UTC m=+43.663868355 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/40960433-ccd7-4aa8-ab03-2cde39f80875-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-sx8h8" (UID: "40960433-ccd7-4aa8-ab03-2cde39f80875") : secret "networking-console-plugin-cert" not found Jul 21 00:41:32.058034 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:32.057781 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 00:41:32.058034 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:32.057801 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9970c160-3cf7-4d90-b892-1a337f23a065-metrics-certs podName:9970c160-3cf7-4d90-b892-1a337f23a065 nodeName:}" failed. No retries permitted until 2026-07-21 00:41:36.057788964 +0000 UTC m=+43.663898006 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9970c160-3cf7-4d90-b892-1a337f23a065-metrics-certs") pod "router-default-69fd648678-dm5nn" (UID: "9970c160-3cf7-4d90-b892-1a337f23a065") : secret "router-metrics-certs-default" not found Jul 21 00:41:32.058034 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:32.057841 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49cc45d2-5087-447a-9123-3885c7bc03a2-metrics-tls podName:49cc45d2-5087-447a-9123-3885c7bc03a2 nodeName:}" failed. No retries permitted until 2026-07-21 00:41:36.057829323 +0000 UTC m=+43.663938344 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/49cc45d2-5087-447a-9123-3885c7bc03a2-metrics-tls") pod "dns-default-nlszd" (UID: "49cc45d2-5087-447a-9123-3885c7bc03a2") : secret "dns-default-metrics-tls" not found Jul 21 00:41:32.058034 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:32.057783 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 21 00:41:32.058034 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:32.057864 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bdd745b78-7cccc: secret "image-registry-tls" not found Jul 21 00:41:32.058034 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:32.057904 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6543206b-bf43-43c5-930d-c3c8834b0f2c-registry-tls podName:6543206b-bf43-43c5-930d-c3c8834b0f2c nodeName:}" failed. No retries permitted until 2026-07-21 00:41:36.057893062 +0000 UTC m=+43.664002082 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6543206b-bf43-43c5-930d-c3c8834b0f2c-registry-tls") pod "image-registry-bdd745b78-7cccc" (UID: "6543206b-bf43-43c5-930d-c3c8834b0f2c") : secret "image-registry-tls" not found Jul 21 00:41:35.990542 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:35.990498 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8379df88-86a0-4d28-95fc-a102fff737fd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-b8c9w\" (UID: \"8379df88-86a0-4d28-95fc-a102fff737fd\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-b8c9w" Jul 21 00:41:35.991045 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:35.990594 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2ec824c1-400c-4ddb-ad88-bc5a37215aa8-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-twlx9\" (UID: \"2ec824c1-400c-4ddb-ad88-bc5a37215aa8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-twlx9" Jul 21 00:41:35.991045 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:35.990661 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 21 00:41:35.991045 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:35.990736 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8379df88-86a0-4d28-95fc-a102fff737fd-cluster-monitoring-operator-tls podName:8379df88-86a0-4d28-95fc-a102fff737fd nodeName:}" failed. No retries permitted until 2026-07-21 00:41:43.990715252 +0000 UTC m=+51.596824270 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8379df88-86a0-4d28-95fc-a102fff737fd-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-b8c9w" (UID: "8379df88-86a0-4d28-95fc-a102fff737fd") : secret "cluster-monitoring-operator-tls" not found Jul 21 00:41:35.991045 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:35.990745 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 21 00:41:35.991045 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:35.990809 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ec824c1-400c-4ddb-ad88-bc5a37215aa8-samples-operator-tls podName:2ec824c1-400c-4ddb-ad88-bc5a37215aa8 nodeName:}" failed. No retries permitted until 2026-07-21 00:41:43.990792933 +0000 UTC m=+51.596901955 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2ec824c1-400c-4ddb-ad88-bc5a37215aa8-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-twlx9" (UID: "2ec824c1-400c-4ddb-ad88-bc5a37215aa8") : secret "samples-operator-tls" not found Jul 21 00:41:36.091521 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:36.091463 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49cc45d2-5087-447a-9123-3885c7bc03a2-metrics-tls\") pod \"dns-default-nlszd\" (UID: \"49cc45d2-5087-447a-9123-3885c7bc03a2\") " pod="openshift-dns/dns-default-nlszd" Jul 21 00:41:36.091666 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:36.091542 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6543206b-bf43-43c5-930d-c3c8834b0f2c-registry-tls\") pod \"image-registry-bdd745b78-7cccc\" (UID: \"6543206b-bf43-43c5-930d-c3c8834b0f2c\") " pod="openshift-image-registry/image-registry-bdd745b78-7cccc" Jul 21 00:41:36.091666 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:36.091590 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 00:41:36.091666 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:36.091656 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49cc45d2-5087-447a-9123-3885c7bc03a2-metrics-tls podName:49cc45d2-5087-447a-9123-3885c7bc03a2 nodeName:}" failed. No retries permitted until 2026-07-21 00:41:44.091637572 +0000 UTC m=+51.697746596 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/49cc45d2-5087-447a-9123-3885c7bc03a2-metrics-tls") pod "dns-default-nlszd" (UID: "49cc45d2-5087-447a-9123-3885c7bc03a2") : secret "dns-default-metrics-tls" not found Jul 21 00:41:36.091821 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:36.091660 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 21 00:41:36.091821 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:36.091681 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bdd745b78-7cccc: secret "image-registry-tls" not found Jul 21 00:41:36.091821 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:36.091700 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/40960433-ccd7-4aa8-ab03-2cde39f80875-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-sx8h8\" (UID: \"40960433-ccd7-4aa8-ab03-2cde39f80875\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-sx8h8" Jul 21 00:41:36.091821 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:36.091735 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6543206b-bf43-43c5-930d-c3c8834b0f2c-registry-tls podName:6543206b-bf43-43c5-930d-c3c8834b0f2c nodeName:}" failed. No retries permitted until 2026-07-21 00:41:44.091718292 +0000 UTC m=+51.697827315 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6543206b-bf43-43c5-930d-c3c8834b0f2c-registry-tls") pod "image-registry-bdd745b78-7cccc" (UID: "6543206b-bf43-43c5-930d-c3c8834b0f2c") : secret "image-registry-tls" not found Jul 21 00:41:36.091821 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:36.091775 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 21 00:41:36.091821 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:36.091778 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9970c160-3cf7-4d90-b892-1a337f23a065-service-ca-bundle\") pod \"router-default-69fd648678-dm5nn\" (UID: \"9970c160-3cf7-4d90-b892-1a337f23a065\") " pod="openshift-ingress/router-default-69fd648678-dm5nn" Jul 21 00:41:36.091821 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:36.091815 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9970c160-3cf7-4d90-b892-1a337f23a065-metrics-certs\") pod \"router-default-69fd648678-dm5nn\" (UID: \"9970c160-3cf7-4d90-b892-1a337f23a065\") " pod="openshift-ingress/router-default-69fd648678-dm5nn" Jul 21 00:41:36.092101 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:36.091868 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40960433-ccd7-4aa8-ab03-2cde39f80875-networking-console-plugin-cert podName:40960433-ccd7-4aa8-ab03-2cde39f80875 nodeName:}" failed. No retries permitted until 2026-07-21 00:41:44.091855816 +0000 UTC m=+51.697964837 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/40960433-ccd7-4aa8-ab03-2cde39f80875-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-sx8h8" (UID: "40960433-ccd7-4aa8-ab03-2cde39f80875") : secret "networking-console-plugin-cert" not found Jul 21 00:41:36.092101 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:36.091883 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9970c160-3cf7-4d90-b892-1a337f23a065-service-ca-bundle podName:9970c160-3cf7-4d90-b892-1a337f23a065 nodeName:}" failed. No retries permitted until 2026-07-21 00:41:44.091875025 +0000 UTC m=+51.697984051 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9970c160-3cf7-4d90-b892-1a337f23a065-service-ca-bundle") pod "router-default-69fd648678-dm5nn" (UID: "9970c160-3cf7-4d90-b892-1a337f23a065") : configmap references non-existent config key: service-ca.crt Jul 21 00:41:36.092101 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:36.091931 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 21 00:41:36.092101 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:36.091997 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9970c160-3cf7-4d90-b892-1a337f23a065-metrics-certs podName:9970c160-3cf7-4d90-b892-1a337f23a065 nodeName:}" failed. No retries permitted until 2026-07-21 00:41:44.091981583 +0000 UTC m=+51.698090613 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9970c160-3cf7-4d90-b892-1a337f23a065-metrics-certs") pod "router-default-69fd648678-dm5nn" (UID: "9970c160-3cf7-4d90-b892-1a337f23a065") : secret "router-metrics-certs-default" not found Jul 21 00:41:39.115045 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:39.115005 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-9rqvh" event={"ID":"826d0b75-8b78-4bf5-9a54-d71162f8e0da","Type":"ContainerStarted","Data":"37a0a59066a0159cd3fcbe80b860f141d42c2910670b641c2e18fda804c2c6f9"} Jul 21 00:41:39.117448 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:39.117072 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-tdb28" event={"ID":"f7fa2eb6-5645-47f1-8199-1dd64ea512dc","Type":"ContainerStarted","Data":"81eb9dcac9118a97418b3bd9352bdc026dd055ef18e01faebf499006acb0804d"} Jul 21 00:41:39.119073 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:39.119038 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2vh4t" event={"ID":"a625da2a-a501-4fa3-a8c0-21ac34950978","Type":"ContainerStarted","Data":"3fbb45522491b05c7de463285af48820667a6958cf009256f47b30e0002f0f21"} Jul 21 00:41:39.122058 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:39.122022 2580 generic.go:356] "Generic (PLEG): container finished" podID="f3e26871-c996-46c1-adfb-c96c1869814b" containerID="3c0a4a89bc141cd0223f740aba9d8404b3389892ed79da43783ef9cdec9c6f28" exitCode=0 Jul 21 00:41:39.122058 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:39.122049 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6zg78" event={"ID":"f3e26871-c996-46c1-adfb-c96c1869814b","Type":"ContainerDied","Data":"3c0a4a89bc141cd0223f740aba9d8404b3389892ed79da43783ef9cdec9c6f28"} Jul 21 00:41:39.123744 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:39.123726 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-rb4ph_3ee1dbf0-b4ff-4871-9432-284eb25f97b1/console-operator/0.log" Jul 21 00:41:39.123852 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:39.123757 2580 generic.go:356] "Generic (PLEG): container finished" podID="3ee1dbf0-b4ff-4871-9432-284eb25f97b1" containerID="d29c4bdbe37872ef494cdd682c8c31b1631e30d2c56656721d16a3895b724059" exitCode=255 Jul 21 00:41:39.123981 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:39.123944 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-rb4ph" event={"ID":"3ee1dbf0-b4ff-4871-9432-284eb25f97b1","Type":"ContainerDied","Data":"d29c4bdbe37872ef494cdd682c8c31b1631e30d2c56656721d16a3895b724059"} Jul 21 00:41:39.124126 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:39.124109 2580 scope.go:117] "RemoveContainer" containerID="d29c4bdbe37872ef494cdd682c8c31b1631e30d2c56656721d16a3895b724059" Jul 21 00:41:39.125776 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:39.125732 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-lsf2j" event={"ID":"c2e92c7d-ed96-4cba-96c5-e1032d141bd4","Type":"ContainerStarted","Data":"226fd857bbcddfa83d664192aaebfbfe2487e107ab7c3c45fff04103f5031783"} Jul 21 00:41:39.128052 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:39.128015 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-r8lcd" event={"ID":"2d480ddf-1712-4f08-a5ef-98d3627df334","Type":"ContainerStarted","Data":"d39b695ff8792cf12c06a37fd2f9cd45958c679e53963333cb8d472d277c80c7"} Jul 21 00:41:39.129528 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:39.129509 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-mss59" event={"ID":"f43e318f-e1aa-4f27-971b-615e797691d9","Type":"ContainerStarted","Data":"c55af92c958f43b8089d4e7d1ed37e93691e0efd37e4c5f813c2391d30091413"} Jul 21 00:41:39.131827 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:39.131783 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-555d9d7885-9rqvh" podStartSLOduration=24.834812597 podStartE2EDuration="34.131766998s" totalImagesPullingTime="9.296954401s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:41:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:41:39.131275198 +0000 UTC m=+46.737384236" watchObservedRunningTime="2026-07-21 00:41:39.131766998 +0000 UTC m=+46.737876021" Jul 21 00:41:39.145741 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:39.145696 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-2vh4t" podStartSLOduration=1.887301624 podStartE2EDuration="11.145680142s" totalImagesPullingTime="9.258378518s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:41:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:41:39.145483052 +0000 UTC m=+46.751592094" watchObservedRunningTime="2026-07-21 00:41:39.145680142 +0000 UTC m=+46.751789186" Jul 21 00:41:39.162170 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:39.160755 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-lsf2j" podStartSLOduration=24.635311176 podStartE2EDuration="34.16073834s" totalImagesPullingTime="9.525427164s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:41:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:41:39.159830527 +0000 UTC m=+46.765939569" watchObservedRunningTime="2026-07-21 00:41:39.16073834 +0000 UTC m=+46.766847382" Jul 21 00:41:39.178779 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:39.178736 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-tdb28" podStartSLOduration=25.843851002 podStartE2EDuration="35.178725926s" totalImagesPullingTime="9.334874924s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:41:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:41:39.178143462 +0000 UTC m=+46.784252484" watchObservedRunningTime="2026-07-21 00:41:39.178725926 +0000 UTC m=+46.784834948" Jul 21 00:41:39.238261 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:39.238198 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-r8lcd" podStartSLOduration=24.954231556 podStartE2EDuration="34.236988028s" totalImagesPullingTime="9.282756472s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:41:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:41:39.23604809 +0000 UTC m=+46.842157130" watchObservedRunningTime="2026-07-21 00:41:39.236988028 +0000 UTC m=+46.843097069" Jul 21 00:41:39.258272 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:39.257648 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-mss59" podStartSLOduration=25.997762231 podStartE2EDuration="35.257627736s" totalImagesPullingTime="9.259865505s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:41:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:41:39.25483973 +0000 UTC m=+46.860948770" watchObservedRunningTime="2026-07-21 00:41:39.257627736 +0000 UTC m=+46.863736777" Jul 21 00:41:40.136587 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:40.136498 2580 generic.go:356] "Generic (PLEG): container finished" podID="f3e26871-c996-46c1-adfb-c96c1869814b" containerID="e2b4108de5b641ad635e6fb948ee7428ccca3924a54a4d3c199ec8ec7e2d70d9" exitCode=0 Jul 21 00:41:40.137004 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:40.136590 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6zg78" event={"ID":"f3e26871-c996-46c1-adfb-c96c1869814b","Type":"ContainerDied","Data":"e2b4108de5b641ad635e6fb948ee7428ccca3924a54a4d3c199ec8ec7e2d70d9"} Jul 21 00:41:40.138268 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:40.138249 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-rb4ph_3ee1dbf0-b4ff-4871-9432-284eb25f97b1/console-operator/1.log" Jul 21 00:41:40.138648 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:40.138629 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-rb4ph_3ee1dbf0-b4ff-4871-9432-284eb25f97b1/console-operator/0.log" Jul 21 00:41:40.138783 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:40.138663 2580 generic.go:356] "Generic (PLEG): container finished" podID="3ee1dbf0-b4ff-4871-9432-284eb25f97b1" containerID="53298a4b69c8aa9fd6a6c7dd921a4923031bbf1503ff4ea1a994c1b004a08a3d" exitCode=255 Jul 21 00:41:40.138783 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:40.138732 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-rb4ph" event={"ID":"3ee1dbf0-b4ff-4871-9432-284eb25f97b1","Type":"ContainerDied","Data":"53298a4b69c8aa9fd6a6c7dd921a4923031bbf1503ff4ea1a994c1b004a08a3d"} Jul 21 00:41:40.139050 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:40.139034 2580 scope.go:117] "RemoveContainer" containerID="53298a4b69c8aa9fd6a6c7dd921a4923031bbf1503ff4ea1a994c1b004a08a3d" Jul 21 00:41:40.139512 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:40.139179 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-rb4ph_openshift-console-operator(3ee1dbf0-b4ff-4871-9432-284eb25f97b1)\"" pod="openshift-console-operator/console-operator-565b7b569d-rb4ph" podUID="3ee1dbf0-b4ff-4871-9432-284eb25f97b1" Jul 21 00:41:40.148713 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:40.148690 2580 scope.go:117] "RemoveContainer" containerID="d29c4bdbe37872ef494cdd682c8c31b1631e30d2c56656721d16a3895b724059" Jul 21 00:41:40.328337 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:40.328296 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-5xx6h"] Jul 21 00:41:40.348890 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:40.348856 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5xx6h"] Jul 21 00:41:40.349030 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:40.348993 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5xx6h" Jul 21 00:41:40.352082 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:40.352056 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 21 00:41:40.352477 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:40.352455 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 21 00:41:40.352580 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:40.352556 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fhmkx\"" Jul 21 00:41:40.535215 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:40.535122 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2d8da6d4-7c9a-42d9-ac48-581a8640f880-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5xx6h\" (UID: \"2d8da6d4-7c9a-42d9-ac48-581a8640f880\") " pod="openshift-insights/insights-runtime-extractor-5xx6h" Jul 21 00:41:40.535571 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:40.535554 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2d8da6d4-7c9a-42d9-ac48-581a8640f880-data-volume\") pod \"insights-runtime-extractor-5xx6h\" (UID: \"2d8da6d4-7c9a-42d9-ac48-581a8640f880\") " pod="openshift-insights/insights-runtime-extractor-5xx6h" Jul 21 00:41:40.535639 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:40.535606 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2d8da6d4-7c9a-42d9-ac48-581a8640f880-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5xx6h\" (UID: \"2d8da6d4-7c9a-42d9-ac48-581a8640f880\") " pod="openshift-insights/insights-runtime-extractor-5xx6h" Jul 21 00:41:40.535697 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:40.535642 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsscq\" (UniqueName: \"kubernetes.io/projected/2d8da6d4-7c9a-42d9-ac48-581a8640f880-kube-api-access-nsscq\") pod \"insights-runtime-extractor-5xx6h\" (UID: \"2d8da6d4-7c9a-42d9-ac48-581a8640f880\") " pod="openshift-insights/insights-runtime-extractor-5xx6h" Jul 21 00:41:40.535697 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:40.535671 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2d8da6d4-7c9a-42d9-ac48-581a8640f880-crio-socket\") pod \"insights-runtime-extractor-5xx6h\" (UID: \"2d8da6d4-7c9a-42d9-ac48-581a8640f880\") " pod="openshift-insights/insights-runtime-extractor-5xx6h" Jul 21 00:41:40.636722 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:40.636681 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2d8da6d4-7c9a-42d9-ac48-581a8640f880-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5xx6h\" (UID: \"2d8da6d4-7c9a-42d9-ac48-581a8640f880\") " pod="openshift-insights/insights-runtime-extractor-5xx6h" Jul 21 00:41:40.636909 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:40.636744 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2d8da6d4-7c9a-42d9-ac48-581a8640f880-data-volume\") pod \"insights-runtime-extractor-5xx6h\" (UID: \"2d8da6d4-7c9a-42d9-ac48-581a8640f880\") " pod="openshift-insights/insights-runtime-extractor-5xx6h" Jul 21 00:41:40.636909 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:40.636775 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2d8da6d4-7c9a-42d9-ac48-581a8640f880-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5xx6h\" (UID: \"2d8da6d4-7c9a-42d9-ac48-581a8640f880\") " pod="openshift-insights/insights-runtime-extractor-5xx6h" Jul 21 00:41:40.636909 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:40.636807 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsscq\" (UniqueName: \"kubernetes.io/projected/2d8da6d4-7c9a-42d9-ac48-581a8640f880-kube-api-access-nsscq\") pod \"insights-runtime-extractor-5xx6h\" (UID: \"2d8da6d4-7c9a-42d9-ac48-581a8640f880\") " pod="openshift-insights/insights-runtime-extractor-5xx6h" Jul 21 00:41:40.636909 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:40.636824 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 21 00:41:40.636909 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:40.636906 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2d8da6d4-7c9a-42d9-ac48-581a8640f880-kube-rbac-proxy-cm podName:2d8da6d4-7c9a-42d9-ac48-581a8640f880 nodeName:}" failed. No retries permitted until 2026-07-21 00:41:41.136884986 +0000 UTC m=+48.742994007 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2d8da6d4-7c9a-42d9-ac48-581a8640f880-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5xx6h" (UID: "2d8da6d4-7c9a-42d9-ac48-581a8640f880") : configmap "kube-rbac-proxy" not found Jul 21 00:41:40.637128 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:40.636930 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 21 00:41:40.637128 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:40.636974 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2d8da6d4-7c9a-42d9-ac48-581a8640f880-crio-socket\") pod \"insights-runtime-extractor-5xx6h\" (UID: \"2d8da6d4-7c9a-42d9-ac48-581a8640f880\") " pod="openshift-insights/insights-runtime-extractor-5xx6h" Jul 21 00:41:40.637128 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:40.636992 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d8da6d4-7c9a-42d9-ac48-581a8640f880-insights-runtime-extractor-tls podName:2d8da6d4-7c9a-42d9-ac48-581a8640f880 nodeName:}" failed. No retries permitted until 2026-07-21 00:41:41.136973392 +0000 UTC m=+48.743082433 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2d8da6d4-7c9a-42d9-ac48-581a8640f880-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5xx6h" (UID: "2d8da6d4-7c9a-42d9-ac48-581a8640f880") : secret "insights-runtime-extractor-tls" not found Jul 21 00:41:40.637128 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:40.636828 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2d8da6d4-7c9a-42d9-ac48-581a8640f880-crio-socket\") pod \"insights-runtime-extractor-5xx6h\" (UID: \"2d8da6d4-7c9a-42d9-ac48-581a8640f880\") " pod="openshift-insights/insights-runtime-extractor-5xx6h" Jul 21 00:41:40.637285 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:40.637118 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2d8da6d4-7c9a-42d9-ac48-581a8640f880-data-volume\") pod \"insights-runtime-extractor-5xx6h\" (UID: \"2d8da6d4-7c9a-42d9-ac48-581a8640f880\") " pod="openshift-insights/insights-runtime-extractor-5xx6h" Jul 21 00:41:40.649248 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:40.649201 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsscq\" (UniqueName: \"kubernetes.io/projected/2d8da6d4-7c9a-42d9-ac48-581a8640f880-kube-api-access-nsscq\") pod \"insights-runtime-extractor-5xx6h\" (UID: \"2d8da6d4-7c9a-42d9-ac48-581a8640f880\") " pod="openshift-insights/insights-runtime-extractor-5xx6h" Jul 21 00:41:41.139984 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:41.139951 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2d8da6d4-7c9a-42d9-ac48-581a8640f880-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5xx6h\" (UID: \"2d8da6d4-7c9a-42d9-ac48-581a8640f880\") " pod="openshift-insights/insights-runtime-extractor-5xx6h" Jul 21 00:41:41.140430 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:41.140062 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2d8da6d4-7c9a-42d9-ac48-581a8640f880-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5xx6h\" (UID: \"2d8da6d4-7c9a-42d9-ac48-581a8640f880\") " pod="openshift-insights/insights-runtime-extractor-5xx6h" Jul 21 00:41:41.140430 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:41.140081 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 21 00:41:41.140430 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:41.140153 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2d8da6d4-7c9a-42d9-ac48-581a8640f880-kube-rbac-proxy-cm podName:2d8da6d4-7c9a-42d9-ac48-581a8640f880 nodeName:}" failed. No retries permitted until 2026-07-21 00:41:42.140131416 +0000 UTC m=+49.746240446 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2d8da6d4-7c9a-42d9-ac48-581a8640f880-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5xx6h" (UID: "2d8da6d4-7c9a-42d9-ac48-581a8640f880") : configmap "kube-rbac-proxy" not found Jul 21 00:41:41.140430 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:41.140216 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 21 00:41:41.140430 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:41.140285 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d8da6d4-7c9a-42d9-ac48-581a8640f880-insights-runtime-extractor-tls podName:2d8da6d4-7c9a-42d9-ac48-581a8640f880 nodeName:}" failed. No retries permitted until 2026-07-21 00:41:42.140273667 +0000 UTC m=+49.746382685 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2d8da6d4-7c9a-42d9-ac48-581a8640f880-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5xx6h" (UID: "2d8da6d4-7c9a-42d9-ac48-581a8640f880") : secret "insights-runtime-extractor-tls" not found Jul 21 00:41:41.147397 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:41.147364 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6zg78" event={"ID":"f3e26871-c996-46c1-adfb-c96c1869814b","Type":"ContainerStarted","Data":"fc6ad154573dc5235a7507ff57ed69e3e32a6a5b1a8da7f632cac5391eca43f9"} Jul 21 00:41:41.148871 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:41.148856 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-rb4ph_3ee1dbf0-b4ff-4871-9432-284eb25f97b1/console-operator/1.log" Jul 21 00:41:41.149127 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:41.149113 2580 scope.go:117] "RemoveContainer" containerID="53298a4b69c8aa9fd6a6c7dd921a4923031bbf1503ff4ea1a994c1b004a08a3d" Jul 21 00:41:41.149273 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:41.149256 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-rb4ph_openshift-console-operator(3ee1dbf0-b4ff-4871-9432-284eb25f97b1)\"" pod="openshift-console-operator/console-operator-565b7b569d-rb4ph" podUID="3ee1dbf0-b4ff-4871-9432-284eb25f97b1" Jul 21 00:41:41.169675 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:41.169609 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-6zg78" podStartSLOduration=9.274957358 podStartE2EDuration="48.169598273s" totalImagesPullingTime="38.894640915s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 00:40:53 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:41:41.167739244 +0000 UTC m=+48.773848285" watchObservedRunningTime="2026-07-21 00:41:41.169598273 +0000 UTC m=+48.775707349" Jul 21 00:41:41.712315 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:41.712284 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-858wv_df3aad65-414d-4dd0-9e23-c0c0e46e4f06/dns-node-resolver/0.log" Jul 21 00:41:42.149669 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:42.149638 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2d8da6d4-7c9a-42d9-ac48-581a8640f880-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5xx6h\" (UID: \"2d8da6d4-7c9a-42d9-ac48-581a8640f880\") " pod="openshift-insights/insights-runtime-extractor-5xx6h" Jul 21 00:41:42.150070 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:42.149705 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2d8da6d4-7c9a-42d9-ac48-581a8640f880-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5xx6h\" (UID: \"2d8da6d4-7c9a-42d9-ac48-581a8640f880\") " pod="openshift-insights/insights-runtime-extractor-5xx6h" Jul 21 00:41:42.150070 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:42.149759 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 21 00:41:42.150070 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:42.149802 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 21 00:41:42.150070 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:42.149828 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2d8da6d4-7c9a-42d9-ac48-581a8640f880-kube-rbac-proxy-cm podName:2d8da6d4-7c9a-42d9-ac48-581a8640f880 nodeName:}" failed. No retries permitted until 2026-07-21 00:41:44.149808418 +0000 UTC m=+51.755917437 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2d8da6d4-7c9a-42d9-ac48-581a8640f880-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5xx6h" (UID: "2d8da6d4-7c9a-42d9-ac48-581a8640f880") : configmap "kube-rbac-proxy" not found Jul 21 00:41:42.150070 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:42.149844 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d8da6d4-7c9a-42d9-ac48-581a8640f880-insights-runtime-extractor-tls podName:2d8da6d4-7c9a-42d9-ac48-581a8640f880 nodeName:}" failed. No retries permitted until 2026-07-21 00:41:44.149838241 +0000 UTC m=+51.755947259 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2d8da6d4-7c9a-42d9-ac48-581a8640f880-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5xx6h" (UID: "2d8da6d4-7c9a-42d9-ac48-581a8640f880") : secret "insights-runtime-extractor-tls" not found Jul 21 00:41:42.712285 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:42.712256 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2rxh4_6fbd5477-ce7d-444d-938b-2c54aa3c040c/node-ca/0.log" Jul 21 00:41:43.513247 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:43.513203 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2vh4t_a625da2a-a501-4fa3-a8c0-21ac34950978/serve-healthcheck-canary/0.log" Jul 21 00:41:44.065604 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:44.065565 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8379df88-86a0-4d28-95fc-a102fff737fd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-b8c9w\" (UID: \"8379df88-86a0-4d28-95fc-a102fff737fd\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-b8c9w" Jul 21 00:41:44.065774 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:44.065622 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2ec824c1-400c-4ddb-ad88-bc5a37215aa8-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-twlx9\" (UID: \"2ec824c1-400c-4ddb-ad88-bc5a37215aa8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-twlx9" Jul 21 00:41:44.065774 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:44.065713 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 21 00:41:44.065774 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:44.065747 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 21 00:41:44.065870 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:44.065782 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8379df88-86a0-4d28-95fc-a102fff737fd-cluster-monitoring-operator-tls podName:8379df88-86a0-4d28-95fc-a102fff737fd nodeName:}" failed. No retries permitted until 2026-07-21 00:42:00.065766034 +0000 UTC m=+67.671875053 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8379df88-86a0-4d28-95fc-a102fff737fd-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-b8c9w" (UID: "8379df88-86a0-4d28-95fc-a102fff737fd") : secret "cluster-monitoring-operator-tls" not found Jul 21 00:41:44.065870 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:44.065796 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ec824c1-400c-4ddb-ad88-bc5a37215aa8-samples-operator-tls podName:2ec824c1-400c-4ddb-ad88-bc5a37215aa8 nodeName:}" failed. No retries permitted until 2026-07-21 00:42:00.065790426 +0000 UTC m=+67.671899444 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2ec824c1-400c-4ddb-ad88-bc5a37215aa8-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-twlx9" (UID: "2ec824c1-400c-4ddb-ad88-bc5a37215aa8") : secret "samples-operator-tls" not found Jul 21 00:41:44.166534 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:44.166511 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49cc45d2-5087-447a-9123-3885c7bc03a2-metrics-tls\") pod \"dns-default-nlszd\" (UID: \"49cc45d2-5087-447a-9123-3885c7bc03a2\") " pod="openshift-dns/dns-default-nlszd" Jul 21 00:41:44.166668 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:44.166557 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6543206b-bf43-43c5-930d-c3c8834b0f2c-registry-tls\") pod \"image-registry-bdd745b78-7cccc\" (UID: \"6543206b-bf43-43c5-930d-c3c8834b0f2c\") " pod="openshift-image-registry/image-registry-bdd745b78-7cccc" Jul 21 00:41:44.166668 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:44.166633 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2d8da6d4-7c9a-42d9-ac48-581a8640f880-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5xx6h\" (UID: \"2d8da6d4-7c9a-42d9-ac48-581a8640f880\") " pod="openshift-insights/insights-runtime-extractor-5xx6h" Jul 21 00:41:44.166668 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:44.166650 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 00:41:44.166812 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:44.166682 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/40960433-ccd7-4aa8-ab03-2cde39f80875-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-sx8h8\" (UID: \"40960433-ccd7-4aa8-ab03-2cde39f80875\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-sx8h8" Jul 21 00:41:44.166812 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:44.166696 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49cc45d2-5087-447a-9123-3885c7bc03a2-metrics-tls podName:49cc45d2-5087-447a-9123-3885c7bc03a2 nodeName:}" failed. No retries permitted until 2026-07-21 00:42:00.166682371 +0000 UTC m=+67.772791389 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/49cc45d2-5087-447a-9123-3885c7bc03a2-metrics-tls") pod "dns-default-nlszd" (UID: "49cc45d2-5087-447a-9123-3885c7bc03a2") : secret "dns-default-metrics-tls" not found Jul 21 00:41:44.166812 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:44.166726 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 21 00:41:44.166812 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:44.166739 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bdd745b78-7cccc: secret "image-registry-tls" not found Jul 21 00:41:44.166812 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:44.166739 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9970c160-3cf7-4d90-b892-1a337f23a065-service-ca-bundle\") pod \"router-default-69fd648678-dm5nn\" (UID: \"9970c160-3cf7-4d90-b892-1a337f23a065\") " pod="openshift-ingress/router-default-69fd648678-dm5nn" Jul 21 00:41:44.166812 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:44.166748 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 21 00:41:44.166812 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:44.166751 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 21 00:41:44.166812 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:44.166783 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6543206b-bf43-43c5-930d-c3c8834b0f2c-registry-tls podName:6543206b-bf43-43c5-930d-c3c8834b0f2c nodeName:}" failed. No retries permitted until 2026-07-21 00:42:00.166768143 +0000 UTC m=+67.772877175 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6543206b-bf43-43c5-930d-c3c8834b0f2c-registry-tls") pod "image-registry-bdd745b78-7cccc" (UID: "6543206b-bf43-43c5-930d-c3c8834b0f2c") : secret "image-registry-tls" not found Jul 21 00:41:44.166812 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:44.166760 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9970c160-3cf7-4d90-b892-1a337f23a065-metrics-certs\") pod \"router-default-69fd648678-dm5nn\" (UID: \"9970c160-3cf7-4d90-b892-1a337f23a065\") " pod="openshift-ingress/router-default-69fd648678-dm5nn" Jul 21 00:41:44.166812 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:44.166798 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40960433-ccd7-4aa8-ab03-2cde39f80875-networking-console-plugin-cert podName:40960433-ccd7-4aa8-ab03-2cde39f80875 nodeName:}" failed. No retries permitted until 2026-07-21 00:42:00.166791335 +0000 UTC m=+67.772900356 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/40960433-ccd7-4aa8-ab03-2cde39f80875-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-sx8h8" (UID: "40960433-ccd7-4aa8-ab03-2cde39f80875") : secret "networking-console-plugin-cert" not found Jul 21 00:41:44.166812 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:44.166807 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 21 00:41:44.167172 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:44.166833 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2d8da6d4-7c9a-42d9-ac48-581a8640f880-kube-rbac-proxy-cm podName:2d8da6d4-7c9a-42d9-ac48-581a8640f880 nodeName:}" failed. No retries permitted until 2026-07-21 00:41:48.166817743 +0000 UTC m=+55.772926763 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2d8da6d4-7c9a-42d9-ac48-581a8640f880-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5xx6h" (UID: "2d8da6d4-7c9a-42d9-ac48-581a8640f880") : configmap "kube-rbac-proxy" not found Jul 21 00:41:44.167172 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:44.166856 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2d8da6d4-7c9a-42d9-ac48-581a8640f880-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5xx6h\" (UID: \"2d8da6d4-7c9a-42d9-ac48-581a8640f880\") " pod="openshift-insights/insights-runtime-extractor-5xx6h" Jul 21 00:41:44.167172 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:44.166887 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 21 00:41:44.167172 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:44.166903 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9970c160-3cf7-4d90-b892-1a337f23a065-service-ca-bundle podName:9970c160-3cf7-4d90-b892-1a337f23a065 nodeName:}" failed. No retries permitted until 2026-07-21 00:42:00.166886143 +0000 UTC m=+67.772995164 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9970c160-3cf7-4d90-b892-1a337f23a065-service-ca-bundle") pod "router-default-69fd648678-dm5nn" (UID: "9970c160-3cf7-4d90-b892-1a337f23a065") : configmap references non-existent config key: service-ca.crt Jul 21 00:41:44.167172 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:44.166928 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9970c160-3cf7-4d90-b892-1a337f23a065-metrics-certs podName:9970c160-3cf7-4d90-b892-1a337f23a065 nodeName:}" failed. No retries permitted until 2026-07-21 00:42:00.166917022 +0000 UTC m=+67.773026045 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9970c160-3cf7-4d90-b892-1a337f23a065-metrics-certs") pod "router-default-69fd648678-dm5nn" (UID: "9970c160-3cf7-4d90-b892-1a337f23a065") : secret "router-metrics-certs-default" not found Jul 21 00:41:44.167172 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:44.166942 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d8da6d4-7c9a-42d9-ac48-581a8640f880-insights-runtime-extractor-tls podName:2d8da6d4-7c9a-42d9-ac48-581a8640f880 nodeName:}" failed. No retries permitted until 2026-07-21 00:41:48.166934135 +0000 UTC m=+55.773043154 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2d8da6d4-7c9a-42d9-ac48-581a8640f880-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5xx6h" (UID: "2d8da6d4-7c9a-42d9-ac48-581a8640f880") : secret "insights-runtime-extractor-tls" not found Jul 21 00:41:48.201270 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:48.201227 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2d8da6d4-7c9a-42d9-ac48-581a8640f880-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5xx6h\" (UID: \"2d8da6d4-7c9a-42d9-ac48-581a8640f880\") " pod="openshift-insights/insights-runtime-extractor-5xx6h" Jul 21 00:41:48.201683 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:48.201334 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2d8da6d4-7c9a-42d9-ac48-581a8640f880-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5xx6h\" (UID: \"2d8da6d4-7c9a-42d9-ac48-581a8640f880\") " pod="openshift-insights/insights-runtime-extractor-5xx6h" Jul 21 00:41:48.201683 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:48.201502 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 21 00:41:48.201683 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:48.201566 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d8da6d4-7c9a-42d9-ac48-581a8640f880-insights-runtime-extractor-tls podName:2d8da6d4-7c9a-42d9-ac48-581a8640f880 nodeName:}" failed. No retries permitted until 2026-07-21 00:41:56.201546398 +0000 UTC m=+63.807655416 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2d8da6d4-7c9a-42d9-ac48-581a8640f880-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5xx6h" (UID: "2d8da6d4-7c9a-42d9-ac48-581a8640f880") : secret "insights-runtime-extractor-tls" not found Jul 21 00:41:48.201815 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:48.201798 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2d8da6d4-7c9a-42d9-ac48-581a8640f880-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5xx6h\" (UID: \"2d8da6d4-7c9a-42d9-ac48-581a8640f880\") " pod="openshift-insights/insights-runtime-extractor-5xx6h" Jul 21 00:41:48.556877 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:48.556782 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-rb4ph" Jul 21 00:41:48.556877 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:48.556831 2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-565b7b569d-rb4ph" Jul 21 00:41:48.557691 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:48.557669 2580 scope.go:117] "RemoveContainer" containerID="53298a4b69c8aa9fd6a6c7dd921a4923031bbf1503ff4ea1a994c1b004a08a3d" Jul 21 00:41:48.557890 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:41:48.557868 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-rb4ph_openshift-console-operator(3ee1dbf0-b4ff-4871-9432-284eb25f97b1)\"" pod="openshift-console-operator/console-operator-565b7b569d-rb4ph" podUID="3ee1dbf0-b4ff-4871-9432-284eb25f97b1" Jul 21 00:41:54.082426 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:54.082396 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bctst" Jul 21 00:41:56.272370 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:56.272337 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2d8da6d4-7c9a-42d9-ac48-581a8640f880-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5xx6h\" (UID: \"2d8da6d4-7c9a-42d9-ac48-581a8640f880\") " pod="openshift-insights/insights-runtime-extractor-5xx6h" Jul 21 00:41:56.274638 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:56.274618 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2d8da6d4-7c9a-42d9-ac48-581a8640f880-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5xx6h\" (UID: \"2d8da6d4-7c9a-42d9-ac48-581a8640f880\") " pod="openshift-insights/insights-runtime-extractor-5xx6h" Jul 21 00:41:56.562348 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:56.562269 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fhmkx\"" Jul 21 00:41:56.570287 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:56.570268 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5xx6h" Jul 21 00:41:56.688644 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:56.688616 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5xx6h"] Jul 21 00:41:56.691430 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:41:56.691404 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d8da6d4_7c9a_42d9_ac48_581a8640f880.slice/crio-731eb0e9deffcacac10bc4ab2fae3870f45824d8ad87131feaccec6bd362b7c3 WatchSource:0}: Error finding container 731eb0e9deffcacac10bc4ab2fae3870f45824d8ad87131feaccec6bd362b7c3: Status 404 returned error can't find the container with id 731eb0e9deffcacac10bc4ab2fae3870f45824d8ad87131feaccec6bd362b7c3 Jul 21 00:41:57.198118 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:57.198084 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5xx6h" event={"ID":"2d8da6d4-7c9a-42d9-ac48-581a8640f880","Type":"ContainerStarted","Data":"731eb0e9deffcacac10bc4ab2fae3870f45824d8ad87131feaccec6bd362b7c3"} Jul 21 00:41:58.696871 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:58.696834 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cfb338e-19eb-45ee-a3de-ea7311469109-metrics-certs\") pod \"network-metrics-daemon-4nq5b\" (UID: \"6cfb338e-19eb-45ee-a3de-ea7311469109\") " pod="openshift-multus/network-metrics-daemon-4nq5b" Jul 21 00:41:58.697335 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:58.696959 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fda1cd2e-3d5d-4dc1-94a1-1b8ebbb4a199-original-pull-secret\") pod \"global-pull-secret-syncer-dzcmw\" (UID: \"fda1cd2e-3d5d-4dc1-94a1-1b8ebbb4a199\") " pod="kube-system/global-pull-secret-syncer-dzcmw" Jul 21 00:41:58.699866 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:58.699847 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 21 00:41:58.699972 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:58.699848 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 21 00:41:58.709590 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:58.709556 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cfb338e-19eb-45ee-a3de-ea7311469109-metrics-certs\") pod \"network-metrics-daemon-4nq5b\" (UID: \"6cfb338e-19eb-45ee-a3de-ea7311469109\") " pod="openshift-multus/network-metrics-daemon-4nq5b" Jul 21 00:41:58.709770 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:58.709725 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fda1cd2e-3d5d-4dc1-94a1-1b8ebbb4a199-original-pull-secret\") pod \"global-pull-secret-syncer-dzcmw\" (UID: \"fda1cd2e-3d5d-4dc1-94a1-1b8ebbb4a199\") " pod="kube-system/global-pull-secret-syncer-dzcmw" Jul 21 00:41:58.723796 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:58.723775 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzcmw" Jul 21 00:41:58.735375 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:58.735354 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wcfmk\"" Jul 21 00:41:58.743599 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:58.743574 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4nq5b" Jul 21 00:41:58.800090 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:58.799552 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wxxq\" (UniqueName: \"kubernetes.io/projected/e11587d4-f449-4b99-8355-787a52243ba4-kube-api-access-8wxxq\") pod \"network-check-target-9r2ff\" (UID: \"e11587d4-f449-4b99-8355-787a52243ba4\") " pod="openshift-network-diagnostics/network-check-target-9r2ff" Jul 21 00:41:58.804505 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:58.804411 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wxxq\" (UniqueName: \"kubernetes.io/projected/e11587d4-f449-4b99-8355-787a52243ba4-kube-api-access-8wxxq\") pod \"network-check-target-9r2ff\" (UID: \"e11587d4-f449-4b99-8355-787a52243ba4\") " pod="openshift-network-diagnostics/network-check-target-9r2ff" Jul 21 00:41:58.866986 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:58.866930 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dzcmw"] Jul 21 00:41:58.870316 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:41:58.870283 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfda1cd2e_3d5d_4dc1_94a1_1b8ebbb4a199.slice/crio-1ca729065b3f35f84f4ce9e3e4f0e8224b9f557f5b04a4976e92833e691f0793 WatchSource:0}: Error finding container 1ca729065b3f35f84f4ce9e3e4f0e8224b9f557f5b04a4976e92833e691f0793: Status 404 returned error can't find the container with id 1ca729065b3f35f84f4ce9e3e4f0e8224b9f557f5b04a4976e92833e691f0793 Jul 21 00:41:58.880197 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:58.880174 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4nq5b"] Jul 21 00:41:58.882901 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:41:58.882878 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cfb338e_19eb_45ee_a3de_ea7311469109.slice/crio-74f4a7e8bb1a40efee42772a3e0c847a2152ceb71ae81758adffe42d9503a260 WatchSource:0}: Error finding container 74f4a7e8bb1a40efee42772a3e0c847a2152ceb71ae81758adffe42d9503a260: Status 404 returned error can't find the container with id 74f4a7e8bb1a40efee42772a3e0c847a2152ceb71ae81758adffe42d9503a260 Jul 21 00:41:58.922855 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:58.922836 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-nqmsz\"" Jul 21 00:41:58.931035 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:58.931018 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9r2ff" Jul 21 00:41:59.065279 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:59.065252 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9r2ff"] Jul 21 00:41:59.067643 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:41:59.067616 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode11587d4_f449_4b99_8355_787a52243ba4.slice/crio-2987e8bee8869f6a78b6ecc350af800a5b9ed65508eba86f22fcb001259b3b22 WatchSource:0}: Error finding container 2987e8bee8869f6a78b6ecc350af800a5b9ed65508eba86f22fcb001259b3b22: Status 404 returned error can't find the container with id 2987e8bee8869f6a78b6ecc350af800a5b9ed65508eba86f22fcb001259b3b22 Jul 21 00:41:59.205905 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:59.205861 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9r2ff" event={"ID":"e11587d4-f449-4b99-8355-787a52243ba4","Type":"ContainerStarted","Data":"100bd63ea38ab59967cfec52f51f4aca5a16109ca8e4477b59f407b6e814f05f"} Jul 21 00:41:59.206088 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:59.205920 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9r2ff" event={"ID":"e11587d4-f449-4b99-8355-787a52243ba4","Type":"ContainerStarted","Data":"2987e8bee8869f6a78b6ecc350af800a5b9ed65508eba86f22fcb001259b3b22"} Jul 21 00:41:59.206088 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:59.205989 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-9r2ff" Jul 21 00:41:59.207047 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:59.207015 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4nq5b" event={"ID":"6cfb338e-19eb-45ee-a3de-ea7311469109","Type":"ContainerStarted","Data":"74f4a7e8bb1a40efee42772a3e0c847a2152ceb71ae81758adffe42d9503a260"} Jul 21 00:41:59.208051 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:59.208030 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dzcmw" event={"ID":"fda1cd2e-3d5d-4dc1-94a1-1b8ebbb4a199","Type":"ContainerStarted","Data":"1ca729065b3f35f84f4ce9e3e4f0e8224b9f557f5b04a4976e92833e691f0793"} Jul 21 00:41:59.222552 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:41:59.222510 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-9r2ff" podStartSLOduration=66.222496518 podStartE2EDuration="1m6.222496518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:40:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:41:59.221322192 +0000 UTC m=+66.827431232" watchObservedRunningTime="2026-07-21 00:41:59.222496518 +0000 UTC m=+66.828605536" Jul 21 00:42:00.111454 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:00.111418 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8379df88-86a0-4d28-95fc-a102fff737fd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-b8c9w\" (UID: \"8379df88-86a0-4d28-95fc-a102fff737fd\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-b8c9w" Jul 21 00:42:00.111922 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:00.111477 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2ec824c1-400c-4ddb-ad88-bc5a37215aa8-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-twlx9\" (UID: \"2ec824c1-400c-4ddb-ad88-bc5a37215aa8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-twlx9" Jul 21 00:42:00.114397 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:00.114365 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8379df88-86a0-4d28-95fc-a102fff737fd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-b8c9w\" (UID: \"8379df88-86a0-4d28-95fc-a102fff737fd\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-b8c9w" Jul 21 00:42:00.114542 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:00.114370 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2ec824c1-400c-4ddb-ad88-bc5a37215aa8-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-twlx9\" (UID: \"2ec824c1-400c-4ddb-ad88-bc5a37215aa8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-twlx9" Jul 21 00:42:00.211976 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:00.211931 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/40960433-ccd7-4aa8-ab03-2cde39f80875-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-sx8h8\" (UID: \"40960433-ccd7-4aa8-ab03-2cde39f80875\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-sx8h8" Jul 21 00:42:00.212154 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:00.211986 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9970c160-3cf7-4d90-b892-1a337f23a065-service-ca-bundle\") pod \"router-default-69fd648678-dm5nn\" (UID: \"9970c160-3cf7-4d90-b892-1a337f23a065\") " pod="openshift-ingress/router-default-69fd648678-dm5nn" Jul 21 00:42:00.212154 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:00.212006 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9970c160-3cf7-4d90-b892-1a337f23a065-metrics-certs\") pod \"router-default-69fd648678-dm5nn\" (UID: \"9970c160-3cf7-4d90-b892-1a337f23a065\") " pod="openshift-ingress/router-default-69fd648678-dm5nn" Jul 21 00:42:00.212154 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:00.212025 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49cc45d2-5087-447a-9123-3885c7bc03a2-metrics-tls\") pod \"dns-default-nlszd\" (UID: \"49cc45d2-5087-447a-9123-3885c7bc03a2\") " pod="openshift-dns/dns-default-nlszd" Jul 21 00:42:00.212154 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:00.212072 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6543206b-bf43-43c5-930d-c3c8834b0f2c-registry-tls\") pod \"image-registry-bdd745b78-7cccc\" (UID: \"6543206b-bf43-43c5-930d-c3c8834b0f2c\") " pod="openshift-image-registry/image-registry-bdd745b78-7cccc" Jul 21 00:42:00.212773 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:00.212738 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9970c160-3cf7-4d90-b892-1a337f23a065-service-ca-bundle\") pod \"router-default-69fd648678-dm5nn\" (UID: \"9970c160-3cf7-4d90-b892-1a337f23a065\") " pod="openshift-ingress/router-default-69fd648678-dm5nn" Jul 21 00:42:00.214640 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:00.214599 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49cc45d2-5087-447a-9123-3885c7bc03a2-metrics-tls\") pod \"dns-default-nlszd\" (UID: \"49cc45d2-5087-447a-9123-3885c7bc03a2\") " pod="openshift-dns/dns-default-nlszd" Jul 21 00:42:00.214796 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:00.214757 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/40960433-ccd7-4aa8-ab03-2cde39f80875-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-sx8h8\" (UID: \"40960433-ccd7-4aa8-ab03-2cde39f80875\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-sx8h8" Jul 21 00:42:00.214909 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:00.214888 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6543206b-bf43-43c5-930d-c3c8834b0f2c-registry-tls\") pod \"image-registry-bdd745b78-7cccc\" (UID: \"6543206b-bf43-43c5-930d-c3c8834b0f2c\") " pod="openshift-image-registry/image-registry-bdd745b78-7cccc" Jul 21 00:42:00.214981 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:00.214888 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9970c160-3cf7-4d90-b892-1a337f23a065-metrics-certs\") pod \"router-default-69fd648678-dm5nn\" (UID: \"9970c160-3cf7-4d90-b892-1a337f23a065\") " pod="openshift-ingress/router-default-69fd648678-dm5nn" Jul 21 00:42:00.379293 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:00.379196 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-rntns\"" Jul 21 00:42:00.385728 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:00.385704 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-gfgjp\"" Jul 21 00:42:00.386716 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:00.386698 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-twlx9" Jul 21 00:42:00.393751 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:00.393735 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-b8c9w" Jul 21 00:42:00.398452 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:00.398435 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-rpj4j\"" Jul 21 00:42:00.404926 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:00.404909 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-9r7r2\"" Jul 21 00:42:00.406828 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:00.406807 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-69fd648678-dm5nn" Jul 21 00:42:00.413059 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:00.413043 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-bdd745b78-7cccc" Jul 21 00:42:00.418491 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:00.418474 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-pxt2s\"" Jul 21 00:42:00.426525 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:00.426509 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-sx8h8" Jul 21 00:42:00.430939 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:00.430922 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lf5vb\"" Jul 21 00:42:00.438880 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:00.438866 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nlszd" Jul 21 00:42:01.957465 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:01.957432 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-b8c9w"] Jul 21 00:42:02.094450 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:42:02.094223 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8379df88_86a0_4d28_95fc_a102fff737fd.slice/crio-7366931f84dfb84d3b87074a955d3c2c8575c7e60c75d87610f7c758f885438c WatchSource:0}: Error finding container 7366931f84dfb84d3b87074a955d3c2c8575c7e60c75d87610f7c758f885438c: Status 404 returned error can't find the container with id 7366931f84dfb84d3b87074a955d3c2c8575c7e60c75d87610f7c758f885438c Jul 21 00:42:02.228157 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:02.228058 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-b8c9w" event={"ID":"8379df88-86a0-4d28-95fc-a102fff737fd","Type":"ContainerStarted","Data":"7366931f84dfb84d3b87074a955d3c2c8575c7e60c75d87610f7c758f885438c"} Jul 21 00:42:02.327591 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:02.319606 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-twlx9"] Jul 21 00:42:02.333015 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:02.332940 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-sx8h8"] Jul 21 00:42:02.346455 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:02.344797 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nlszd"] Jul 21 00:42:02.351851 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:02.351361 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-bdd745b78-7cccc"] Jul 21 00:42:02.359959 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:42:02.359929 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6543206b_bf43_43c5_930d_c3c8834b0f2c.slice/crio-f2b94f798b8354222140b152f142887c38778eb9bdaa4bd6707e8c0b010cb37d WatchSource:0}: Error finding container f2b94f798b8354222140b152f142887c38778eb9bdaa4bd6707e8c0b010cb37d: Status 404 returned error can't find the container with id f2b94f798b8354222140b152f142887c38778eb9bdaa4bd6707e8c0b010cb37d Jul 21 00:42:02.383940 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:02.383844 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-69fd648678-dm5nn"] Jul 21 00:42:02.386529 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:42:02.386306 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9970c160_3cf7_4d90_b892_1a337f23a065.slice/crio-6b0ed60cfe855c825f553812cf78846dc6d3d9f399acb722f089893aaf9f182d WatchSource:0}: Error finding container 6b0ed60cfe855c825f553812cf78846dc6d3d9f399acb722f089893aaf9f182d: Status 404 returned error can't find the container with id 6b0ed60cfe855c825f553812cf78846dc6d3d9f399acb722f089893aaf9f182d Jul 21 00:42:02.908272 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:02.907946 2580 scope.go:117] "RemoveContainer" containerID="53298a4b69c8aa9fd6a6c7dd921a4923031bbf1503ff4ea1a994c1b004a08a3d" Jul 21 00:42:03.239159 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:03.239085 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-rb4ph_3ee1dbf0-b4ff-4871-9432-284eb25f97b1/console-operator/1.log" Jul 21 00:42:03.239611 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:03.239178 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-rb4ph" event={"ID":"3ee1dbf0-b4ff-4871-9432-284eb25f97b1","Type":"ContainerStarted","Data":"7fcc6440d45f7f8b888a984ffb826edfdaec928b3e415a81fc52f27d8d534f30"} Jul 21 00:42:03.240037 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:03.239992 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-rb4ph" Jul 21 00:42:03.246028 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:03.245534 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-69fd648678-dm5nn" event={"ID":"9970c160-3cf7-4d90-b892-1a337f23a065","Type":"ContainerStarted","Data":"3867949498e2b9b8f27f931e1f86493ed46a044dc267371f68f47421a0c06306"} Jul 21 00:42:03.246028 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:03.245580 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-69fd648678-dm5nn" event={"ID":"9970c160-3cf7-4d90-b892-1a337f23a065","Type":"ContainerStarted","Data":"6b0ed60cfe855c825f553812cf78846dc6d3d9f399acb722f089893aaf9f182d"} Jul 21 00:42:03.252893 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:03.252425 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4nq5b" event={"ID":"6cfb338e-19eb-45ee-a3de-ea7311469109","Type":"ContainerStarted","Data":"7aed9f0eb41af24c85f0b02a25ff4c54233facde6415647f570dafd7a3aa2eb7"} Jul 21 00:42:03.252893 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:03.252466 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4nq5b" event={"ID":"6cfb338e-19eb-45ee-a3de-ea7311469109","Type":"ContainerStarted","Data":"48774f5d15f363889cf9bc2922fbf2c94d6ef1e9010615a69188148bcb927216"} Jul 21 00:42:03.255638 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:03.255565 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-twlx9" event={"ID":"2ec824c1-400c-4ddb-ad88-bc5a37215aa8","Type":"ContainerStarted","Data":"e55ec512e35a96f338acfd81684a32870eb9fc160a96e524866206a93dd41b88"} Jul 21 00:42:03.258696 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:03.258645 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-565b7b569d-rb4ph" podStartSLOduration=49.951843928 podStartE2EDuration="59.258627442s" totalImagesPullingTime="9.306783514s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:41:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:42:03.257600279 +0000 UTC m=+70.863709318" watchObservedRunningTime="2026-07-21 00:42:03.258627442 +0000 UTC m=+70.864736486" Jul 21 00:42:03.260414 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:03.260354 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nlszd" event={"ID":"49cc45d2-5087-447a-9123-3885c7bc03a2","Type":"ContainerStarted","Data":"ee48bc9defa290184ba98c07e4b8a6c35a0ecbdbd7e8f29925cf6f671c1fa699"} Jul 21 00:42:03.272102 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:03.271868 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-sx8h8" event={"ID":"40960433-ccd7-4aa8-ab03-2cde39f80875","Type":"ContainerStarted","Data":"a8db1e5497f246267826559c436fc691ae85e82679b3604eca318d76dbb752ec"} Jul 21 00:42:03.274670 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:03.274625 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-4nq5b" podStartSLOduration=67.009413041 podStartE2EDuration="1m10.27461038s" totalImagesPullingTime="3.265197339s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:40:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:42:03.273197619 +0000 UTC m=+70.879306661" watchObservedRunningTime="2026-07-21 00:42:03.27461038 +0000 UTC m=+70.880719420" Jul 21 00:42:03.278087 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:03.278015 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5xx6h" event={"ID":"2d8da6d4-7c9a-42d9-ac48-581a8640f880","Type":"ContainerStarted","Data":"9285aec46d2969ca20ddbea9519fe389c08df4019bb3d5a6eac843fdb145437b"} Jul 21 00:42:03.281941 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:03.281918 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-bdd745b78-7cccc" event={"ID":"6543206b-bf43-43c5-930d-c3c8834b0f2c","Type":"ContainerStarted","Data":"04d7b35d2d264368e571479044ec2d30dc9116c489732bff4839cdcd9ecb42b7"} Jul 21 00:42:03.282051 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:03.281961 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-bdd745b78-7cccc" event={"ID":"6543206b-bf43-43c5-930d-c3c8834b0f2c","Type":"ContainerStarted","Data":"f2b94f798b8354222140b152f142887c38778eb9bdaa4bd6707e8c0b010cb37d"} Jul 21 00:42:03.282421 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:03.282367 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-bdd745b78-7cccc" Jul 21 00:42:03.311295 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:03.311131 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-69fd648678-dm5nn" podStartSLOduration=58.311113797 podStartE2EDuration="58.311113797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:41:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:42:03.292306446 +0000 UTC m=+70.898415482" watchObservedRunningTime="2026-07-21 00:42:03.311113797 +0000 UTC m=+70.917222840" Jul 21 00:42:03.312863 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:03.311481 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-bdd745b78-7cccc" podStartSLOduration=58.311470779 podStartE2EDuration="58.311470779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:41:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:42:03.31085469 +0000 UTC m=+70.916963733" watchObservedRunningTime="2026-07-21 00:42:03.311470779 +0000 UTC m=+70.917579820" Jul 21 00:42:03.407921 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:03.407890 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-69fd648678-dm5nn" Jul 21 00:42:03.410990 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:03.410778 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-69fd648678-dm5nn" Jul 21 00:42:04.151514 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:04.151439 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-bdd745b78-7cccc"] Jul 21 00:42:04.211339 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:04.211108 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-565b7b569d-rb4ph" Jul 21 00:42:04.228408 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:04.227142 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5dbc6848f8-w729h"] Jul 21 00:42:04.230990 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:04.230646 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5dbc6848f8-w729h" Jul 21 00:42:04.261965 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:04.259380 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5dbc6848f8-w729h"] Jul 21 00:42:04.291863 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:04.291745 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-69fd648678-dm5nn" Jul 21 00:42:04.296478 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:04.296417 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-69fd648678-dm5nn" Jul 21 00:42:04.356755 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:04.353317 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6c1e84ef-e02d-4b63-8538-e88272dfdffd-image-registry-private-configuration\") pod \"image-registry-5dbc6848f8-w729h\" (UID: \"6c1e84ef-e02d-4b63-8538-e88272dfdffd\") " pod="openshift-image-registry/image-registry-5dbc6848f8-w729h" Jul 21 00:42:04.356755 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:04.353431 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6c1e84ef-e02d-4b63-8538-e88272dfdffd-registry-tls\") pod \"image-registry-5dbc6848f8-w729h\" (UID: \"6c1e84ef-e02d-4b63-8538-e88272dfdffd\") " pod="openshift-image-registry/image-registry-5dbc6848f8-w729h" Jul 21 00:42:04.356755 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:04.353472 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6c1e84ef-e02d-4b63-8538-e88272dfdffd-registry-certificates\") pod \"image-registry-5dbc6848f8-w729h\" (UID: \"6c1e84ef-e02d-4b63-8538-e88272dfdffd\") " pod="openshift-image-registry/image-registry-5dbc6848f8-w729h" Jul 21 00:42:04.356755 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:04.353509 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6c1e84ef-e02d-4b63-8538-e88272dfdffd-ca-trust-extracted\") pod \"image-registry-5dbc6848f8-w729h\" (UID: \"6c1e84ef-e02d-4b63-8538-e88272dfdffd\") " pod="openshift-image-registry/image-registry-5dbc6848f8-w729h" Jul 21 00:42:04.356755 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:04.353540 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6c1e84ef-e02d-4b63-8538-e88272dfdffd-installation-pull-secrets\") pod \"image-registry-5dbc6848f8-w729h\" (UID: \"6c1e84ef-e02d-4b63-8538-e88272dfdffd\") " pod="openshift-image-registry/image-registry-5dbc6848f8-w729h" Jul 21 00:42:04.356755 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:04.353566 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq4th\" (UniqueName: \"kubernetes.io/projected/6c1e84ef-e02d-4b63-8538-e88272dfdffd-kube-api-access-cq4th\") pod \"image-registry-5dbc6848f8-w729h\" (UID: \"6c1e84ef-e02d-4b63-8538-e88272dfdffd\") " pod="openshift-image-registry/image-registry-5dbc6848f8-w729h" Jul 21 00:42:04.356755 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:04.353650 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6c1e84ef-e02d-4b63-8538-e88272dfdffd-bound-sa-token\") pod \"image-registry-5dbc6848f8-w729h\" (UID: \"6c1e84ef-e02d-4b63-8538-e88272dfdffd\") " pod="openshift-image-registry/image-registry-5dbc6848f8-w729h" Jul 21 00:42:04.356755 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:04.353788 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6c1e84ef-e02d-4b63-8538-e88272dfdffd-trusted-ca\") pod \"image-registry-5dbc6848f8-w729h\" (UID: \"6c1e84ef-e02d-4b63-8538-e88272dfdffd\") " pod="openshift-image-registry/image-registry-5dbc6848f8-w729h" Jul 21 00:42:04.454780 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:04.454746 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6c1e84ef-e02d-4b63-8538-e88272dfdffd-registry-tls\") pod \"image-registry-5dbc6848f8-w729h\" (UID: \"6c1e84ef-e02d-4b63-8538-e88272dfdffd\") " pod="openshift-image-registry/image-registry-5dbc6848f8-w729h" Jul 21 00:42:04.454951 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:04.454788 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6c1e84ef-e02d-4b63-8538-e88272dfdffd-registry-certificates\") pod \"image-registry-5dbc6848f8-w729h\" (UID: \"6c1e84ef-e02d-4b63-8538-e88272dfdffd\") " pod="openshift-image-registry/image-registry-5dbc6848f8-w729h" Jul 21 00:42:04.454951 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:04.454806 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6c1e84ef-e02d-4b63-8538-e88272dfdffd-ca-trust-extracted\") pod \"image-registry-5dbc6848f8-w729h\" (UID: \"6c1e84ef-e02d-4b63-8538-e88272dfdffd\") " pod="openshift-image-registry/image-registry-5dbc6848f8-w729h" Jul 21 00:42:04.454951 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:04.454826 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6c1e84ef-e02d-4b63-8538-e88272dfdffd-installation-pull-secrets\") pod \"image-registry-5dbc6848f8-w729h\" (UID: \"6c1e84ef-e02d-4b63-8538-e88272dfdffd\") " pod="openshift-image-registry/image-registry-5dbc6848f8-w729h" Jul 21 00:42:04.454951 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:04.454845 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq4th\" (UniqueName: \"kubernetes.io/projected/6c1e84ef-e02d-4b63-8538-e88272dfdffd-kube-api-access-cq4th\") pod \"image-registry-5dbc6848f8-w729h\" (UID: \"6c1e84ef-e02d-4b63-8538-e88272dfdffd\") " pod="openshift-image-registry/image-registry-5dbc6848f8-w729h" Jul 21 00:42:04.455168 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:04.454959 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6c1e84ef-e02d-4b63-8538-e88272dfdffd-bound-sa-token\") pod \"image-registry-5dbc6848f8-w729h\" (UID: \"6c1e84ef-e02d-4b63-8538-e88272dfdffd\") " pod="openshift-image-registry/image-registry-5dbc6848f8-w729h" Jul 21 00:42:04.455168 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:04.455023 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6c1e84ef-e02d-4b63-8538-e88272dfdffd-trusted-ca\") pod \"image-registry-5dbc6848f8-w729h\" (UID: \"6c1e84ef-e02d-4b63-8538-e88272dfdffd\") " pod="openshift-image-registry/image-registry-5dbc6848f8-w729h" Jul 21 00:42:04.455168 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:04.455074 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6c1e84ef-e02d-4b63-8538-e88272dfdffd-image-registry-private-configuration\") pod \"image-registry-5dbc6848f8-w729h\" (UID: \"6c1e84ef-e02d-4b63-8538-e88272dfdffd\") " pod="openshift-image-registry/image-registry-5dbc6848f8-w729h" Jul 21 00:42:04.456685 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:04.456262 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6c1e84ef-e02d-4b63-8538-e88272dfdffd-ca-trust-extracted\") pod \"image-registry-5dbc6848f8-w729h\" (UID: \"6c1e84ef-e02d-4b63-8538-e88272dfdffd\") " pod="openshift-image-registry/image-registry-5dbc6848f8-w729h" Jul 21 00:42:04.456685 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:04.456303 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6c1e84ef-e02d-4b63-8538-e88272dfdffd-trusted-ca\") pod \"image-registry-5dbc6848f8-w729h\" (UID: \"6c1e84ef-e02d-4b63-8538-e88272dfdffd\") " pod="openshift-image-registry/image-registry-5dbc6848f8-w729h" Jul 21 00:42:04.456685 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:04.456578 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6c1e84ef-e02d-4b63-8538-e88272dfdffd-registry-certificates\") pod \"image-registry-5dbc6848f8-w729h\" (UID: \"6c1e84ef-e02d-4b63-8538-e88272dfdffd\") " pod="openshift-image-registry/image-registry-5dbc6848f8-w729h" Jul 21 00:42:04.461665 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:04.461641 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6c1e84ef-e02d-4b63-8538-e88272dfdffd-installation-pull-secrets\") pod \"image-registry-5dbc6848f8-w729h\" (UID: \"6c1e84ef-e02d-4b63-8538-e88272dfdffd\") " pod="openshift-image-registry/image-registry-5dbc6848f8-w729h" Jul 21 00:42:04.463534 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:04.463487 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6c1e84ef-e02d-4b63-8538-e88272dfdffd-registry-tls\") pod \"image-registry-5dbc6848f8-w729h\" (UID: \"6c1e84ef-e02d-4b63-8538-e88272dfdffd\") " pod="openshift-image-registry/image-registry-5dbc6848f8-w729h" Jul 21 00:42:04.465564 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:04.464075 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-5bc67dcd46-wnqtz"] Jul 21 00:42:04.466102 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:04.466055 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6c1e84ef-e02d-4b63-8538-e88272dfdffd-image-registry-private-configuration\") pod \"image-registry-5dbc6848f8-w729h\" (UID: \"6c1e84ef-e02d-4b63-8538-e88272dfdffd\") " pod="openshift-image-registry/image-registry-5dbc6848f8-w729h" Jul 21 00:42:04.468088 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:04.467413 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-wnqtz" Jul 21 00:42:04.471315 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:04.471183 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 21 00:42:04.471960 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:04.471907 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 21 00:42:04.472918 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:04.472492 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-7dcms\"" Jul 21 00:42:04.479728 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:04.479687 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6c1e84ef-e02d-4b63-8538-e88272dfdffd-bound-sa-token\") pod \"image-registry-5dbc6848f8-w729h\" (UID: \"6c1e84ef-e02d-4b63-8538-e88272dfdffd\") " pod="openshift-image-registry/image-registry-5dbc6848f8-w729h" Jul 21 00:42:04.483385 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:04.483342 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq4th\" (UniqueName: \"kubernetes.io/projected/6c1e84ef-e02d-4b63-8538-e88272dfdffd-kube-api-access-cq4th\") pod \"image-registry-5dbc6848f8-w729h\" (UID: \"6c1e84ef-e02d-4b63-8538-e88272dfdffd\") " pod="openshift-image-registry/image-registry-5dbc6848f8-w729h" Jul 21 00:42:04.484817 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:04.484765 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-wnqtz"] Jul 21 00:42:04.546266 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:04.546048 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5dbc6848f8-w729h" Jul 21 00:42:04.557079 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:04.557015 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpdh9\" (UniqueName: \"kubernetes.io/projected/d4c2ffde-2686-4b53-a5f8-03df48ab6b6f-kube-api-access-rpdh9\") pod \"downloads-5bc67dcd46-wnqtz\" (UID: \"d4c2ffde-2686-4b53-a5f8-03df48ab6b6f\") " pod="openshift-console/downloads-5bc67dcd46-wnqtz" Jul 21 00:42:04.658393 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:04.657992 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpdh9\" (UniqueName: \"kubernetes.io/projected/d4c2ffde-2686-4b53-a5f8-03df48ab6b6f-kube-api-access-rpdh9\") pod \"downloads-5bc67dcd46-wnqtz\" (UID: \"d4c2ffde-2686-4b53-a5f8-03df48ab6b6f\") " pod="openshift-console/downloads-5bc67dcd46-wnqtz" Jul 21 00:42:04.667899 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:04.667813 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpdh9\" (UniqueName: \"kubernetes.io/projected/d4c2ffde-2686-4b53-a5f8-03df48ab6b6f-kube-api-access-rpdh9\") pod \"downloads-5bc67dcd46-wnqtz\" (UID: \"d4c2ffde-2686-4b53-a5f8-03df48ab6b6f\") " pod="openshift-console/downloads-5bc67dcd46-wnqtz" Jul 21 00:42:04.732440 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:04.732328 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5dbc6848f8-w729h"] Jul 21 00:42:04.806820 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:04.806591 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-wnqtz" Jul 21 00:42:05.295016 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:05.294945 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-sx8h8" event={"ID":"40960433-ccd7-4aa8-ab03-2cde39f80875","Type":"ContainerStarted","Data":"8c5fb92eb3fcc1f3537b4be4bee94b8be8bc38766c49b1566a1b2346af1e3c64"} Jul 21 00:42:05.310567 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:05.310513 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-sx8h8" podStartSLOduration=43.141586758 podStartE2EDuration="45.310492658s" totalImagesPullingTime="2.1689059s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:41:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:42:05.309885263 +0000 UTC m=+72.915994317" watchObservedRunningTime="2026-07-21 00:42:05.310492658 +0000 UTC m=+72.916601695" Jul 21 00:42:09.123632 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:42:09.123600 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c1e84ef_e02d_4b63_8538_e88272dfdffd.slice/crio-37ec2df95beebdeb51fdd2f77e2adb1be71c2044baa17f599e7e079eaa1b12aa WatchSource:0}: Error finding container 37ec2df95beebdeb51fdd2f77e2adb1be71c2044baa17f599e7e079eaa1b12aa: Status 404 returned error can't find the container with id 37ec2df95beebdeb51fdd2f77e2adb1be71c2044baa17f599e7e079eaa1b12aa Jul 21 00:42:09.311817 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:09.311770 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5dbc6848f8-w729h" event={"ID":"6c1e84ef-e02d-4b63-8538-e88272dfdffd","Type":"ContainerStarted","Data":"37ec2df95beebdeb51fdd2f77e2adb1be71c2044baa17f599e7e079eaa1b12aa"} Jul 21 00:42:10.782785 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:10.782751 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-wnqtz"] Jul 21 00:42:10.832557 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:42:10.832509 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4c2ffde_2686_4b53_a5f8_03df48ab6b6f.slice/crio-0ec4d87fd478923c201c9936a85bdb359fb52e1c9194373b4d13111854e62386 WatchSource:0}: Error finding container 0ec4d87fd478923c201c9936a85bdb359fb52e1c9194373b4d13111854e62386: Status 404 returned error can't find the container with id 0ec4d87fd478923c201c9936a85bdb359fb52e1c9194373b4d13111854e62386 Jul 21 00:42:11.322258 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:11.322208 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nlszd" event={"ID":"49cc45d2-5087-447a-9123-3885c7bc03a2","Type":"ContainerStarted","Data":"88ab3490a51286eed4f7443d2ec2a1b33167b6afb073510478e2c51086cb6fed"} Jul 21 00:42:11.322422 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:11.322294 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nlszd" event={"ID":"49cc45d2-5087-447a-9123-3885c7bc03a2","Type":"ContainerStarted","Data":"ddf5076215d73e2f1166acd84e007674ea92c8a33f32b6e3c8061a336f0fa1f1"} Jul 21 00:42:11.322422 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:11.322375 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-nlszd" Jul 21 00:42:11.323827 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:11.323802 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-b8c9w" event={"ID":"8379df88-86a0-4d28-95fc-a102fff737fd","Type":"ContainerStarted","Data":"d58ec7347f398bc52f445cc8a1acfdcbb10383dcbdd3d635708129d02160b63f"} Jul 21 00:42:11.325417 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:11.325389 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5dbc6848f8-w729h" event={"ID":"6c1e84ef-e02d-4b63-8538-e88272dfdffd","Type":"ContainerStarted","Data":"ab9812035457242af4aca0f82c371f3f7570f73dd1cd55704042589205a09c3c"} Jul 21 00:42:11.325898 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:11.325879 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5dbc6848f8-w729h" Jul 21 00:42:11.327381 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:11.327359 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-wnqtz" event={"ID":"d4c2ffde-2686-4b53-a5f8-03df48ab6b6f","Type":"ContainerStarted","Data":"0ec4d87fd478923c201c9936a85bdb359fb52e1c9194373b4d13111854e62386"} Jul 21 00:42:11.330032 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:11.329987 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5xx6h" event={"ID":"2d8da6d4-7c9a-42d9-ac48-581a8640f880","Type":"ContainerStarted","Data":"749652d307f6572e26284516c89e2042acef66915aabf68126591c03949da981"} Jul 21 00:42:11.331785 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:11.331756 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dzcmw" event={"ID":"fda1cd2e-3d5d-4dc1-94a1-1b8ebbb4a199","Type":"ContainerStarted","Data":"83fd95c02591a817d86a562f98e6ea4dd68e5f10520b1d6653207fb21c57609a"} Jul 21 00:42:11.334629 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:11.334278 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-twlx9" event={"ID":"2ec824c1-400c-4ddb-ad88-bc5a37215aa8","Type":"ContainerStarted","Data":"6f050657224ce2d96cc6b77d6c430336ff4a59f6aaf750c5daa29e7996bea72b"} Jul 21 00:42:11.334629 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:11.334310 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-twlx9" event={"ID":"2ec824c1-400c-4ddb-ad88-bc5a37215aa8","Type":"ContainerStarted","Data":"55caa449cbf8b99f9c63b2b86a703cbc981f33e7e0351d3ec09d7a4d25d7c5ab"} Jul 21 00:42:11.340222 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:11.340168 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-nlszd" podStartSLOduration=34.873481586 podStartE2EDuration="43.340152561s" totalImagesPullingTime="8.466670975s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:41:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:42:11.338929292 +0000 UTC m=+78.945038333" watchObservedRunningTime="2026-07-21 00:42:11.340152561 +0000 UTC m=+78.946261599" Jul 21 00:42:11.355396 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:11.355346 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-b8c9w" podStartSLOduration=57.661870693 podStartE2EDuration="1m6.355329371s" totalImagesPullingTime="8.693458678s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:41:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:42:11.354698189 +0000 UTC m=+78.960807230" watchObservedRunningTime="2026-07-21 00:42:11.355329371 +0000 UTC m=+78.961438389" Jul 21 00:42:11.370743 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:11.370687 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-dzcmw" podStartSLOduration=66.407832728 podStartE2EDuration="1m18.370668639s" totalImagesPullingTime="11.962835911s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 00:40:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:42:11.370422387 +0000 UTC m=+78.976531428" watchObservedRunningTime="2026-07-21 00:42:11.370668639 +0000 UTC m=+78.976777679" Jul 21 00:42:11.392575 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:11.392401 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5dbc6848f8-w729h" podStartSLOduration=7.39237808 podStartE2EDuration="7.39237808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:42:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:42:11.39043623 +0000 UTC m=+78.996545271" watchObservedRunningTime="2026-07-21 00:42:11.39237808 +0000 UTC m=+78.998487105" Jul 21 00:42:11.406878 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:11.406617 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-twlx9" podStartSLOduration=57.956522275 podStartE2EDuration="1m6.406598921s" totalImagesPullingTime="8.450076646s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:41:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:42:11.406293938 +0000 UTC m=+79.012402980" watchObservedRunningTime="2026-07-21 00:42:11.406598921 +0000 UTC m=+79.012707962" Jul 21 00:42:16.358395 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:16.358350 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5xx6h" event={"ID":"2d8da6d4-7c9a-42d9-ac48-581a8640f880","Type":"ContainerStarted","Data":"ffff5a02f1076f95bd719d3c206f4b4258b5cc11e3ad5517b32e6098df26b642"} Jul 21 00:42:16.378659 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:16.378610 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-5xx6h" podStartSLOduration=17.929129251 podStartE2EDuration="36.378596056s" totalImagesPullingTime="18.449466805s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 00:41:40 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:42:16.377665384 +0000 UTC m=+83.983774437" watchObservedRunningTime="2026-07-21 00:42:16.378596056 +0000 UTC m=+83.984705155" Jul 21 00:42:21.341827 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:21.341796 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-nlszd" Jul 21 00:42:25.302851 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:25.302817 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-bdd745b78-7cccc" Jul 21 00:42:27.844813 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:27.844771 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-z6c8m"] Jul 21 00:42:27.990199 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:27.990158 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-z6c8m" Jul 21 00:42:27.993057 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:27.993034 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 21 00:42:27.993057 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:27.993050 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 21 00:42:27.993303 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:27.993050 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 21 00:42:27.993303 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:27.993106 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-cx8t6\"" Jul 21 00:42:27.994274 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:27.994252 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 21 00:42:28.062594 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:28.062558 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/de793b0f-6ee5-4ed5-b11c-9850496fd84a-sys\") pod \"node-exporter-z6c8m\" (UID: \"de793b0f-6ee5-4ed5-b11c-9850496fd84a\") " pod="openshift-monitoring/node-exporter-z6c8m" Jul 21 00:42:28.062750 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:28.062604 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/de793b0f-6ee5-4ed5-b11c-9850496fd84a-node-exporter-tls\") pod \"node-exporter-z6c8m\" (UID: \"de793b0f-6ee5-4ed5-b11c-9850496fd84a\") " pod="openshift-monitoring/node-exporter-z6c8m" Jul 21 00:42:28.062750 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:28.062635 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/de793b0f-6ee5-4ed5-b11c-9850496fd84a-node-exporter-textfile\") pod \"node-exporter-z6c8m\" (UID: \"de793b0f-6ee5-4ed5-b11c-9850496fd84a\") " pod="openshift-monitoring/node-exporter-z6c8m" Jul 21 00:42:28.062750 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:28.062673 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/de793b0f-6ee5-4ed5-b11c-9850496fd84a-node-exporter-accelerators-collector-config\") pod \"node-exporter-z6c8m\" (UID: \"de793b0f-6ee5-4ed5-b11c-9850496fd84a\") " pod="openshift-monitoring/node-exporter-z6c8m" Jul 21 00:42:28.062750 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:28.062702 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/de793b0f-6ee5-4ed5-b11c-9850496fd84a-root\") pod \"node-exporter-z6c8m\" (UID: \"de793b0f-6ee5-4ed5-b11c-9850496fd84a\") " pod="openshift-monitoring/node-exporter-z6c8m" Jul 21 00:42:28.062925 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:28.062756 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/de793b0f-6ee5-4ed5-b11c-9850496fd84a-node-exporter-wtmp\") pod \"node-exporter-z6c8m\" (UID: \"de793b0f-6ee5-4ed5-b11c-9850496fd84a\") " pod="openshift-monitoring/node-exporter-z6c8m" Jul 21 00:42:28.062925 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:28.062781 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de793b0f-6ee5-4ed5-b11c-9850496fd84a-metrics-client-ca\") pod \"node-exporter-z6c8m\" (UID: \"de793b0f-6ee5-4ed5-b11c-9850496fd84a\") " pod="openshift-monitoring/node-exporter-z6c8m" Jul 21 00:42:28.062925 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:28.062820 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zd5z6\" (UniqueName: \"kubernetes.io/projected/de793b0f-6ee5-4ed5-b11c-9850496fd84a-kube-api-access-zd5z6\") pod \"node-exporter-z6c8m\" (UID: \"de793b0f-6ee5-4ed5-b11c-9850496fd84a\") " pod="openshift-monitoring/node-exporter-z6c8m" Jul 21 00:42:28.062925 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:28.062871 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/de793b0f-6ee5-4ed5-b11c-9850496fd84a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-z6c8m\" (UID: \"de793b0f-6ee5-4ed5-b11c-9850496fd84a\") " pod="openshift-monitoring/node-exporter-z6c8m" Jul 21 00:42:28.163665 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:28.163586 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/de793b0f-6ee5-4ed5-b11c-9850496fd84a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-z6c8m\" (UID: \"de793b0f-6ee5-4ed5-b11c-9850496fd84a\") " pod="openshift-monitoring/node-exporter-z6c8m" Jul 21 00:42:28.163665 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:28.163652 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/de793b0f-6ee5-4ed5-b11c-9850496fd84a-sys\") pod \"node-exporter-z6c8m\" (UID: \"de793b0f-6ee5-4ed5-b11c-9850496fd84a\") " pod="openshift-monitoring/node-exporter-z6c8m" Jul 21 00:42:28.163932 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:28.163682 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/de793b0f-6ee5-4ed5-b11c-9850496fd84a-node-exporter-tls\") pod \"node-exporter-z6c8m\" (UID: \"de793b0f-6ee5-4ed5-b11c-9850496fd84a\") " pod="openshift-monitoring/node-exporter-z6c8m" Jul 21 00:42:28.163932 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:28.163709 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/de793b0f-6ee5-4ed5-b11c-9850496fd84a-node-exporter-textfile\") pod \"node-exporter-z6c8m\" (UID: \"de793b0f-6ee5-4ed5-b11c-9850496fd84a\") " pod="openshift-monitoring/node-exporter-z6c8m" Jul 21 00:42:28.163932 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:28.163780 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/de793b0f-6ee5-4ed5-b11c-9850496fd84a-sys\") pod \"node-exporter-z6c8m\" (UID: \"de793b0f-6ee5-4ed5-b11c-9850496fd84a\") " pod="openshift-monitoring/node-exporter-z6c8m" Jul 21 00:42:28.163932 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:28.163832 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/de793b0f-6ee5-4ed5-b11c-9850496fd84a-node-exporter-accelerators-collector-config\") pod \"node-exporter-z6c8m\" (UID: \"de793b0f-6ee5-4ed5-b11c-9850496fd84a\") " pod="openshift-monitoring/node-exporter-z6c8m" Jul 21 00:42:28.163932 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:28.163865 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/de793b0f-6ee5-4ed5-b11c-9850496fd84a-root\") pod \"node-exporter-z6c8m\" (UID: \"de793b0f-6ee5-4ed5-b11c-9850496fd84a\") " pod="openshift-monitoring/node-exporter-z6c8m" Jul 21 00:42:28.163932 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:28.163923 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/de793b0f-6ee5-4ed5-b11c-9850496fd84a-node-exporter-wtmp\") pod \"node-exporter-z6c8m\" (UID: \"de793b0f-6ee5-4ed5-b11c-9850496fd84a\") " pod="openshift-monitoring/node-exporter-z6c8m" Jul 21 00:42:28.164221 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:28.163946 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de793b0f-6ee5-4ed5-b11c-9850496fd84a-metrics-client-ca\") pod \"node-exporter-z6c8m\" (UID: \"de793b0f-6ee5-4ed5-b11c-9850496fd84a\") " pod="openshift-monitoring/node-exporter-z6c8m" Jul 21 00:42:28.164221 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:28.163975 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zd5z6\" (UniqueName: \"kubernetes.io/projected/de793b0f-6ee5-4ed5-b11c-9850496fd84a-kube-api-access-zd5z6\") pod \"node-exporter-z6c8m\" (UID: \"de793b0f-6ee5-4ed5-b11c-9850496fd84a\") " pod="openshift-monitoring/node-exporter-z6c8m" Jul 21 00:42:28.164221 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:28.164062 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/de793b0f-6ee5-4ed5-b11c-9850496fd84a-root\") pod \"node-exporter-z6c8m\" (UID: \"de793b0f-6ee5-4ed5-b11c-9850496fd84a\") " pod="openshift-monitoring/node-exporter-z6c8m" Jul 21 00:42:28.164221 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:28.164076 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/de793b0f-6ee5-4ed5-b11c-9850496fd84a-node-exporter-textfile\") pod \"node-exporter-z6c8m\" (UID: \"de793b0f-6ee5-4ed5-b11c-9850496fd84a\") " pod="openshift-monitoring/node-exporter-z6c8m" Jul 21 00:42:28.164221 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:28.164134 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/de793b0f-6ee5-4ed5-b11c-9850496fd84a-node-exporter-wtmp\") pod \"node-exporter-z6c8m\" (UID: \"de793b0f-6ee5-4ed5-b11c-9850496fd84a\") " pod="openshift-monitoring/node-exporter-z6c8m" Jul 21 00:42:28.164653 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:28.164623 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/de793b0f-6ee5-4ed5-b11c-9850496fd84a-node-exporter-accelerators-collector-config\") pod \"node-exporter-z6c8m\" (UID: \"de793b0f-6ee5-4ed5-b11c-9850496fd84a\") " pod="openshift-monitoring/node-exporter-z6c8m" Jul 21 00:42:28.164798 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:28.164778 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de793b0f-6ee5-4ed5-b11c-9850496fd84a-metrics-client-ca\") pod \"node-exporter-z6c8m\" (UID: \"de793b0f-6ee5-4ed5-b11c-9850496fd84a\") " pod="openshift-monitoring/node-exporter-z6c8m" Jul 21 00:42:28.166260 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:28.166220 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/de793b0f-6ee5-4ed5-b11c-9850496fd84a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-z6c8m\" (UID: \"de793b0f-6ee5-4ed5-b11c-9850496fd84a\") " pod="openshift-monitoring/node-exporter-z6c8m" Jul 21 00:42:28.166368 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:28.166350 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/de793b0f-6ee5-4ed5-b11c-9850496fd84a-node-exporter-tls\") pod \"node-exporter-z6c8m\" (UID: \"de793b0f-6ee5-4ed5-b11c-9850496fd84a\") " pod="openshift-monitoring/node-exporter-z6c8m" Jul 21 00:42:28.171719 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:28.171698 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zd5z6\" (UniqueName: \"kubernetes.io/projected/de793b0f-6ee5-4ed5-b11c-9850496fd84a-kube-api-access-zd5z6\") pod \"node-exporter-z6c8m\" (UID: \"de793b0f-6ee5-4ed5-b11c-9850496fd84a\") " pod="openshift-monitoring/node-exporter-z6c8m" Jul 21 00:42:28.300336 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:28.300293 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-z6c8m" Jul 21 00:42:29.303032 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:42:29.302992 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde793b0f_6ee5_4ed5_b11c_9850496fd84a.slice/crio-b9e1cd620a89f26e455d1c00701675b05f69ad70b0c3d62b698f6d8c597e7102 WatchSource:0}: Error finding container b9e1cd620a89f26e455d1c00701675b05f69ad70b0c3d62b698f6d8c597e7102: Status 404 returned error can't find the container with id b9e1cd620a89f26e455d1c00701675b05f69ad70b0c3d62b698f6d8c597e7102 Jul 21 00:42:29.413190 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:29.413156 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-z6c8m" event={"ID":"de793b0f-6ee5-4ed5-b11c-9850496fd84a","Type":"ContainerStarted","Data":"b9e1cd620a89f26e455d1c00701675b05f69ad70b0c3d62b698f6d8c597e7102"} Jul 21 00:42:30.214417 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:30.214382 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-9r2ff" Jul 21 00:42:30.317797 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:30.317752 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-bdd745b78-7cccc" podUID="6543206b-bf43-43c5-930d-c3c8834b0f2c" containerName="registry" containerID="cri-o://04d7b35d2d264368e571479044ec2d30dc9116c489732bff4839cdcd9ecb42b7" gracePeriod=30 Jul 21 00:42:30.420120 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:30.420085 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-wnqtz" event={"ID":"d4c2ffde-2686-4b53-a5f8-03df48ab6b6f","Type":"ContainerStarted","Data":"923f18de2a37a799a7ff9e4ee2749e739b211005157015d41183cfe20fd65b73"} Jul 21 00:42:30.420434 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:30.420409 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-5bc67dcd46-wnqtz" Jul 21 00:42:30.431248 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:30.431212 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-5bc67dcd46-wnqtz" Jul 21 00:42:30.436100 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:30.436055 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-5bc67dcd46-wnqtz" podStartSLOduration=7.599090253 podStartE2EDuration="26.436035802s" totalImagesPullingTime="18.836945549s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:42:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:42:30.434836491 +0000 UTC m=+98.040945532" watchObservedRunningTime="2026-07-21 00:42:30.436035802 +0000 UTC m=+98.042144843" Jul 21 00:42:30.873930 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:30.873894 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-bdd745b78-7cccc" Jul 21 00:42:30.987704 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:30.987661 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6543206b-bf43-43c5-930d-c3c8834b0f2c-registry-tls\") pod \"6543206b-bf43-43c5-930d-c3c8834b0f2c\" (UID: \"6543206b-bf43-43c5-930d-c3c8834b0f2c\") " Jul 21 00:42:30.987889 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:30.987726 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6543206b-bf43-43c5-930d-c3c8834b0f2c-image-registry-private-configuration\") pod \"6543206b-bf43-43c5-930d-c3c8834b0f2c\" (UID: \"6543206b-bf43-43c5-930d-c3c8834b0f2c\") " Jul 21 00:42:30.987889 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:30.987760 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6543206b-bf43-43c5-930d-c3c8834b0f2c-bound-sa-token\") pod \"6543206b-bf43-43c5-930d-c3c8834b0f2c\" (UID: \"6543206b-bf43-43c5-930d-c3c8834b0f2c\") " Jul 21 00:42:30.988007 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:30.987920 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6543206b-bf43-43c5-930d-c3c8834b0f2c-installation-pull-secrets\") pod \"6543206b-bf43-43c5-930d-c3c8834b0f2c\" (UID: \"6543206b-bf43-43c5-930d-c3c8834b0f2c\") " Jul 21 00:42:30.988007 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:30.987982 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6543206b-bf43-43c5-930d-c3c8834b0f2c-ca-trust-extracted\") pod \"6543206b-bf43-43c5-930d-c3c8834b0f2c\" (UID: \"6543206b-bf43-43c5-930d-c3c8834b0f2c\") " Jul 21 00:42:30.988103 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:30.988031 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k2bj4\" (UniqueName: \"kubernetes.io/projected/6543206b-bf43-43c5-930d-c3c8834b0f2c-kube-api-access-k2bj4\") pod \"6543206b-bf43-43c5-930d-c3c8834b0f2c\" (UID: \"6543206b-bf43-43c5-930d-c3c8834b0f2c\") " Jul 21 00:42:30.988103 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:30.988085 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6543206b-bf43-43c5-930d-c3c8834b0f2c-registry-certificates\") pod \"6543206b-bf43-43c5-930d-c3c8834b0f2c\" (UID: \"6543206b-bf43-43c5-930d-c3c8834b0f2c\") " Jul 21 00:42:30.988199 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:30.988122 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6543206b-bf43-43c5-930d-c3c8834b0f2c-trusted-ca\") pod \"6543206b-bf43-43c5-930d-c3c8834b0f2c\" (UID: \"6543206b-bf43-43c5-930d-c3c8834b0f2c\") " Jul 21 00:42:30.988812 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:30.988737 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6543206b-bf43-43c5-930d-c3c8834b0f2c-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "6543206b-bf43-43c5-930d-c3c8834b0f2c" (UID: "6543206b-bf43-43c5-930d-c3c8834b0f2c"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 00:42:30.988812 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:30.988752 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6543206b-bf43-43c5-930d-c3c8834b0f2c-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "6543206b-bf43-43c5-930d-c3c8834b0f2c" (UID: "6543206b-bf43-43c5-930d-c3c8834b0f2c"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 00:42:30.991873 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:30.991837 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6543206b-bf43-43c5-930d-c3c8834b0f2c-kube-api-access-k2bj4" (OuterVolumeSpecName: "kube-api-access-k2bj4") pod "6543206b-bf43-43c5-930d-c3c8834b0f2c" (UID: "6543206b-bf43-43c5-930d-c3c8834b0f2c"). InnerVolumeSpecName "kube-api-access-k2bj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 00:42:30.992741 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:30.992719 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6543206b-bf43-43c5-930d-c3c8834b0f2c-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "6543206b-bf43-43c5-930d-c3c8834b0f2c" (UID: "6543206b-bf43-43c5-930d-c3c8834b0f2c"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 00:42:30.992846 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:30.992825 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6543206b-bf43-43c5-930d-c3c8834b0f2c-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "6543206b-bf43-43c5-930d-c3c8834b0f2c" (UID: "6543206b-bf43-43c5-930d-c3c8834b0f2c"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 00:42:30.993898 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:30.993866 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6543206b-bf43-43c5-930d-c3c8834b0f2c-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "6543206b-bf43-43c5-930d-c3c8834b0f2c" (UID: "6543206b-bf43-43c5-930d-c3c8834b0f2c"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 00:42:30.994024 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:30.993992 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6543206b-bf43-43c5-930d-c3c8834b0f2c-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "6543206b-bf43-43c5-930d-c3c8834b0f2c" (UID: "6543206b-bf43-43c5-930d-c3c8834b0f2c"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 00:42:30.999054 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:30.999019 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6543206b-bf43-43c5-930d-c3c8834b0f2c-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "6543206b-bf43-43c5-930d-c3c8834b0f2c" (UID: "6543206b-bf43-43c5-930d-c3c8834b0f2c"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 00:42:31.089719 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:31.089632 2580 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6543206b-bf43-43c5-930d-c3c8834b0f2c-registry-tls\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 21 00:42:31.089948 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:31.089928 2580 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6543206b-bf43-43c5-930d-c3c8834b0f2c-image-registry-private-configuration\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 21 00:42:31.090061 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:31.090044 2580 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6543206b-bf43-43c5-930d-c3c8834b0f2c-bound-sa-token\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 21 00:42:31.090153 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:31.090139 2580 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6543206b-bf43-43c5-930d-c3c8834b0f2c-installation-pull-secrets\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 21 00:42:31.090282 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:31.090266 2580 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6543206b-bf43-43c5-930d-c3c8834b0f2c-ca-trust-extracted\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 21 00:42:31.090396 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:31.090382 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2bj4\" (UniqueName: \"kubernetes.io/projected/6543206b-bf43-43c5-930d-c3c8834b0f2c-kube-api-access-k2bj4\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 21 00:42:31.090497 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:31.090482 2580 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6543206b-bf43-43c5-930d-c3c8834b0f2c-registry-certificates\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 21 00:42:31.090582 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:31.090571 2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6543206b-bf43-43c5-930d-c3c8834b0f2c-trusted-ca\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 21 00:42:31.426209 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:31.426118 2580 generic.go:356] "Generic (PLEG): container finished" podID="6543206b-bf43-43c5-930d-c3c8834b0f2c" containerID="04d7b35d2d264368e571479044ec2d30dc9116c489732bff4839cdcd9ecb42b7" exitCode=0 Jul 21 00:42:31.426209 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:31.426178 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-bdd745b78-7cccc" Jul 21 00:42:31.426714 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:31.426216 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-bdd745b78-7cccc" event={"ID":"6543206b-bf43-43c5-930d-c3c8834b0f2c","Type":"ContainerDied","Data":"04d7b35d2d264368e571479044ec2d30dc9116c489732bff4839cdcd9ecb42b7"} Jul 21 00:42:31.426714 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:31.426306 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-bdd745b78-7cccc" event={"ID":"6543206b-bf43-43c5-930d-c3c8834b0f2c","Type":"ContainerDied","Data":"f2b94f798b8354222140b152f142887c38778eb9bdaa4bd6707e8c0b010cb37d"} Jul 21 00:42:31.426714 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:31.426346 2580 scope.go:117] "RemoveContainer" containerID="04d7b35d2d264368e571479044ec2d30dc9116c489732bff4839cdcd9ecb42b7" Jul 21 00:42:31.432755 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:31.432337 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-z6c8m" event={"ID":"de793b0f-6ee5-4ed5-b11c-9850496fd84a","Type":"ContainerStarted","Data":"47e7692920e0cf74c87002e2286e3a73c6e7fbf514d527ff3cf817bb14a4d614"} Jul 21 00:42:31.438407 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:31.438388 2580 scope.go:117] "RemoveContainer" containerID="04d7b35d2d264368e571479044ec2d30dc9116c489732bff4839cdcd9ecb42b7" Jul 21 00:42:31.438703 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:42:31.438676 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04d7b35d2d264368e571479044ec2d30dc9116c489732bff4839cdcd9ecb42b7\": container with ID starting with 04d7b35d2d264368e571479044ec2d30dc9116c489732bff4839cdcd9ecb42b7 not found: ID does not exist" containerID="04d7b35d2d264368e571479044ec2d30dc9116c489732bff4839cdcd9ecb42b7" Jul 21 00:42:31.438788 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:31.438712 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04d7b35d2d264368e571479044ec2d30dc9116c489732bff4839cdcd9ecb42b7"} err="failed to get container status \"04d7b35d2d264368e571479044ec2d30dc9116c489732bff4839cdcd9ecb42b7\": rpc error: code = NotFound desc = could not find container \"04d7b35d2d264368e571479044ec2d30dc9116c489732bff4839cdcd9ecb42b7\": container with ID starting with 04d7b35d2d264368e571479044ec2d30dc9116c489732bff4839cdcd9ecb42b7 not found: ID does not exist" Jul 21 00:42:31.466441 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:31.466415 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-bdd745b78-7cccc"] Jul 21 00:42:31.470643 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:31.470616 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-bdd745b78-7cccc"] Jul 21 00:42:32.439401 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:32.439309 2580 generic.go:356] "Generic (PLEG): container finished" podID="de793b0f-6ee5-4ed5-b11c-9850496fd84a" containerID="47e7692920e0cf74c87002e2286e3a73c6e7fbf514d527ff3cf817bb14a4d614" exitCode=0 Jul 21 00:42:32.439924 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:32.439409 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-z6c8m" event={"ID":"de793b0f-6ee5-4ed5-b11c-9850496fd84a","Type":"ContainerDied","Data":"47e7692920e0cf74c87002e2286e3a73c6e7fbf514d527ff3cf817bb14a4d614"} Jul 21 00:42:32.911653 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:32.911613 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6543206b-bf43-43c5-930d-c3c8834b0f2c" path="/var/lib/kubelet/pods/6543206b-bf43-43c5-930d-c3c8834b0f2c/volumes" Jul 21 00:42:33.348329 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:33.348300 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5dbc6848f8-w729h" Jul 21 00:42:33.446491 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:33.446445 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-z6c8m" event={"ID":"de793b0f-6ee5-4ed5-b11c-9850496fd84a","Type":"ContainerStarted","Data":"038ba90f1299a33d6bbabc47ae468ca04dc3a79ec6e4e220a74038890c01e995"} Jul 21 00:42:33.446919 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:33.446518 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-z6c8m" event={"ID":"de793b0f-6ee5-4ed5-b11c-9850496fd84a","Type":"ContainerStarted","Data":"3e6418b82c3f3f705a5fd7fa5247c0292de6deabafb2be0359a94e2b073171b7"} Jul 21 00:42:33.465664 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:33.465604 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-z6c8m" podStartSLOduration=4.458268794 podStartE2EDuration="6.465581883s" totalImagesPullingTime="1.92088945s" totalInitContainerRuntime="86.423639ms" isStatefulPod=true podCreationTimestamp="2026-07-21 00:42:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:42:33.465183823 +0000 UTC m=+101.071292875" watchObservedRunningTime="2026-07-21 00:42:33.465581883 +0000 UTC m=+101.071690923" Jul 21 00:42:34.083692 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:34.083653 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 21 00:42:34.084014 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:34.083994 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6543206b-bf43-43c5-930d-c3c8834b0f2c" containerName="registry" Jul 21 00:42:34.084014 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:34.084012 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6543206b-bf43-43c5-930d-c3c8834b0f2c" containerName="registry" Jul 21 00:42:34.084165 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:34.084066 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6543206b-bf43-43c5-930d-c3c8834b0f2c" containerName="registry" Jul 21 00:42:34.125615 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:34.125579 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 21 00:42:34.125877 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:34.125742 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 00:42:34.128666 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:34.128644 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 21 00:42:34.129076 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:34.129050 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 21 00:42:34.129076 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:34.129072 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 21 00:42:34.129255 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:34.128673 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-58rq75l9905m6\"" Jul 21 00:42:34.129255 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:34.128679 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 21 00:42:34.129255 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:34.128711 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 21 00:42:34.129255 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:34.128878 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 21 00:42:34.129461 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:34.128916 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 21 00:42:34.129461 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:34.128989 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-chhps\"" Jul 21 00:42:34.129461 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:34.128644 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 21 00:42:34.129461 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:34.128870 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 21 00:42:34.129810 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:34.129792 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 21 00:42:34.129987 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:34.129823 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 21 00:42:34.130085 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:34.130004 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 21 00:42:34.130141 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:34.130106 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 21 00:42:34.133927 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:34.133906 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 21 00:42:34.138468 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:34.138448 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 21 00:42:34.220222 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:34.220177 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 00:42:34.220431 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:34.220254 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 00:42:34.220431 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:34.220281 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 00:42:34.220431 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:34.220311 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 00:42:34.220431 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:34.220338 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f-config\") pod \"prometheus-k8s-0\" (UID: \"5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 00:42:34.220663 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:34.220430 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 00:42:34.220663 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:34.220466 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 00:42:34.220663 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:34.220493 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 00:42:34.220663 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:34.220520 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 00:42:34.220663 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:34.220553 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 00:42:34.220663 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:34.220582 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 00:42:34.220663 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:34.220607 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 00:42:34.220663 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:34.220642 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 00:42:34.220911 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:34.220667 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5ss6\" (UniqueName: \"kubernetes.io/projected/5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f-kube-api-access-p5ss6\") pod \"prometheus-k8s-0\" (UID: \"5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 00:42:34.220911 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:34.220714 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f-config-out\") pod \"prometheus-k8s-0\" (UID: \"5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 00:42:34.220911 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:34.220754 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 00:42:34.220911 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:34.220788 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f-web-config\") pod \"prometheus-k8s-0\" (UID: \"5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 00:42:34.220911 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:34.220815 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 00:42:34.220911 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:34.220846 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 00:42:34.220911 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:34.220868 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 00:42:34.322061 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:34.322020 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 00:42:34.322061 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:34.322061 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f-config\") pod \"prometheus-k8s-0\" (UID: \"5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 00:42:34.322337 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:34.322096 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 00:42:34.322337 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:34.322122 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 00:42:34.322337 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:34.322148 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 00:42:34.322337 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:34.322185 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 00:42:34.322337 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:34.322223 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 00:42:34.322337 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:34.322267 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 00:42:34.322337 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:34.322290 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 00:42:34.322337 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:34.322322 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 00:42:34.322715 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:34.322347 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5ss6\" (UniqueName: \"kubernetes.io/projected/5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f-kube-api-access-p5ss6\") pod \"prometheus-k8s-0\" (UID: \"5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 00:42:34.322715 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:34.322373 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f-config-out\") pod \"prometheus-k8s-0\" (UID: \"5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 00:42:34.322715 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:34.322400 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 00:42:34.322715 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:34.322427 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f-web-config\") pod \"prometheus-k8s-0\" (UID: \"5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 00:42:34.322715 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:34.322453 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 00:42:34.322715 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:34.322479 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 00:42:34.322715 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:34.322509 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 00:42:34.322715 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:34.322583 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 00:42:34.323085 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:34.322605 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 00:42:34.323146 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:34.323112 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 00:42:34.323146 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:34.323133 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 00:42:34.324771 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:34.323284 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 00:42:34.324771 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:34.323523 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 00:42:34.324771 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:34.324261 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 00:42:34.324771 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:34.324470 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 00:42:34.324771 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:34.324761 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 00:42:34.326688 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:34.326660 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 00:42:34.327022 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:34.326709 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 00:42:34.327365 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:34.327274 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 00:42:34.327687 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:34.327665 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 00:42:34.328602 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:34.328561 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 00:42:34.329083 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:34.329058 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f-config\") pod \"prometheus-k8s-0\" (UID: \"5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 00:42:34.329188 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:34.329171 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f-web-config\") pod \"prometheus-k8s-0\" (UID: \"5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 00:42:34.329303 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:34.329277 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f-config-out\") pod \"prometheus-k8s-0\" (UID: \"5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 00:42:34.329727 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:34.329700 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 00:42:34.329853 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:34.329827 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 00:42:34.330072 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:34.330014 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 00:42:34.330508 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:34.330488 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 00:42:34.330973 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:34.330940 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 00:42:34.333767 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:34.333696 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5ss6\" (UniqueName: \"kubernetes.io/projected/5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f-kube-api-access-p5ss6\") pod \"prometheus-k8s-0\" (UID: \"5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 00:42:34.438759 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:34.438712 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 00:42:34.639180 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:34.639105 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 21 00:42:34.642397 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:42:34.642367 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f8dde4a_ce95_4b0d_80d5_c7c26fa3a41f.slice/crio-dbbe63eba61563009304a8f2b798ebbd476ac2ecade7dd08fddafc3b3144f019 WatchSource:0}: Error finding container dbbe63eba61563009304a8f2b798ebbd476ac2ecade7dd08fddafc3b3144f019: Status 404 returned error can't find the container with id dbbe63eba61563009304a8f2b798ebbd476ac2ecade7dd08fddafc3b3144f019 Jul 21 00:42:35.456201 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:35.456157 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f","Type":"ContainerStarted","Data":"dbbe63eba61563009304a8f2b798ebbd476ac2ecade7dd08fddafc3b3144f019"} Jul 21 00:42:36.462176 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:36.462139 2580 generic.go:356] "Generic (PLEG): container finished" podID="5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f" containerID="45479a624c9f9b0f35964ff91eff4d0b9c339e1271fbcbc25481796f320dd1aa" exitCode=0 Jul 21 00:42:36.462670 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:36.462245 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f","Type":"ContainerDied","Data":"45479a624c9f9b0f35964ff91eff4d0b9c339e1271fbcbc25481796f320dd1aa"} Jul 21 00:42:41.486246 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:41.486193 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f","Type":"ContainerStarted","Data":"d49dc5dbf85d836200656488a25187eb4d091b82d02a0d21d73838c609031c69"} Jul 21 00:42:41.486681 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:41.486296 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f","Type":"ContainerStarted","Data":"3cea979b91b64dab0141fae4ade55c50cc9925c78dd43dee6537af8d3aad6512"} Jul 21 00:42:44.502275 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:44.502225 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f","Type":"ContainerStarted","Data":"d7bc3b8f48db1f2ba8f93b9271aec0a1237a09e1825b4e57255a6628b94869c4"} Jul 21 00:42:44.502630 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:44.502336 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f","Type":"ContainerStarted","Data":"f5b020eb453c0606fd77624ecb6f4aa571b89907e518ba474439de65d0dc01a9"} Jul 21 00:42:44.502630 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:44.502385 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f","Type":"ContainerStarted","Data":"285d333792da9b90e400ea054ed625464a813d491a6650b8a0c6bc10d9b0dc4f"} Jul 21 00:42:44.502630 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:44.502408 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f","Type":"ContainerStarted","Data":"ca8d94de8e0708d5d505fec1e7c75584db3974067a323483aed8e0ad117399a6"} Jul 21 00:42:44.529055 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:44.528997 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.235737158 podStartE2EDuration="10.528976305s" totalImagesPullingTime="8.277752493s" totalInitContainerRuntime="15.486654ms" isStatefulPod=false podCreationTimestamp="2026-07-21 00:42:34 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:42:44.527382304 +0000 UTC m=+112.133491344" watchObservedRunningTime="2026-07-21 00:42:44.528976305 +0000 UTC m=+112.135085346" Jul 21 00:42:45.507403 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:45.507365 2580 generic.go:356] "Generic (PLEG): container finished" podID="f7fa2eb6-5645-47f1-8199-1dd64ea512dc" containerID="81eb9dcac9118a97418b3bd9352bdc026dd055ef18e01faebf499006acb0804d" exitCode=0 Jul 21 00:42:45.507823 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:45.507443 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-tdb28" event={"ID":"f7fa2eb6-5645-47f1-8199-1dd64ea512dc","Type":"ContainerDied","Data":"81eb9dcac9118a97418b3bd9352bdc026dd055ef18e01faebf499006acb0804d"} Jul 21 00:42:45.507869 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:45.507824 2580 scope.go:117] "RemoveContainer" containerID="81eb9dcac9118a97418b3bd9352bdc026dd055ef18e01faebf499006acb0804d" Jul 21 00:42:46.514954 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:46.514916 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-tdb28" event={"ID":"f7fa2eb6-5645-47f1-8199-1dd64ea512dc","Type":"ContainerStarted","Data":"aa9cdef25eb86324a81a66a257182c230cb7ff617ea6a366a8d8a41fa194c1aa"} Jul 21 00:42:49.439144 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:49.439103 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 00:42:54.544305 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:54.544266 2580 generic.go:356] "Generic (PLEG): container finished" podID="2d480ddf-1712-4f08-a5ef-98d3627df334" containerID="d39b695ff8792cf12c06a37fd2f9cd45958c679e53963333cb8d472d277c80c7" exitCode=0 Jul 21 00:42:54.544690 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:54.544341 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-r8lcd" event={"ID":"2d480ddf-1712-4f08-a5ef-98d3627df334","Type":"ContainerDied","Data":"d39b695ff8792cf12c06a37fd2f9cd45958c679e53963333cb8d472d277c80c7"} Jul 21 00:42:54.544690 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:54.544652 2580 scope.go:117] "RemoveContainer" containerID="d39b695ff8792cf12c06a37fd2f9cd45958c679e53963333cb8d472d277c80c7" Jul 21 00:42:55.549888 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:42:55.549835 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-r8lcd" event={"ID":"2d480ddf-1712-4f08-a5ef-98d3627df334","Type":"ContainerStarted","Data":"f2fc2da3d547a485e1f0a72d22d0198f83e7c04b952dc60b6e014ee3fcc64d88"} Jul 21 00:43:09.602045 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:43:09.602012 2580 generic.go:356] "Generic (PLEG): container finished" podID="826d0b75-8b78-4bf5-9a54-d71162f8e0da" containerID="37a0a59066a0159cd3fcbe80b860f141d42c2910670b641c2e18fda804c2c6f9" exitCode=0 Jul 21 00:43:09.602447 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:43:09.602084 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-9rqvh" event={"ID":"826d0b75-8b78-4bf5-9a54-d71162f8e0da","Type":"ContainerDied","Data":"37a0a59066a0159cd3fcbe80b860f141d42c2910670b641c2e18fda804c2c6f9"} Jul 21 00:43:09.602447 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:43:09.602433 2580 scope.go:117] "RemoveContainer" containerID="37a0a59066a0159cd3fcbe80b860f141d42c2910670b641c2e18fda804c2c6f9" Jul 21 00:43:10.608027 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:43:10.607995 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-9rqvh" event={"ID":"826d0b75-8b78-4bf5-9a54-d71162f8e0da","Type":"ContainerStarted","Data":"0ba996677fdfce42958b9f55447a3f18defb366c84dc0d514b401bc9604e8563"} Jul 21 00:43:34.439423 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:43:34.439390 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 00:43:34.454679 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:43:34.454656 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 00:43:34.711466 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:43:34.711393 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 00:45:52.733176 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:45:52.733145 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-rb4ph_3ee1dbf0-b4ff-4871-9432-284eb25f97b1/console-operator/1.log" Jul 21 00:45:52.739378 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:45:52.739356 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-rb4ph_3ee1dbf0-b4ff-4871-9432-284eb25f97b1/console-operator/1.log" Jul 21 00:45:52.743068 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:45:52.743050 2580 kubelet.go:1648] "Image garbage collection succeeded" Jul 21 00:46:47.219194 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:46:47.219115 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5f889c5d8-szrmh"] Jul 21 00:46:47.221441 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:46:47.221425 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5f889c5d8-szrmh" Jul 21 00:46:47.224276 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:46:47.224257 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 21 00:46:47.224706 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:46:47.224687 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 21 00:46:47.224796 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:46:47.224752 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 21 00:46:47.225082 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:46:47.225060 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-j5g8w\"" Jul 21 00:46:47.232182 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:46:47.232165 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 21 00:46:47.261726 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:46:47.261706 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5f889c5d8-szrmh"] Jul 21 00:46:47.284459 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:46:47.284433 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/dec0567c-68f4-474f-862e-bf50cb9e0af5-manifests\") pod \"opendatahub-operator-controller-manager-5f889c5d8-szrmh\" (UID: \"dec0567c-68f4-474f-862e-bf50cb9e0af5\") " pod="opendatahub/opendatahub-operator-controller-manager-5f889c5d8-szrmh" Jul 21 00:46:47.284546 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:46:47.284467 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dec0567c-68f4-474f-862e-bf50cb9e0af5-webhook-cert\") pod \"opendatahub-operator-controller-manager-5f889c5d8-szrmh\" (UID: \"dec0567c-68f4-474f-862e-bf50cb9e0af5\") " pod="opendatahub/opendatahub-operator-controller-manager-5f889c5d8-szrmh" Jul 21 00:46:47.284546 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:46:47.284514 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7sq6\" (UniqueName: \"kubernetes.io/projected/dec0567c-68f4-474f-862e-bf50cb9e0af5-kube-api-access-h7sq6\") pod \"opendatahub-operator-controller-manager-5f889c5d8-szrmh\" (UID: \"dec0567c-68f4-474f-862e-bf50cb9e0af5\") " pod="opendatahub/opendatahub-operator-controller-manager-5f889c5d8-szrmh" Jul 21 00:46:47.284546 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:46:47.284537 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/dec0567c-68f4-474f-862e-bf50cb9e0af5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5f889c5d8-szrmh\" (UID: \"dec0567c-68f4-474f-862e-bf50cb9e0af5\") " pod="opendatahub/opendatahub-operator-controller-manager-5f889c5d8-szrmh" Jul 21 00:46:47.284661 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:46:47.284578 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dec0567c-68f4-474f-862e-bf50cb9e0af5-tmp\") pod \"opendatahub-operator-controller-manager-5f889c5d8-szrmh\" (UID: \"dec0567c-68f4-474f-862e-bf50cb9e0af5\") " pod="opendatahub/opendatahub-operator-controller-manager-5f889c5d8-szrmh" Jul 21 00:46:47.385356 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:46:47.385324 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7sq6\" (UniqueName: \"kubernetes.io/projected/dec0567c-68f4-474f-862e-bf50cb9e0af5-kube-api-access-h7sq6\") pod \"opendatahub-operator-controller-manager-5f889c5d8-szrmh\" (UID: \"dec0567c-68f4-474f-862e-bf50cb9e0af5\") " pod="opendatahub/opendatahub-operator-controller-manager-5f889c5d8-szrmh" Jul 21 00:46:47.385534 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:46:47.385364 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/dec0567c-68f4-474f-862e-bf50cb9e0af5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5f889c5d8-szrmh\" (UID: \"dec0567c-68f4-474f-862e-bf50cb9e0af5\") " pod="opendatahub/opendatahub-operator-controller-manager-5f889c5d8-szrmh" Jul 21 00:46:47.385534 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:46:47.385412 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dec0567c-68f4-474f-862e-bf50cb9e0af5-tmp\") pod \"opendatahub-operator-controller-manager-5f889c5d8-szrmh\" (UID: \"dec0567c-68f4-474f-862e-bf50cb9e0af5\") " pod="opendatahub/opendatahub-operator-controller-manager-5f889c5d8-szrmh" Jul 21 00:46:47.385534 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:46:47.385435 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/dec0567c-68f4-474f-862e-bf50cb9e0af5-manifests\") pod \"opendatahub-operator-controller-manager-5f889c5d8-szrmh\" (UID: \"dec0567c-68f4-474f-862e-bf50cb9e0af5\") " pod="opendatahub/opendatahub-operator-controller-manager-5f889c5d8-szrmh" Jul 21 00:46:47.385534 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:46:47.385458 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dec0567c-68f4-474f-862e-bf50cb9e0af5-webhook-cert\") pod \"opendatahub-operator-controller-manager-5f889c5d8-szrmh\" (UID: \"dec0567c-68f4-474f-862e-bf50cb9e0af5\") " pod="opendatahub/opendatahub-operator-controller-manager-5f889c5d8-szrmh" Jul 21 00:46:47.385971 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:46:47.385947 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/dec0567c-68f4-474f-862e-bf50cb9e0af5-manifests\") pod \"opendatahub-operator-controller-manager-5f889c5d8-szrmh\" (UID: \"dec0567c-68f4-474f-862e-bf50cb9e0af5\") " pod="opendatahub/opendatahub-operator-controller-manager-5f889c5d8-szrmh" Jul 21 00:46:47.387912 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:46:47.387887 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dec0567c-68f4-474f-862e-bf50cb9e0af5-tmp\") pod \"opendatahub-operator-controller-manager-5f889c5d8-szrmh\" (UID: \"dec0567c-68f4-474f-862e-bf50cb9e0af5\") " pod="opendatahub/opendatahub-operator-controller-manager-5f889c5d8-szrmh" Jul 21 00:46:47.388020 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:46:47.387909 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/dec0567c-68f4-474f-862e-bf50cb9e0af5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5f889c5d8-szrmh\" (UID: \"dec0567c-68f4-474f-862e-bf50cb9e0af5\") " pod="opendatahub/opendatahub-operator-controller-manager-5f889c5d8-szrmh" Jul 21 00:46:47.388196 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:46:47.388177 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dec0567c-68f4-474f-862e-bf50cb9e0af5-webhook-cert\") pod \"opendatahub-operator-controller-manager-5f889c5d8-szrmh\" (UID: \"dec0567c-68f4-474f-862e-bf50cb9e0af5\") " pod="opendatahub/opendatahub-operator-controller-manager-5f889c5d8-szrmh" Jul 21 00:46:47.395345 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:46:47.395321 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7sq6\" (UniqueName: \"kubernetes.io/projected/dec0567c-68f4-474f-862e-bf50cb9e0af5-kube-api-access-h7sq6\") pod \"opendatahub-operator-controller-manager-5f889c5d8-szrmh\" (UID: \"dec0567c-68f4-474f-862e-bf50cb9e0af5\") " pod="opendatahub/opendatahub-operator-controller-manager-5f889c5d8-szrmh" Jul 21 00:46:47.532134 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:46:47.532057 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5f889c5d8-szrmh" Jul 21 00:46:47.663202 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:46:47.663176 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5f889c5d8-szrmh"] Jul 21 00:46:47.665620 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:46:47.665592 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddec0567c_68f4_474f_862e_bf50cb9e0af5.slice/crio-8158c0709fd7a840026ec43ef0dbd4e1a32c1e03199246a0d7f6097689d6141d WatchSource:0}: Error finding container 8158c0709fd7a840026ec43ef0dbd4e1a32c1e03199246a0d7f6097689d6141d: Status 404 returned error can't find the container with id 8158c0709fd7a840026ec43ef0dbd4e1a32c1e03199246a0d7f6097689d6141d Jul 21 00:46:47.667867 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:46:47.667848 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 00:46:48.417429 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:46:48.417345 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5f889c5d8-szrmh" event={"ID":"dec0567c-68f4-474f-862e-bf50cb9e0af5","Type":"ContainerStarted","Data":"8158c0709fd7a840026ec43ef0dbd4e1a32c1e03199246a0d7f6097689d6141d"} Jul 21 00:46:51.433679 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:46:51.433647 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5f889c5d8-szrmh" event={"ID":"dec0567c-68f4-474f-862e-bf50cb9e0af5","Type":"ContainerStarted","Data":"8e69f1802b589e8f27b34dd58102eff319e6e7f5cc74d98a08e66f69a0256d07"} Jul 21 00:46:52.439797 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:46:52.439758 2580 generic.go:356] "Generic (PLEG): container finished" podID="dec0567c-68f4-474f-862e-bf50cb9e0af5" containerID="8e69f1802b589e8f27b34dd58102eff319e6e7f5cc74d98a08e66f69a0256d07" exitCode=0 Jul 21 00:46:52.440200 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:46:52.439845 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5f889c5d8-szrmh" event={"ID":"dec0567c-68f4-474f-862e-bf50cb9e0af5","Type":"ContainerDied","Data":"8e69f1802b589e8f27b34dd58102eff319e6e7f5cc74d98a08e66f69a0256d07"} Jul 21 00:46:53.446549 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:46:53.446514 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5f889c5d8-szrmh" event={"ID":"dec0567c-68f4-474f-862e-bf50cb9e0af5","Type":"ContainerStarted","Data":"a66216110bb06a30e82e87f97c6622d9581a5ccd6ce069c9cc82f2585c87c749"} Jul 21 00:46:53.446992 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:46:53.446620 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5f889c5d8-szrmh" Jul 21 00:46:53.468392 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:46:53.468345 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5f889c5d8-szrmh" podStartSLOduration=1.851402403 podStartE2EDuration="6.468329401s" totalImagesPullingTime="3.583747319s" totalInitContainerRuntime="1.033179679s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:46:47 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:46:53.466177691 +0000 UTC m=+361.072286731" watchObservedRunningTime="2026-07-21 00:46:53.468329401 +0000 UTC m=+361.074438440" Jul 21 00:47:04.453028 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:04.452998 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5f889c5d8-szrmh" Jul 21 00:47:07.031063 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:07.031022 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5c7494f749-46ckj"] Jul 21 00:47:07.040522 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:07.040492 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5c7494f749-46ckj" Jul 21 00:47:07.043840 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:07.043818 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-g72pk\"" Jul 21 00:47:07.043969 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:07.043819 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 21 00:47:07.045326 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:07.045300 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 21 00:47:07.047098 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:07.047076 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5c7494f749-46ckj"] Jul 21 00:47:07.153797 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:07.153764 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkztf\" (UniqueName: \"kubernetes.io/projected/8d95153d-b67d-4c35-81cb-7dc970ac6ba6-kube-api-access-wkztf\") pod \"kube-auth-proxy-5c7494f749-46ckj\" (UID: \"8d95153d-b67d-4c35-81cb-7dc970ac6ba6\") " pod="openshift-ingress/kube-auth-proxy-5c7494f749-46ckj" Jul 21 00:47:07.153968 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:07.153803 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d95153d-b67d-4c35-81cb-7dc970ac6ba6-tmp\") pod \"kube-auth-proxy-5c7494f749-46ckj\" (UID: \"8d95153d-b67d-4c35-81cb-7dc970ac6ba6\") " pod="openshift-ingress/kube-auth-proxy-5c7494f749-46ckj" Jul 21 00:47:07.153968 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:07.153837 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d95153d-b67d-4c35-81cb-7dc970ac6ba6-tls-certs\") pod \"kube-auth-proxy-5c7494f749-46ckj\" (UID: \"8d95153d-b67d-4c35-81cb-7dc970ac6ba6\") " pod="openshift-ingress/kube-auth-proxy-5c7494f749-46ckj" Jul 21 00:47:07.254537 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:07.254496 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkztf\" (UniqueName: \"kubernetes.io/projected/8d95153d-b67d-4c35-81cb-7dc970ac6ba6-kube-api-access-wkztf\") pod \"kube-auth-proxy-5c7494f749-46ckj\" (UID: \"8d95153d-b67d-4c35-81cb-7dc970ac6ba6\") " pod="openshift-ingress/kube-auth-proxy-5c7494f749-46ckj" Jul 21 00:47:07.254537 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:07.254538 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d95153d-b67d-4c35-81cb-7dc970ac6ba6-tmp\") pod \"kube-auth-proxy-5c7494f749-46ckj\" (UID: \"8d95153d-b67d-4c35-81cb-7dc970ac6ba6\") " pod="openshift-ingress/kube-auth-proxy-5c7494f749-46ckj" Jul 21 00:47:07.254766 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:07.254574 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d95153d-b67d-4c35-81cb-7dc970ac6ba6-tls-certs\") pod \"kube-auth-proxy-5c7494f749-46ckj\" (UID: \"8d95153d-b67d-4c35-81cb-7dc970ac6ba6\") " pod="openshift-ingress/kube-auth-proxy-5c7494f749-46ckj" Jul 21 00:47:07.256916 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:07.256884 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d95153d-b67d-4c35-81cb-7dc970ac6ba6-tmp\") pod \"kube-auth-proxy-5c7494f749-46ckj\" (UID: \"8d95153d-b67d-4c35-81cb-7dc970ac6ba6\") " pod="openshift-ingress/kube-auth-proxy-5c7494f749-46ckj" Jul 21 00:47:07.257094 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:07.257078 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d95153d-b67d-4c35-81cb-7dc970ac6ba6-tls-certs\") pod \"kube-auth-proxy-5c7494f749-46ckj\" (UID: \"8d95153d-b67d-4c35-81cb-7dc970ac6ba6\") " pod="openshift-ingress/kube-auth-proxy-5c7494f749-46ckj" Jul 21 00:47:07.263659 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:07.263636 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkztf\" (UniqueName: \"kubernetes.io/projected/8d95153d-b67d-4c35-81cb-7dc970ac6ba6-kube-api-access-wkztf\") pod \"kube-auth-proxy-5c7494f749-46ckj\" (UID: \"8d95153d-b67d-4c35-81cb-7dc970ac6ba6\") " pod="openshift-ingress/kube-auth-proxy-5c7494f749-46ckj" Jul 21 00:47:07.351332 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:07.351294 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5c7494f749-46ckj" Jul 21 00:47:07.416666 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:07.415019 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-zmq6v"] Jul 21 00:47:07.419076 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:07.419003 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-zmq6v" Jul 21 00:47:07.422207 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:07.422171 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-65xdr\"" Jul 21 00:47:07.422345 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:07.422177 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 21 00:47:07.430712 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:07.430687 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-zmq6v"] Jul 21 00:47:07.512523 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:07.512498 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5c7494f749-46ckj"] Jul 21 00:47:07.515302 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:47:07.515267 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d95153d_b67d_4c35_81cb_7dc970ac6ba6.slice/crio-07ce8106ce0ad07e6a7d7a2430016601fbb8cd7bc80abdf09f17b5b9db80d5fa WatchSource:0}: Error finding container 07ce8106ce0ad07e6a7d7a2430016601fbb8cd7bc80abdf09f17b5b9db80d5fa: Status 404 returned error can't find the container with id 07ce8106ce0ad07e6a7d7a2430016601fbb8cd7bc80abdf09f17b5b9db80d5fa Jul 21 00:47:07.557919 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:07.557886 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jshwx\" (UniqueName: \"kubernetes.io/projected/115ede03-1e4e-432b-a874-579f0fa66f73-kube-api-access-jshwx\") pod \"odh-model-controller-7998c67dc4-zmq6v\" (UID: \"115ede03-1e4e-432b-a874-579f0fa66f73\") " pod="opendatahub/odh-model-controller-7998c67dc4-zmq6v" Jul 21 00:47:07.558058 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:07.557934 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/115ede03-1e4e-432b-a874-579f0fa66f73-cert\") pod \"odh-model-controller-7998c67dc4-zmq6v\" (UID: \"115ede03-1e4e-432b-a874-579f0fa66f73\") " pod="opendatahub/odh-model-controller-7998c67dc4-zmq6v" Jul 21 00:47:07.658814 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:07.658730 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jshwx\" (UniqueName: \"kubernetes.io/projected/115ede03-1e4e-432b-a874-579f0fa66f73-kube-api-access-jshwx\") pod \"odh-model-controller-7998c67dc4-zmq6v\" (UID: \"115ede03-1e4e-432b-a874-579f0fa66f73\") " pod="opendatahub/odh-model-controller-7998c67dc4-zmq6v" Jul 21 00:47:07.658814 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:07.658785 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/115ede03-1e4e-432b-a874-579f0fa66f73-cert\") pod \"odh-model-controller-7998c67dc4-zmq6v\" (UID: \"115ede03-1e4e-432b-a874-579f0fa66f73\") " pod="opendatahub/odh-model-controller-7998c67dc4-zmq6v" Jul 21 00:47:07.658984 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:47:07.658898 2580 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 21 00:47:07.658984 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:47:07.658962 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/115ede03-1e4e-432b-a874-579f0fa66f73-cert podName:115ede03-1e4e-432b-a874-579f0fa66f73 nodeName:}" failed. No retries permitted until 2026-07-21 00:47:08.158942974 +0000 UTC m=+375.765052008 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/115ede03-1e4e-432b-a874-579f0fa66f73-cert") pod "odh-model-controller-7998c67dc4-zmq6v" (UID: "115ede03-1e4e-432b-a874-579f0fa66f73") : secret "odh-model-controller-webhook-cert" not found Jul 21 00:47:07.676071 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:07.676046 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jshwx\" (UniqueName: \"kubernetes.io/projected/115ede03-1e4e-432b-a874-579f0fa66f73-kube-api-access-jshwx\") pod \"odh-model-controller-7998c67dc4-zmq6v\" (UID: \"115ede03-1e4e-432b-a874-579f0fa66f73\") " pod="opendatahub/odh-model-controller-7998c67dc4-zmq6v" Jul 21 00:47:08.164475 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:08.164438 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/115ede03-1e4e-432b-a874-579f0fa66f73-cert\") pod \"odh-model-controller-7998c67dc4-zmq6v\" (UID: \"115ede03-1e4e-432b-a874-579f0fa66f73\") " pod="opendatahub/odh-model-controller-7998c67dc4-zmq6v" Jul 21 00:47:08.166818 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:08.166792 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/115ede03-1e4e-432b-a874-579f0fa66f73-cert\") pod \"odh-model-controller-7998c67dc4-zmq6v\" (UID: \"115ede03-1e4e-432b-a874-579f0fa66f73\") " pod="opendatahub/odh-model-controller-7998c67dc4-zmq6v" Jul 21 00:47:08.330764 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:08.330729 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-zmq6v" Jul 21 00:47:08.335383 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:08.335354 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-vr6pj"] Jul 21 00:47:08.338474 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:08.338451 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-vr6pj" Jul 21 00:47:08.341365 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:08.341177 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 21 00:47:08.352294 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:08.352186 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-vr6pj"] Jul 21 00:47:08.466910 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:08.466866 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9a7a22c-1842-4eca-9b2d-4e3509bcf9f1-tls-certs\") pod \"model-serving-api-54d89bd5fc-vr6pj\" (UID: \"d9a7a22c-1842-4eca-9b2d-4e3509bcf9f1\") " pod="opendatahub/model-serving-api-54d89bd5fc-vr6pj" Jul 21 00:47:08.467081 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:08.466922 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sth9n\" (UniqueName: \"kubernetes.io/projected/d9a7a22c-1842-4eca-9b2d-4e3509bcf9f1-kube-api-access-sth9n\") pod \"model-serving-api-54d89bd5fc-vr6pj\" (UID: \"d9a7a22c-1842-4eca-9b2d-4e3509bcf9f1\") " pod="opendatahub/model-serving-api-54d89bd5fc-vr6pj" Jul 21 00:47:08.504846 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:08.504784 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-zmq6v"] Jul 21 00:47:08.506429 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:47:08.506397 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod115ede03_1e4e_432b_a874_579f0fa66f73.slice/crio-c210ec850ef65c2042c9500e57821e5e48ca9ee8ee14c132a37aa86b2aa8d724 WatchSource:0}: Error finding container c210ec850ef65c2042c9500e57821e5e48ca9ee8ee14c132a37aa86b2aa8d724: Status 404 returned error can't find the container with id c210ec850ef65c2042c9500e57821e5e48ca9ee8ee14c132a37aa86b2aa8d724 Jul 21 00:47:08.510885 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:08.510856 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5c7494f749-46ckj" event={"ID":"8d95153d-b67d-4c35-81cb-7dc970ac6ba6","Type":"ContainerStarted","Data":"07ce8106ce0ad07e6a7d7a2430016601fbb8cd7bc80abdf09f17b5b9db80d5fa"} Jul 21 00:47:08.568051 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:08.568019 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9a7a22c-1842-4eca-9b2d-4e3509bcf9f1-tls-certs\") pod \"model-serving-api-54d89bd5fc-vr6pj\" (UID: \"d9a7a22c-1842-4eca-9b2d-4e3509bcf9f1\") " pod="opendatahub/model-serving-api-54d89bd5fc-vr6pj" Jul 21 00:47:08.568252 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:08.568060 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sth9n\" (UniqueName: \"kubernetes.io/projected/d9a7a22c-1842-4eca-9b2d-4e3509bcf9f1-kube-api-access-sth9n\") pod \"model-serving-api-54d89bd5fc-vr6pj\" (UID: \"d9a7a22c-1842-4eca-9b2d-4e3509bcf9f1\") " pod="opendatahub/model-serving-api-54d89bd5fc-vr6pj" Jul 21 00:47:08.568331 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:47:08.568278 2580 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Jul 21 00:47:08.568393 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:47:08.568343 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9a7a22c-1842-4eca-9b2d-4e3509bcf9f1-tls-certs podName:d9a7a22c-1842-4eca-9b2d-4e3509bcf9f1 nodeName:}" failed. No retries permitted until 2026-07-21 00:47:09.068324085 +0000 UTC m=+376.674433103 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/d9a7a22c-1842-4eca-9b2d-4e3509bcf9f1-tls-certs") pod "model-serving-api-54d89bd5fc-vr6pj" (UID: "d9a7a22c-1842-4eca-9b2d-4e3509bcf9f1") : secret "model-serving-api-tls" not found Jul 21 00:47:08.578964 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:08.578940 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sth9n\" (UniqueName: \"kubernetes.io/projected/d9a7a22c-1842-4eca-9b2d-4e3509bcf9f1-kube-api-access-sth9n\") pod \"model-serving-api-54d89bd5fc-vr6pj\" (UID: \"d9a7a22c-1842-4eca-9b2d-4e3509bcf9f1\") " pod="opendatahub/model-serving-api-54d89bd5fc-vr6pj" Jul 21 00:47:08.801539 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:08.801461 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-trzbg"] Jul 21 00:47:08.805485 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:08.805467 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-trzbg" Jul 21 00:47:08.808078 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:08.808048 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 21 00:47:08.808203 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:08.808159 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 21 00:47:08.808203 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:08.808172 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 21 00:47:08.808325 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:08.808219 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 21 00:47:08.808325 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:08.808223 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-9r62s\"" Jul 21 00:47:08.823433 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:08.823344 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-trzbg"] Jul 21 00:47:08.972503 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:08.972462 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5-istio-token\") pod \"istiod-openshift-gateway-7574459656-trzbg\" (UID: \"36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-trzbg" Jul 21 00:47:08.972503 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:08.972500 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-trzbg\" (UID: \"36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-trzbg" Jul 21 00:47:08.972737 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:08.972528 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-trzbg\" (UID: \"36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-trzbg" Jul 21 00:47:08.972737 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:08.972553 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfsmr\" (UniqueName: \"kubernetes.io/projected/36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5-kube-api-access-wfsmr\") pod \"istiod-openshift-gateway-7574459656-trzbg\" (UID: \"36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-trzbg" Jul 21 00:47:08.972737 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:08.972599 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5-local-certs\") pod \"istiod-openshift-gateway-7574459656-trzbg\" (UID: \"36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-trzbg" Jul 21 00:47:08.972737 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:08.972659 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-trzbg\" (UID: \"36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-trzbg" Jul 21 00:47:08.972737 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:08.972726 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5-cacerts\") pod \"istiod-openshift-gateway-7574459656-trzbg\" (UID: \"36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-trzbg" Jul 21 00:47:09.074209 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:09.074164 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-trzbg\" (UID: \"36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-trzbg" Jul 21 00:47:09.074413 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:09.074259 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfsmr\" (UniqueName: \"kubernetes.io/projected/36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5-kube-api-access-wfsmr\") pod \"istiod-openshift-gateway-7574459656-trzbg\" (UID: \"36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-trzbg" Jul 21 00:47:09.074413 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:09.074336 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5-local-certs\") pod \"istiod-openshift-gateway-7574459656-trzbg\" (UID: \"36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-trzbg" Jul 21 00:47:09.074413 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:09.074370 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-trzbg\" (UID: \"36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-trzbg" Jul 21 00:47:09.074571 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:09.074416 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5-cacerts\") pod \"istiod-openshift-gateway-7574459656-trzbg\" (UID: \"36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-trzbg" Jul 21 00:47:09.074571 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:09.074464 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9a7a22c-1842-4eca-9b2d-4e3509bcf9f1-tls-certs\") pod \"model-serving-api-54d89bd5fc-vr6pj\" (UID: \"d9a7a22c-1842-4eca-9b2d-4e3509bcf9f1\") " pod="opendatahub/model-serving-api-54d89bd5fc-vr6pj" Jul 21 00:47:09.074571 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:09.074499 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5-istio-token\") pod \"istiod-openshift-gateway-7574459656-trzbg\" (UID: \"36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-trzbg" Jul 21 00:47:09.074571 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:09.074528 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-trzbg\" (UID: \"36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-trzbg" Jul 21 00:47:09.075519 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:09.075183 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-trzbg\" (UID: \"36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-trzbg" Jul 21 00:47:09.077221 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:09.077191 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-trzbg\" (UID: \"36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-trzbg" Jul 21 00:47:09.077832 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:09.077803 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5-cacerts\") pod \"istiod-openshift-gateway-7574459656-trzbg\" (UID: \"36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-trzbg" Jul 21 00:47:09.077977 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:09.077940 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9a7a22c-1842-4eca-9b2d-4e3509bcf9f1-tls-certs\") pod \"model-serving-api-54d89bd5fc-vr6pj\" (UID: \"d9a7a22c-1842-4eca-9b2d-4e3509bcf9f1\") " pod="opendatahub/model-serving-api-54d89bd5fc-vr6pj" Jul 21 00:47:09.077977 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:09.077943 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-trzbg\" (UID: \"36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-trzbg" Jul 21 00:47:09.078381 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:09.078365 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5-local-certs\") pod \"istiod-openshift-gateway-7574459656-trzbg\" (UID: \"36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-trzbg" Jul 21 00:47:09.083003 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:09.082975 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfsmr\" (UniqueName: \"kubernetes.io/projected/36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5-kube-api-access-wfsmr\") pod \"istiod-openshift-gateway-7574459656-trzbg\" (UID: \"36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-trzbg" Jul 21 00:47:09.083132 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:09.083110 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5-istio-token\") pod \"istiod-openshift-gateway-7574459656-trzbg\" (UID: \"36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-trzbg" Jul 21 00:47:09.115160 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:09.115134 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-trzbg" Jul 21 00:47:09.268831 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:09.268716 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-vr6pj" Jul 21 00:47:09.295070 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:09.293793 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-trzbg"] Jul 21 00:47:09.300867 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:47:09.300798 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36ad18f7_d8fb_4cbb_b0de_c6abcfd04fb5.slice/crio-b9a98444b0dcec7f9d150e88f62d25c55daf7bd35e6be9d0c377f376efe035fd WatchSource:0}: Error finding container b9a98444b0dcec7f9d150e88f62d25c55daf7bd35e6be9d0c377f376efe035fd: Status 404 returned error can't find the container with id b9a98444b0dcec7f9d150e88f62d25c55daf7bd35e6be9d0c377f376efe035fd Jul 21 00:47:09.452694 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:09.452628 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-vr6pj"] Jul 21 00:47:09.460869 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:47:09.460837 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9a7a22c_1842_4eca_9b2d_4e3509bcf9f1.slice/crio-18cb42753f297e74b757598a7947c79c726729e3673975f7b3d76e4c252a7837 WatchSource:0}: Error finding container 18cb42753f297e74b757598a7947c79c726729e3673975f7b3d76e4c252a7837: Status 404 returned error can't find the container with id 18cb42753f297e74b757598a7947c79c726729e3673975f7b3d76e4c252a7837 Jul 21 00:47:09.522993 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:09.522905 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-vr6pj" event={"ID":"d9a7a22c-1842-4eca-9b2d-4e3509bcf9f1","Type":"ContainerStarted","Data":"18cb42753f297e74b757598a7947c79c726729e3673975f7b3d76e4c252a7837"} Jul 21 00:47:09.524551 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:09.524521 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-zmq6v" event={"ID":"115ede03-1e4e-432b-a874-579f0fa66f73","Type":"ContainerStarted","Data":"c210ec850ef65c2042c9500e57821e5e48ca9ee8ee14c132a37aa86b2aa8d724"} Jul 21 00:47:09.526359 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:09.526333 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-trzbg" event={"ID":"36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5","Type":"ContainerStarted","Data":"b9a98444b0dcec7f9d150e88f62d25c55daf7bd35e6be9d0c377f376efe035fd"} Jul 21 00:47:12.545725 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:12.545670 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5c7494f749-46ckj" event={"ID":"8d95153d-b67d-4c35-81cb-7dc970ac6ba6","Type":"ContainerStarted","Data":"acb7902c7a19b5b11286ba8b15e8be5c12621061618d9f04cfcdc2b3b1322fe8"} Jul 21 00:47:12.548172 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:12.548093 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-vr6pj" event={"ID":"d9a7a22c-1842-4eca-9b2d-4e3509bcf9f1","Type":"ContainerStarted","Data":"6789b55624b094b143abb09635ccd1dc2e9f78a427025386e6a7002ef1d56914"} Jul 21 00:47:12.548343 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:12.548194 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-vr6pj" Jul 21 00:47:12.549983 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:12.549939 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-zmq6v" event={"ID":"115ede03-1e4e-432b-a874-579f0fa66f73","Type":"ContainerStarted","Data":"6600331fd86fe758c151924d5e52eaa02b2bf93279cd620816988f6b7c5716ca"} Jul 21 00:47:12.550104 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:12.550005 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-zmq6v" Jul 21 00:47:12.562411 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:12.562368 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5c7494f749-46ckj" podStartSLOduration=1.438124534 podStartE2EDuration="5.562354059s" totalImagesPullingTime="4.124229525s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:47:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:47:12.560654341 +0000 UTC m=+380.166763384" watchObservedRunningTime="2026-07-21 00:47:12.562354059 +0000 UTC m=+380.168463098" Jul 21 00:47:12.578465 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:12.578040 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-zmq6v" podStartSLOduration=1.757859738 podStartE2EDuration="5.578018952s" totalImagesPullingTime="3.820159214s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:47:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:47:12.576168531 +0000 UTC m=+380.182277573" watchObservedRunningTime="2026-07-21 00:47:12.578018952 +0000 UTC m=+380.184127992" Jul 21 00:47:12.593721 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:12.593661 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-vr6pj" podStartSLOduration=1.698407325 podStartE2EDuration="4.59363964s" totalImagesPullingTime="2.895232315s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:47:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:47:12.592855184 +0000 UTC m=+380.198964225" watchObservedRunningTime="2026-07-21 00:47:12.59363964 +0000 UTC m=+380.199748681" Jul 21 00:47:13.975921 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:13.975883 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 00:47:13.976166 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:13.975950 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 00:47:14.563945 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:14.563899 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-trzbg" event={"ID":"36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5","Type":"ContainerStarted","Data":"c6ce5f62c5add6dca1b09d02c10e477304547f501a61a76382e6fd3f7d6396c0"} Jul 21 00:47:14.564136 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:14.564044 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-trzbg" Jul 21 00:47:14.595700 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:14.595650 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-trzbg" podStartSLOduration=1.923552588 podStartE2EDuration="6.595635973s" totalImagesPullingTime="4.672083385s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:47:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:47:14.59562939 +0000 UTC m=+382.201738429" watchObservedRunningTime="2026-07-21 00:47:14.595635973 +0000 UTC m=+382.201745013" Jul 21 00:47:15.569976 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:15.569947 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-trzbg" Jul 21 00:47:18.364908 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:18.364870 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-4zltv"] Jul 21 00:47:18.367924 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:18.367899 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-4zltv" Jul 21 00:47:18.372370 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:18.372347 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 21 00:47:18.372483 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:18.372409 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-xvrg2\"" Jul 21 00:47:18.374844 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:18.374821 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-nhnq7"] Jul 21 00:47:18.378497 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:18.378475 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-nhnq7" Jul 21 00:47:18.380214 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:18.380193 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-4zltv"] Jul 21 00:47:18.381447 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:18.381425 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-47cdp\"" Jul 21 00:47:18.381557 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:18.381435 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 21 00:47:18.386500 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:18.386482 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-nhnq7"] Jul 21 00:47:18.459105 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:18.459070 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vjp7\" (UniqueName: \"kubernetes.io/projected/86f64a6f-7e40-4f9d-84e9-f1849e0340c9-kube-api-access-8vjp7\") pod \"llmisvc-controller-manager-777c6bd7d5-nhnq7\" (UID: \"86f64a6f-7e40-4f9d-84e9-f1849e0340c9\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-nhnq7" Jul 21 00:47:18.459279 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:18.459117 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/86f64a6f-7e40-4f9d-84e9-f1849e0340c9-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-nhnq7\" (UID: \"86f64a6f-7e40-4f9d-84e9-f1849e0340c9\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-nhnq7" Jul 21 00:47:18.459279 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:18.459145 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6993a3d6-6e24-48cf-b63c-35f1cbf9c771-cert\") pod \"kserve-controller-manager-d6f4b65b8-4zltv\" (UID: \"6993a3d6-6e24-48cf-b63c-35f1cbf9c771\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-4zltv" Jul 21 00:47:18.459279 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:18.459161 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbdkp\" (UniqueName: \"kubernetes.io/projected/6993a3d6-6e24-48cf-b63c-35f1cbf9c771-kube-api-access-sbdkp\") pod \"kserve-controller-manager-d6f4b65b8-4zltv\" (UID: \"6993a3d6-6e24-48cf-b63c-35f1cbf9c771\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-4zltv" Jul 21 00:47:18.559772 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:18.559741 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vjp7\" (UniqueName: \"kubernetes.io/projected/86f64a6f-7e40-4f9d-84e9-f1849e0340c9-kube-api-access-8vjp7\") pod \"llmisvc-controller-manager-777c6bd7d5-nhnq7\" (UID: \"86f64a6f-7e40-4f9d-84e9-f1849e0340c9\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-nhnq7" Jul 21 00:47:18.559934 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:18.559786 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/86f64a6f-7e40-4f9d-84e9-f1849e0340c9-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-nhnq7\" (UID: \"86f64a6f-7e40-4f9d-84e9-f1849e0340c9\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-nhnq7" Jul 21 00:47:18.559934 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:18.559813 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6993a3d6-6e24-48cf-b63c-35f1cbf9c771-cert\") pod \"kserve-controller-manager-d6f4b65b8-4zltv\" (UID: \"6993a3d6-6e24-48cf-b63c-35f1cbf9c771\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-4zltv" Jul 21 00:47:18.559934 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:18.559836 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbdkp\" (UniqueName: \"kubernetes.io/projected/6993a3d6-6e24-48cf-b63c-35f1cbf9c771-kube-api-access-sbdkp\") pod \"kserve-controller-manager-d6f4b65b8-4zltv\" (UID: \"6993a3d6-6e24-48cf-b63c-35f1cbf9c771\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-4zltv" Jul 21 00:47:18.562342 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:18.562317 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/86f64a6f-7e40-4f9d-84e9-f1849e0340c9-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-nhnq7\" (UID: \"86f64a6f-7e40-4f9d-84e9-f1849e0340c9\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-nhnq7" Jul 21 00:47:18.562440 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:18.562366 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6993a3d6-6e24-48cf-b63c-35f1cbf9c771-cert\") pod \"kserve-controller-manager-d6f4b65b8-4zltv\" (UID: \"6993a3d6-6e24-48cf-b63c-35f1cbf9c771\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-4zltv" Jul 21 00:47:18.574091 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:18.574052 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbdkp\" (UniqueName: \"kubernetes.io/projected/6993a3d6-6e24-48cf-b63c-35f1cbf9c771-kube-api-access-sbdkp\") pod \"kserve-controller-manager-d6f4b65b8-4zltv\" (UID: \"6993a3d6-6e24-48cf-b63c-35f1cbf9c771\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-4zltv" Jul 21 00:47:18.575029 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:18.575012 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vjp7\" (UniqueName: \"kubernetes.io/projected/86f64a6f-7e40-4f9d-84e9-f1849e0340c9-kube-api-access-8vjp7\") pod \"llmisvc-controller-manager-777c6bd7d5-nhnq7\" (UID: \"86f64a6f-7e40-4f9d-84e9-f1849e0340c9\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-nhnq7" Jul 21 00:47:18.680061 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:18.680007 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-4zltv" Jul 21 00:47:18.690465 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:18.690444 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-nhnq7" Jul 21 00:47:18.834225 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:18.834189 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-4zltv"] Jul 21 00:47:18.835600 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:47:18.835555 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6993a3d6_6e24_48cf_b63c_35f1cbf9c771.slice/crio-62d7231b8aa8fa55f8bfa9be8cad709db2af63c85e037045ea62700dc769c2a0 WatchSource:0}: Error finding container 62d7231b8aa8fa55f8bfa9be8cad709db2af63c85e037045ea62700dc769c2a0: Status 404 returned error can't find the container with id 62d7231b8aa8fa55f8bfa9be8cad709db2af63c85e037045ea62700dc769c2a0 Jul 21 00:47:18.856812 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:18.856787 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-nhnq7"] Jul 21 00:47:18.859489 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:47:18.859463 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod86f64a6f_7e40_4f9d_84e9_f1849e0340c9.slice/crio-1957e2b3746c52f21b37329cfb6e7d066b09bbf55f61894ba23748e378ad0c79 WatchSource:0}: Error finding container 1957e2b3746c52f21b37329cfb6e7d066b09bbf55f61894ba23748e378ad0c79: Status 404 returned error can't find the container with id 1957e2b3746c52f21b37329cfb6e7d066b09bbf55f61894ba23748e378ad0c79 Jul 21 00:47:19.588118 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:19.588086 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-nhnq7" event={"ID":"86f64a6f-7e40-4f9d-84e9-f1849e0340c9","Type":"ContainerStarted","Data":"1957e2b3746c52f21b37329cfb6e7d066b09bbf55f61894ba23748e378ad0c79"} Jul 21 00:47:19.589521 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:19.589494 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-4zltv" event={"ID":"6993a3d6-6e24-48cf-b63c-35f1cbf9c771","Type":"ContainerStarted","Data":"62d7231b8aa8fa55f8bfa9be8cad709db2af63c85e037045ea62700dc769c2a0"} Jul 21 00:47:21.250064 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:21.250018 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-trzbg"] Jul 21 00:47:21.250536 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:21.250475 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-trzbg" podUID="36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5" containerName="discovery" containerID="cri-o://c6ce5f62c5add6dca1b09d02c10e477304547f501a61a76382e6fd3f7d6396c0" gracePeriod=30 Jul 21 00:47:21.582516 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:21.582486 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-trzbg" Jul 21 00:47:21.600140 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:21.600108 2580 generic.go:356] "Generic (PLEG): container finished" podID="36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5" containerID="c6ce5f62c5add6dca1b09d02c10e477304547f501a61a76382e6fd3f7d6396c0" exitCode=0 Jul 21 00:47:21.600310 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:21.600158 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-trzbg" Jul 21 00:47:21.600310 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:21.600189 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-trzbg" event={"ID":"36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5","Type":"ContainerDied","Data":"c6ce5f62c5add6dca1b09d02c10e477304547f501a61a76382e6fd3f7d6396c0"} Jul 21 00:47:21.600310 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:21.600286 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-trzbg" event={"ID":"36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5","Type":"ContainerDied","Data":"b9a98444b0dcec7f9d150e88f62d25c55daf7bd35e6be9d0c377f376efe035fd"} Jul 21 00:47:21.600442 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:21.600311 2580 scope.go:117] "RemoveContainer" containerID="c6ce5f62c5add6dca1b09d02c10e477304547f501a61a76382e6fd3f7d6396c0" Jul 21 00:47:21.601947 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:21.601919 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-4zltv" event={"ID":"6993a3d6-6e24-48cf-b63c-35f1cbf9c771","Type":"ContainerStarted","Data":"029c4c7df3cd0bdc27b5b5595e32fd3690eb63624a13fd3fdec17de295f094ec"} Jul 21 00:47:21.602088 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:21.602052 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-4zltv" Jul 21 00:47:21.633341 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:21.633282 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-4zltv" podStartSLOduration=0.994132392 podStartE2EDuration="3.633265495s" totalImagesPullingTime="2.639133103s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:47:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:47:21.632643512 +0000 UTC m=+389.238752551" watchObservedRunningTime="2026-07-21 00:47:21.633265495 +0000 UTC m=+389.239374532" Jul 21 00:47:21.691137 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:21.691103 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5-istio-token\") pod \"36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5\" (UID: \"36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5\") " Jul 21 00:47:21.691318 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:21.691159 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5-local-certs\") pod \"36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5\" (UID: \"36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5\") " Jul 21 00:47:21.691318 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:21.691195 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5-cacerts\") pod \"36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5\" (UID: \"36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5\") " Jul 21 00:47:21.691318 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:21.691246 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5-istio-csr-dns-cert\") pod \"36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5\" (UID: \"36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5\") " Jul 21 00:47:21.691318 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:21.691293 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5-istio-csr-ca-configmap\") pod \"36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5\" (UID: \"36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5\") " Jul 21 00:47:21.691318 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:21.691313 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wfsmr\" (UniqueName: \"kubernetes.io/projected/36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5-kube-api-access-wfsmr\") pod \"36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5\" (UID: \"36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5\") " Jul 21 00:47:21.691569 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:21.691370 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5-istio-kubeconfig\") pod \"36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5\" (UID: \"36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5\") " Jul 21 00:47:21.691972 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:21.691697 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5" (UID: "36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 00:47:21.694283 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:21.694244 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5-cacerts" (OuterVolumeSpecName: "cacerts") pod "36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5" (UID: "36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 00:47:21.694416 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:21.694277 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5-istio-token" (OuterVolumeSpecName: "istio-token") pod "36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5" (UID: "36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 00:47:21.694416 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:21.694381 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5" (UID: "36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 00:47:21.694543 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:21.694446 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5-kube-api-access-wfsmr" (OuterVolumeSpecName: "kube-api-access-wfsmr") pod "36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5" (UID: "36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5"). InnerVolumeSpecName "kube-api-access-wfsmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 00:47:21.694543 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:21.694537 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5-local-certs" (OuterVolumeSpecName: "local-certs") pod "36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5" (UID: "36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 00:47:21.694663 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:21.694638 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5" (UID: "36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 00:47:21.792960 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:21.792871 2580 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5-cacerts\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 21 00:47:21.792960 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:21.792906 2580 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5-istio-csr-dns-cert\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 21 00:47:21.792960 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:21.792921 2580 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5-istio-csr-ca-configmap\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 21 00:47:21.792960 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:21.792935 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfsmr\" (UniqueName: \"kubernetes.io/projected/36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5-kube-api-access-wfsmr\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 21 00:47:21.792960 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:21.792947 2580 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5-istio-kubeconfig\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 21 00:47:21.792960 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:21.792960 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5-istio-token\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 21 00:47:21.793300 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:21.792972 2580 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5-local-certs\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 21 00:47:21.923550 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:21.923517 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-trzbg"] Jul 21 00:47:21.929491 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:21.929465 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-trzbg"] Jul 21 00:47:22.127974 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:22.127951 2580 scope.go:117] "RemoveContainer" containerID="c6ce5f62c5add6dca1b09d02c10e477304547f501a61a76382e6fd3f7d6396c0" Jul 21 00:47:22.128328 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:47:22.128303 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6ce5f62c5add6dca1b09d02c10e477304547f501a61a76382e6fd3f7d6396c0\": container with ID starting with c6ce5f62c5add6dca1b09d02c10e477304547f501a61a76382e6fd3f7d6396c0 not found: ID does not exist" containerID="c6ce5f62c5add6dca1b09d02c10e477304547f501a61a76382e6fd3f7d6396c0" Jul 21 00:47:22.128419 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:22.128341 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6ce5f62c5add6dca1b09d02c10e477304547f501a61a76382e6fd3f7d6396c0"} err="failed to get container status \"c6ce5f62c5add6dca1b09d02c10e477304547f501a61a76382e6fd3f7d6396c0\": rpc error: code = NotFound desc = could not find container \"c6ce5f62c5add6dca1b09d02c10e477304547f501a61a76382e6fd3f7d6396c0\": container with ID starting with c6ce5f62c5add6dca1b09d02c10e477304547f501a61a76382e6fd3f7d6396c0 not found: ID does not exist" Jul 21 00:47:22.608285 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:22.608226 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-nhnq7" event={"ID":"86f64a6f-7e40-4f9d-84e9-f1849e0340c9","Type":"ContainerStarted","Data":"8e4d92a5f38ff62d9f8e7103533db696c27a2ba02c91d6e3c5a3f50e92c81fa2"} Jul 21 00:47:22.626054 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:22.626017 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-nhnq7" podStartSLOduration=1.311391098 podStartE2EDuration="4.62600186s" totalImagesPullingTime="3.314610762s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:47:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:47:22.624762535 +0000 UTC m=+390.230871586" watchObservedRunningTime="2026-07-21 00:47:22.62600186 +0000 UTC m=+390.232110903" Jul 21 00:47:22.912181 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:22.912089 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5" path="/var/lib/kubelet/pods/36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5/volumes" Jul 21 00:47:23.558003 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:23.557969 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-zmq6v" Jul 21 00:47:23.559910 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:23.559890 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-vr6pj" Jul 21 00:47:23.613000 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:23.612962 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-nhnq7" Jul 21 00:47:26.570082 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:26.570035 2580 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-trzbg container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.27:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 21 00:47:26.570595 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:26.570105 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-trzbg" podUID="36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5" containerName="discovery" probeResult="failure" output="Get \"http://10.132.0.27:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 21 00:47:52.613954 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:52.613925 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-4zltv" Jul 21 00:47:54.619603 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:47:54.619574 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-nhnq7" Jul 21 00:48:08.251171 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:08.251138 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-c4l45"] Jul 21 00:48:08.251595 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:08.251483 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5" containerName="discovery" Jul 21 00:48:08.251595 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:08.251494 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5" containerName="discovery" Jul 21 00:48:08.251595 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:08.251554 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="36ad18f7-d8fb-4cbb-b0de-c6abcfd04fb5" containerName="discovery" Jul 21 00:48:08.256509 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:08.256487 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-c4l45" Jul 21 00:48:08.260383 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:08.259959 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 21 00:48:08.263977 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:08.263766 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 21 00:48:08.263977 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:08.263805 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-kzlzf\"" Jul 21 00:48:08.264428 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:08.264410 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-c4l45"] Jul 21 00:48:08.284437 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:08.284413 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ww49b\" (UniqueName: \"kubernetes.io/projected/54fc95ab-b359-448d-ad5f-687fbf7eadf0-kube-api-access-ww49b\") pod \"kuadrant-operator-catalog-c4l45\" (UID: \"54fc95ab-b359-448d-ad5f-687fbf7eadf0\") " pod="kuadrant-system/kuadrant-operator-catalog-c4l45" Jul 21 00:48:08.385364 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:08.385333 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww49b\" (UniqueName: \"kubernetes.io/projected/54fc95ab-b359-448d-ad5f-687fbf7eadf0-kube-api-access-ww49b\") pod \"kuadrant-operator-catalog-c4l45\" (UID: \"54fc95ab-b359-448d-ad5f-687fbf7eadf0\") " pod="kuadrant-system/kuadrant-operator-catalog-c4l45" Jul 21 00:48:08.393285 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:08.393259 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww49b\" (UniqueName: \"kubernetes.io/projected/54fc95ab-b359-448d-ad5f-687fbf7eadf0-kube-api-access-ww49b\") pod \"kuadrant-operator-catalog-c4l45\" (UID: \"54fc95ab-b359-448d-ad5f-687fbf7eadf0\") " pod="kuadrant-system/kuadrant-operator-catalog-c4l45" Jul 21 00:48:08.570096 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:08.570016 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-c4l45" Jul 21 00:48:08.616503 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:08.616475 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-c4l45"] Jul 21 00:48:08.693058 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:08.692980 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-c4l45"] Jul 21 00:48:08.695104 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:48:08.695079 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54fc95ab_b359_448d_ad5f_687fbf7eadf0.slice/crio-bb026820c08dae0bb69ab229382a8a40cd7c3dbabf743cde3e8e6777121efdf1 WatchSource:0}: Error finding container bb026820c08dae0bb69ab229382a8a40cd7c3dbabf743cde3e8e6777121efdf1: Status 404 returned error can't find the container with id bb026820c08dae0bb69ab229382a8a40cd7c3dbabf743cde3e8e6777121efdf1 Jul 21 00:48:08.817997 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:08.817960 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-c4l45" event={"ID":"54fc95ab-b359-448d-ad5f-687fbf7eadf0","Type":"ContainerStarted","Data":"bb026820c08dae0bb69ab229382a8a40cd7c3dbabf743cde3e8e6777121efdf1"} Jul 21 00:48:08.822520 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:08.822497 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-7dg7r"] Jul 21 00:48:08.825335 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:08.825320 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-7dg7r" Jul 21 00:48:08.833929 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:08.833905 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-7dg7r"] Jul 21 00:48:08.889681 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:08.889648 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2rgg\" (UniqueName: \"kubernetes.io/projected/11ed89e1-5c04-404c-84dc-16a12138c760-kube-api-access-h2rgg\") pod \"kuadrant-operator-catalog-7dg7r\" (UID: \"11ed89e1-5c04-404c-84dc-16a12138c760\") " pod="kuadrant-system/kuadrant-operator-catalog-7dg7r" Jul 21 00:48:08.990455 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:08.990421 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2rgg\" (UniqueName: \"kubernetes.io/projected/11ed89e1-5c04-404c-84dc-16a12138c760-kube-api-access-h2rgg\") pod \"kuadrant-operator-catalog-7dg7r\" (UID: \"11ed89e1-5c04-404c-84dc-16a12138c760\") " pod="kuadrant-system/kuadrant-operator-catalog-7dg7r" Jul 21 00:48:08.998206 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:08.998175 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2rgg\" (UniqueName: \"kubernetes.io/projected/11ed89e1-5c04-404c-84dc-16a12138c760-kube-api-access-h2rgg\") pod \"kuadrant-operator-catalog-7dg7r\" (UID: \"11ed89e1-5c04-404c-84dc-16a12138c760\") " pod="kuadrant-system/kuadrant-operator-catalog-7dg7r" Jul 21 00:48:09.135263 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:09.135162 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-7dg7r" Jul 21 00:48:09.270615 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:09.270592 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-7dg7r"] Jul 21 00:48:09.272774 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:48:09.272746 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11ed89e1_5c04_404c_84dc_16a12138c760.slice/crio-4e12ee0678067dfe2b82215dc54b69906500301ea5d0d029a7057888e6806e80 WatchSource:0}: Error finding container 4e12ee0678067dfe2b82215dc54b69906500301ea5d0d029a7057888e6806e80: Status 404 returned error can't find the container with id 4e12ee0678067dfe2b82215dc54b69906500301ea5d0d029a7057888e6806e80 Jul 21 00:48:09.823949 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:09.823914 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-7dg7r" event={"ID":"11ed89e1-5c04-404c-84dc-16a12138c760","Type":"ContainerStarted","Data":"4e12ee0678067dfe2b82215dc54b69906500301ea5d0d029a7057888e6806e80"} Jul 21 00:48:11.835714 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:11.835673 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-7dg7r" event={"ID":"11ed89e1-5c04-404c-84dc-16a12138c760","Type":"ContainerStarted","Data":"88a009635d8235c1b05eee1f7dd390409c6aada95860e2ff6edf35def5539b80"} Jul 21 00:48:11.837071 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:11.837047 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-c4l45" event={"ID":"54fc95ab-b359-448d-ad5f-687fbf7eadf0","Type":"ContainerStarted","Data":"a64fd44c439b3722989b5df9426550d345254a9e4826301723d5ac98bf4d000f"} Jul 21 00:48:11.837206 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:11.837168 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-c4l45" podUID="54fc95ab-b359-448d-ad5f-687fbf7eadf0" containerName="registry-server" containerID="cri-o://a64fd44c439b3722989b5df9426550d345254a9e4826301723d5ac98bf4d000f" gracePeriod=2 Jul 21 00:48:11.851894 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:11.851853 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-7dg7r" podStartSLOduration=1.996921358 podStartE2EDuration="3.851838532s" totalImagesPullingTime="1.854917174s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:48:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:48:11.850411828 +0000 UTC m=+439.456520870" watchObservedRunningTime="2026-07-21 00:48:11.851838532 +0000 UTC m=+439.457947572" Jul 21 00:48:11.864563 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:11.864522 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-c4l45" podStartSLOduration=1.434409186 podStartE2EDuration="3.864508312s" totalImagesPullingTime="2.430099126s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:48:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:48:11.863776772 +0000 UTC m=+439.469885814" watchObservedRunningTime="2026-07-21 00:48:11.864508312 +0000 UTC m=+439.470617352" Jul 21 00:48:12.079921 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:12.079899 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-c4l45" Jul 21 00:48:12.116800 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:12.116743 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ww49b\" (UniqueName: \"kubernetes.io/projected/54fc95ab-b359-448d-ad5f-687fbf7eadf0-kube-api-access-ww49b\") pod \"54fc95ab-b359-448d-ad5f-687fbf7eadf0\" (UID: \"54fc95ab-b359-448d-ad5f-687fbf7eadf0\") " Jul 21 00:48:12.118724 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:12.118697 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54fc95ab-b359-448d-ad5f-687fbf7eadf0-kube-api-access-ww49b" (OuterVolumeSpecName: "kube-api-access-ww49b") pod "54fc95ab-b359-448d-ad5f-687fbf7eadf0" (UID: "54fc95ab-b359-448d-ad5f-687fbf7eadf0"). InnerVolumeSpecName "kube-api-access-ww49b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 00:48:12.217647 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:12.217613 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ww49b\" (UniqueName: \"kubernetes.io/projected/54fc95ab-b359-448d-ad5f-687fbf7eadf0-kube-api-access-ww49b\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 21 00:48:12.842525 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:12.842490 2580 generic.go:356] "Generic (PLEG): container finished" podID="54fc95ab-b359-448d-ad5f-687fbf7eadf0" containerID="a64fd44c439b3722989b5df9426550d345254a9e4826301723d5ac98bf4d000f" exitCode=0 Jul 21 00:48:12.842920 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:12.842536 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-c4l45" Jul 21 00:48:12.842920 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:12.842575 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-c4l45" event={"ID":"54fc95ab-b359-448d-ad5f-687fbf7eadf0","Type":"ContainerDied","Data":"a64fd44c439b3722989b5df9426550d345254a9e4826301723d5ac98bf4d000f"} Jul 21 00:48:12.842920 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:12.842636 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-c4l45" event={"ID":"54fc95ab-b359-448d-ad5f-687fbf7eadf0","Type":"ContainerDied","Data":"bb026820c08dae0bb69ab229382a8a40cd7c3dbabf743cde3e8e6777121efdf1"} Jul 21 00:48:12.842920 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:12.842657 2580 scope.go:117] "RemoveContainer" containerID="a64fd44c439b3722989b5df9426550d345254a9e4826301723d5ac98bf4d000f" Jul 21 00:48:12.851738 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:12.851722 2580 scope.go:117] "RemoveContainer" containerID="a64fd44c439b3722989b5df9426550d345254a9e4826301723d5ac98bf4d000f" Jul 21 00:48:12.851981 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:48:12.851963 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a64fd44c439b3722989b5df9426550d345254a9e4826301723d5ac98bf4d000f\": container with ID starting with a64fd44c439b3722989b5df9426550d345254a9e4826301723d5ac98bf4d000f not found: ID does not exist" containerID="a64fd44c439b3722989b5df9426550d345254a9e4826301723d5ac98bf4d000f" Jul 21 00:48:12.852026 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:12.851989 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a64fd44c439b3722989b5df9426550d345254a9e4826301723d5ac98bf4d000f"} err="failed to get container status \"a64fd44c439b3722989b5df9426550d345254a9e4826301723d5ac98bf4d000f\": rpc error: code = NotFound desc = could not find container \"a64fd44c439b3722989b5df9426550d345254a9e4826301723d5ac98bf4d000f\": container with ID starting with a64fd44c439b3722989b5df9426550d345254a9e4826301723d5ac98bf4d000f not found: ID does not exist" Jul 21 00:48:12.863403 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:12.863382 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-c4l45"] Jul 21 00:48:12.866501 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:12.866480 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-c4l45"] Jul 21 00:48:12.912152 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:12.912128 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54fc95ab-b359-448d-ad5f-687fbf7eadf0" path="/var/lib/kubelet/pods/54fc95ab-b359-448d-ad5f-687fbf7eadf0/volumes" Jul 21 00:48:19.136328 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:19.136229 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-7dg7r" Jul 21 00:48:19.136328 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:19.136307 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-7dg7r" Jul 21 00:48:19.158137 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:19.158114 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-7dg7r" Jul 21 00:48:19.895091 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:19.895067 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-7dg7r" Jul 21 00:48:40.856807 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:40.856771 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zf9br"] Jul 21 00:48:40.857267 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:40.857130 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54fc95ab-b359-448d-ad5f-687fbf7eadf0" containerName="registry-server" Jul 21 00:48:40.857267 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:40.857142 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="54fc95ab-b359-448d-ad5f-687fbf7eadf0" containerName="registry-server" Jul 21 00:48:40.857267 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:40.857204 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="54fc95ab-b359-448d-ad5f-687fbf7eadf0" containerName="registry-server" Jul 21 00:48:40.864818 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:40.864797 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zf9br" Jul 21 00:48:40.867663 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:40.867636 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-twfzq\"" Jul 21 00:48:40.873122 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:40.873096 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zf9br"] Jul 21 00:48:40.950208 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:40.950175 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggkt7\" (UniqueName: \"kubernetes.io/projected/b2da871b-e678-48d6-b871-2f505ac0e278-kube-api-access-ggkt7\") pod \"limitador-operator-controller-manager-859d7b78bd-zf9br\" (UID: \"b2da871b-e678-48d6-b871-2f505ac0e278\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zf9br" Jul 21 00:48:41.051189 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:41.051152 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggkt7\" (UniqueName: \"kubernetes.io/projected/b2da871b-e678-48d6-b871-2f505ac0e278-kube-api-access-ggkt7\") pod \"limitador-operator-controller-manager-859d7b78bd-zf9br\" (UID: \"b2da871b-e678-48d6-b871-2f505ac0e278\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zf9br" Jul 21 00:48:41.064489 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:41.064460 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggkt7\" (UniqueName: \"kubernetes.io/projected/b2da871b-e678-48d6-b871-2f505ac0e278-kube-api-access-ggkt7\") pod \"limitador-operator-controller-manager-859d7b78bd-zf9br\" (UID: \"b2da871b-e678-48d6-b871-2f505ac0e278\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zf9br" Jul 21 00:48:41.175010 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:41.174932 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zf9br" Jul 21 00:48:41.298542 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:41.298518 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zf9br"] Jul 21 00:48:41.301093 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:48:41.301062 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2da871b_e678_48d6_b871_2f505ac0e278.slice/crio-cf2fc643e8898e20e92109fabb8c4f50085cd1eeb68cb0181c7da5e55d1e45fb WatchSource:0}: Error finding container cf2fc643e8898e20e92109fabb8c4f50085cd1eeb68cb0181c7da5e55d1e45fb: Status 404 returned error can't find the container with id cf2fc643e8898e20e92109fabb8c4f50085cd1eeb68cb0181c7da5e55d1e45fb Jul 21 00:48:41.974824 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:41.974783 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zf9br" event={"ID":"b2da871b-e678-48d6-b871-2f505ac0e278","Type":"ContainerStarted","Data":"cf2fc643e8898e20e92109fabb8c4f50085cd1eeb68cb0181c7da5e55d1e45fb"} Jul 21 00:48:43.986158 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:43.986126 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zf9br" event={"ID":"b2da871b-e678-48d6-b871-2f505ac0e278","Type":"ContainerStarted","Data":"58cea4007cfbc6cebe6b2b4690c185cd9e16add9acf18a83c27bc5b903ddd73f"} Jul 21 00:48:43.986570 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:43.986204 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zf9br" Jul 21 00:48:44.005422 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:44.005382 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zf9br" podStartSLOduration=2.143076645 podStartE2EDuration="4.005368482s" totalImagesPullingTime="1.862291837s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:48:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:48:44.003962416 +0000 UTC m=+471.610071455" watchObservedRunningTime="2026-07-21 00:48:44.005368482 +0000 UTC m=+471.611477521" Jul 21 00:48:45.601622 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:45.601586 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9cc4d"] Jul 21 00:48:45.605037 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:45.605018 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9cc4d" Jul 21 00:48:45.609105 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:45.609088 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-72g52\"" Jul 21 00:48:45.618418 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:45.618393 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9cc4d"] Jul 21 00:48:45.687226 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:45.687193 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4ad2b86a-ccaf-4f73-9cff-cca6dbab5f17-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-9cc4d\" (UID: \"4ad2b86a-ccaf-4f73-9cff-cca6dbab5f17\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9cc4d" Jul 21 00:48:45.687399 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:45.687288 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g8c7\" (UniqueName: \"kubernetes.io/projected/4ad2b86a-ccaf-4f73-9cff-cca6dbab5f17-kube-api-access-4g8c7\") pod \"kuadrant-operator-controller-manager-8684c9ff49-9cc4d\" (UID: \"4ad2b86a-ccaf-4f73-9cff-cca6dbab5f17\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9cc4d" Jul 21 00:48:45.788548 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:45.788512 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4ad2b86a-ccaf-4f73-9cff-cca6dbab5f17-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-9cc4d\" (UID: \"4ad2b86a-ccaf-4f73-9cff-cca6dbab5f17\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9cc4d" Jul 21 00:48:45.788687 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:45.788569 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g8c7\" (UniqueName: \"kubernetes.io/projected/4ad2b86a-ccaf-4f73-9cff-cca6dbab5f17-kube-api-access-4g8c7\") pod \"kuadrant-operator-controller-manager-8684c9ff49-9cc4d\" (UID: \"4ad2b86a-ccaf-4f73-9cff-cca6dbab5f17\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9cc4d" Jul 21 00:48:45.788870 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:45.788850 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4ad2b86a-ccaf-4f73-9cff-cca6dbab5f17-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-9cc4d\" (UID: \"4ad2b86a-ccaf-4f73-9cff-cca6dbab5f17\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9cc4d" Jul 21 00:48:45.797594 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:45.797573 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g8c7\" (UniqueName: \"kubernetes.io/projected/4ad2b86a-ccaf-4f73-9cff-cca6dbab5f17-kube-api-access-4g8c7\") pod \"kuadrant-operator-controller-manager-8684c9ff49-9cc4d\" (UID: \"4ad2b86a-ccaf-4f73-9cff-cca6dbab5f17\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9cc4d" Jul 21 00:48:45.914132 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:45.914039 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9cc4d" Jul 21 00:48:46.062969 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:46.062936 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9cc4d"] Jul 21 00:48:46.066387 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:48:46.066360 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ad2b86a_ccaf_4f73_9cff_cca6dbab5f17.slice/crio-a60a185b0e1a293b7e8d0b6092a9bb0fdc7bed8f3e99fb776456a8c51a9f7bd8 WatchSource:0}: Error finding container a60a185b0e1a293b7e8d0b6092a9bb0fdc7bed8f3e99fb776456a8c51a9f7bd8: Status 404 returned error can't find the container with id a60a185b0e1a293b7e8d0b6092a9bb0fdc7bed8f3e99fb776456a8c51a9f7bd8 Jul 21 00:48:47.002061 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:47.002025 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9cc4d" event={"ID":"4ad2b86a-ccaf-4f73-9cff-cca6dbab5f17","Type":"ContainerStarted","Data":"a60a185b0e1a293b7e8d0b6092a9bb0fdc7bed8f3e99fb776456a8c51a9f7bd8"} Jul 21 00:48:48.246324 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:48.246287 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-n5dmt"] Jul 21 00:48:48.249600 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:48.249581 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-n5dmt" Jul 21 00:48:48.252416 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:48.252344 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 21 00:48:48.252576 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:48.252429 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-mgvwx\"" Jul 21 00:48:48.264102 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:48.264079 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-n5dmt"] Jul 21 00:48:48.413699 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:48.413663 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s247t\" (UniqueName: \"kubernetes.io/projected/c33d7edb-ac53-4bd2-9a59-bc09fd48ae4c-kube-api-access-s247t\") pod \"dns-operator-controller-manager-864dd78c4c-n5dmt\" (UID: \"c33d7edb-ac53-4bd2-9a59-bc09fd48ae4c\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-n5dmt" Jul 21 00:48:48.514401 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:48.514336 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s247t\" (UniqueName: \"kubernetes.io/projected/c33d7edb-ac53-4bd2-9a59-bc09fd48ae4c-kube-api-access-s247t\") pod \"dns-operator-controller-manager-864dd78c4c-n5dmt\" (UID: \"c33d7edb-ac53-4bd2-9a59-bc09fd48ae4c\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-n5dmt" Jul 21 00:48:48.525280 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:48.525251 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s247t\" (UniqueName: \"kubernetes.io/projected/c33d7edb-ac53-4bd2-9a59-bc09fd48ae4c-kube-api-access-s247t\") pod \"dns-operator-controller-manager-864dd78c4c-n5dmt\" (UID: \"c33d7edb-ac53-4bd2-9a59-bc09fd48ae4c\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-n5dmt" Jul 21 00:48:48.561919 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:48.561894 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-n5dmt" Jul 21 00:48:48.698008 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:48.697984 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-n5dmt"] Jul 21 00:48:48.700310 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:48:48.700284 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc33d7edb_ac53_4bd2_9a59_bc09fd48ae4c.slice/crio-ce24068e42bfcfeb35121ad9f22fdf0f074f36dacdd689aa03b9fd8791546db0 WatchSource:0}: Error finding container ce24068e42bfcfeb35121ad9f22fdf0f074f36dacdd689aa03b9fd8791546db0: Status 404 returned error can't find the container with id ce24068e42bfcfeb35121ad9f22fdf0f074f36dacdd689aa03b9fd8791546db0 Jul 21 00:48:49.014308 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:49.014272 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-n5dmt" event={"ID":"c33d7edb-ac53-4bd2-9a59-bc09fd48ae4c","Type":"ContainerStarted","Data":"ce24068e42bfcfeb35121ad9f22fdf0f074f36dacdd689aa03b9fd8791546db0"} Jul 21 00:48:51.026763 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:51.026729 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-n5dmt" event={"ID":"c33d7edb-ac53-4bd2-9a59-bc09fd48ae4c","Type":"ContainerStarted","Data":"e050ef4bcd4c2c8979f7ac73986edc3bbb27ffd7114611d797fba37c4462fca9"} Jul 21 00:48:51.027197 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:51.026862 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-n5dmt" Jul 21 00:48:51.043458 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:51.043371 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-n5dmt" podStartSLOduration=0.963355637 podStartE2EDuration="3.043355686s" totalImagesPullingTime="2.080000049s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:48:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:48:51.041863385 +0000 UTC m=+478.647972425" watchObservedRunningTime="2026-07-21 00:48:51.043355686 +0000 UTC m=+478.649464726" Jul 21 00:48:51.748622 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:51.748586 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-gpzbr"] Jul 21 00:48:51.752134 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:51.752116 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-gpzbr" Jul 21 00:48:51.754814 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:51.754790 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-55fp6\"" Jul 21 00:48:51.762434 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:51.762412 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-gpzbr"] Jul 21 00:48:51.847609 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:51.847571 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xztrn\" (UniqueName: \"kubernetes.io/projected/0a38eb15-7b87-4f2b-9033-9c3e8715f642-kube-api-access-xztrn\") pod \"authorino-operator-75cc8d5cb7-gpzbr\" (UID: \"0a38eb15-7b87-4f2b-9033-9c3e8715f642\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-gpzbr" Jul 21 00:48:51.948976 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:51.948938 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xztrn\" (UniqueName: \"kubernetes.io/projected/0a38eb15-7b87-4f2b-9033-9c3e8715f642-kube-api-access-xztrn\") pod \"authorino-operator-75cc8d5cb7-gpzbr\" (UID: \"0a38eb15-7b87-4f2b-9033-9c3e8715f642\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-gpzbr" Jul 21 00:48:51.957607 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:51.957579 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xztrn\" (UniqueName: \"kubernetes.io/projected/0a38eb15-7b87-4f2b-9033-9c3e8715f642-kube-api-access-xztrn\") pod \"authorino-operator-75cc8d5cb7-gpzbr\" (UID: \"0a38eb15-7b87-4f2b-9033-9c3e8715f642\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-gpzbr" Jul 21 00:48:52.062584 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:52.062504 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-gpzbr" Jul 21 00:48:52.196998 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:52.196976 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-gpzbr"] Jul 21 00:48:52.198587 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:48:52.198562 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a38eb15_7b87_4f2b_9033_9c3e8715f642.slice/crio-88d13b57164e28453a0fd9bb235e62efabe04fcc62cb77c09cd0b0bcfda4d115 WatchSource:0}: Error finding container 88d13b57164e28453a0fd9bb235e62efabe04fcc62cb77c09cd0b0bcfda4d115: Status 404 returned error can't find the container with id 88d13b57164e28453a0fd9bb235e62efabe04fcc62cb77c09cd0b0bcfda4d115 Jul 21 00:48:53.038700 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:53.038617 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-gpzbr" event={"ID":"0a38eb15-7b87-4f2b-9033-9c3e8715f642","Type":"ContainerStarted","Data":"88d13b57164e28453a0fd9bb235e62efabe04fcc62cb77c09cd0b0bcfda4d115"} Jul 21 00:48:54.993285 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:54.993254 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zf9br" Jul 21 00:48:56.057682 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:56.057604 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9cc4d" event={"ID":"4ad2b86a-ccaf-4f73-9cff-cca6dbab5f17","Type":"ContainerStarted","Data":"c025ced330de95f1f9d379b58c6d78362be332e59aaa089ae479ec43e3cc05d8"} Jul 21 00:48:56.058095 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:56.057715 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9cc4d" Jul 21 00:48:56.059212 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:56.059182 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-gpzbr" event={"ID":"0a38eb15-7b87-4f2b-9033-9c3e8715f642","Type":"ContainerStarted","Data":"59ffe2d42dc1c15512a473b402509278f224efeb5feb059b9f8a6c35e67a0d63"} Jul 21 00:48:56.059346 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:56.059278 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-gpzbr" Jul 21 00:48:56.077289 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:56.077246 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9cc4d" podStartSLOduration=1.453891536 podStartE2EDuration="11.077219507s" totalImagesPullingTime="9.623327971s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:48:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:48:56.075224977 +0000 UTC m=+483.681334017" watchObservedRunningTime="2026-07-21 00:48:56.077219507 +0000 UTC m=+483.683328546" Jul 21 00:48:56.093069 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:48:56.093028 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-gpzbr" podStartSLOduration=1.568031999 podStartE2EDuration="5.093015705s" totalImagesPullingTime="3.524983706s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:48:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:48:56.091960442 +0000 UTC m=+483.698069482" watchObservedRunningTime="2026-07-21 00:48:56.093015705 +0000 UTC m=+483.699124745" Jul 21 00:49:02.032389 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:02.032359 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-n5dmt" Jul 21 00:49:07.065311 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:07.065283 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-gpzbr" Jul 21 00:49:07.065764 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:07.065351 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9cc4d" Jul 21 00:49:08.088764 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:08.088730 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2flrf"] Jul 21 00:49:08.092414 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:08.092397 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2flrf" Jul 21 00:49:08.104931 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:08.104909 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2flrf"] Jul 21 00:49:08.188087 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:08.188055 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9jwf\" (UniqueName: \"kubernetes.io/projected/f34779a0-f034-4195-a5ec-9145f78cea55-kube-api-access-t9jwf\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-2flrf\" (UID: \"f34779a0-f034-4195-a5ec-9145f78cea55\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2flrf" Jul 21 00:49:08.188087 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:08.188090 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f34779a0-f034-4195-a5ec-9145f78cea55-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-2flrf\" (UID: \"f34779a0-f034-4195-a5ec-9145f78cea55\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2flrf" Jul 21 00:49:08.288544 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:08.288507 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9jwf\" (UniqueName: \"kubernetes.io/projected/f34779a0-f034-4195-a5ec-9145f78cea55-kube-api-access-t9jwf\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-2flrf\" (UID: \"f34779a0-f034-4195-a5ec-9145f78cea55\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2flrf" Jul 21 00:49:08.288707 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:08.288565 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f34779a0-f034-4195-a5ec-9145f78cea55-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-2flrf\" (UID: \"f34779a0-f034-4195-a5ec-9145f78cea55\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2flrf" Jul 21 00:49:08.288887 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:08.288864 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f34779a0-f034-4195-a5ec-9145f78cea55-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-2flrf\" (UID: \"f34779a0-f034-4195-a5ec-9145f78cea55\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2flrf" Jul 21 00:49:08.297464 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:08.297443 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9jwf\" (UniqueName: \"kubernetes.io/projected/f34779a0-f034-4195-a5ec-9145f78cea55-kube-api-access-t9jwf\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-2flrf\" (UID: \"f34779a0-f034-4195-a5ec-9145f78cea55\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2flrf" Jul 21 00:49:08.402627 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:08.402551 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2flrf" Jul 21 00:49:08.540389 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:08.540363 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2flrf"] Jul 21 00:49:08.542428 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:49:08.542404 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf34779a0_f034_4195_a5ec_9145f78cea55.slice/crio-6e99b6d97de157bf66ba3a7fe15cd67cbc35448574b0ef39166f55f73fc60258 WatchSource:0}: Error finding container 6e99b6d97de157bf66ba3a7fe15cd67cbc35448574b0ef39166f55f73fc60258: Status 404 returned error can't find the container with id 6e99b6d97de157bf66ba3a7fe15cd67cbc35448574b0ef39166f55f73fc60258 Jul 21 00:49:08.764916 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:08.764834 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2flrf"] Jul 21 00:49:08.778417 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:08.778386 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2flrf"] Jul 21 00:49:08.793054 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:08.793019 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9cc4d"] Jul 21 00:49:08.793285 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:08.793215 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9cc4d" podUID="4ad2b86a-ccaf-4f73-9cff-cca6dbab5f17" containerName="manager" containerID="cri-o://c025ced330de95f1f9d379b58c6d78362be332e59aaa089ae479ec43e3cc05d8" gracePeriod=2 Jul 21 00:49:08.802818 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:08.802796 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-k9zhw"] Jul 21 00:49:08.803216 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:08.803199 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f34779a0-f034-4195-a5ec-9145f78cea55" containerName="manager" Jul 21 00:49:08.803216 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:08.803216 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f34779a0-f034-4195-a5ec-9145f78cea55" containerName="manager" Jul 21 00:49:08.803405 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:08.803316 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f34779a0-f034-4195-a5ec-9145f78cea55" containerName="manager" Jul 21 00:49:08.806577 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:08.806558 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-k9zhw" Jul 21 00:49:08.818133 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:08.818110 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9cc4d"] Jul 21 00:49:08.826915 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:08.826893 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-k9zhw"] Jul 21 00:49:08.829927 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:49:08.829897 2580 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"4ad2b86a-ccaf-4f73-9cff-cca6dbab5f17\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-9cc4d\": pods \"kuadrant-operator-controller-manager-8684c9ff49-9cc4d\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9cc4d" Jul 21 00:49:08.838201 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:08.838181 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zf9br"] Jul 21 00:49:08.838433 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:08.838396 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zf9br" podUID="b2da871b-e678-48d6-b871-2f505ac0e278" containerName="manager" containerID="cri-o://58cea4007cfbc6cebe6b2b4690c185cd9e16add9acf18a83c27bc5b903ddd73f" gracePeriod=2 Jul 21 00:49:08.847036 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:08.846984 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8jbtl"] Jul 21 00:49:08.848331 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:08.847542 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ad2b86a-ccaf-4f73-9cff-cca6dbab5f17" containerName="manager" Jul 21 00:49:08.848331 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:08.847564 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ad2b86a-ccaf-4f73-9cff-cca6dbab5f17" containerName="manager" Jul 21 00:49:08.848331 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:08.847682 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ad2b86a-ccaf-4f73-9cff-cca6dbab5f17" containerName="manager" Jul 21 00:49:08.850890 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:08.850861 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zf9br"] Jul 21 00:49:08.851001 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:08.850968 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8jbtl" Jul 21 00:49:08.864609 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:49:08.862596 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-9cc4d\" is forbidden: User \"system:node:ip-10-0-129-137.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-137.ec2.internal' and this object" podUID="4ad2b86a-ccaf-4f73-9cff-cca6dbab5f17" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9cc4d" Jul 21 00:49:08.867891 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:49:08.867304 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-9cc4d\" is forbidden: User \"system:node:ip-10-0-129-137.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-137.ec2.internal' and this object" podUID="4ad2b86a-ccaf-4f73-9cff-cca6dbab5f17" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9cc4d" Jul 21 00:49:08.874390 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:08.874368 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tgtqn"] Jul 21 00:49:08.874735 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:08.874724 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2da871b-e678-48d6-b871-2f505ac0e278" containerName="manager" Jul 21 00:49:08.874774 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:08.874737 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2da871b-e678-48d6-b871-2f505ac0e278" containerName="manager" Jul 21 00:49:08.874808 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:08.874798 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2da871b-e678-48d6-b871-2f505ac0e278" containerName="manager" Jul 21 00:49:08.877706 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:08.877690 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tgtqn" Jul 21 00:49:08.884585 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:08.884563 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8jbtl"] Jul 21 00:49:08.893326 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:08.893300 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4fba9395-dc6c-4360-8b07-b72dd0ff835c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-k9zhw\" (UID: \"4fba9395-dc6c-4360-8b07-b72dd0ff835c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-k9zhw" Jul 21 00:49:08.893425 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:08.893332 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znzkd\" (UniqueName: \"kubernetes.io/projected/4fba9395-dc6c-4360-8b07-b72dd0ff835c-kube-api-access-znzkd\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-k9zhw\" (UID: \"4fba9395-dc6c-4360-8b07-b72dd0ff835c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-k9zhw" Jul 21 00:49:08.893425 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:08.893364 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvjlt\" (UniqueName: \"kubernetes.io/projected/0b21ebd1-7350-47d0-92d8-c0429488af16-kube-api-access-nvjlt\") pod \"kuadrant-operator-controller-manager-8684c9ff49-8jbtl\" (UID: \"0b21ebd1-7350-47d0-92d8-c0429488af16\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8jbtl" Jul 21 00:49:08.893507 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:08.893430 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0b21ebd1-7350-47d0-92d8-c0429488af16-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-8jbtl\" (UID: \"0b21ebd1-7350-47d0-92d8-c0429488af16\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8jbtl" Jul 21 00:49:08.893507 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:08.893451 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fjqv\" (UniqueName: \"kubernetes.io/projected/a53c5164-d2ff-43eb-bffa-3dd49d9f0310-kube-api-access-7fjqv\") pod \"limitador-operator-controller-manager-859d7b78bd-tgtqn\" (UID: \"a53c5164-d2ff-43eb-bffa-3dd49d9f0310\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tgtqn" Jul 21 00:49:08.897692 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:08.897667 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tgtqn"] Jul 21 00:49:08.909980 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:49:08.909946 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-zf9br\" is forbidden: User \"system:node:ip-10-0-129-137.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-137.ec2.internal' and this object" podUID="b2da871b-e678-48d6-b871-2f505ac0e278" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zf9br" Jul 21 00:49:08.912518 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:49:08.912312 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-zf9br\" is forbidden: User \"system:node:ip-10-0-129-137.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-137.ec2.internal' and this object" podUID="b2da871b-e678-48d6-b871-2f505ac0e278" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zf9br" Jul 21 00:49:08.936360 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:49:08.936332 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-9cc4d\" is forbidden: User \"system:node:ip-10-0-129-137.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-137.ec2.internal' and this object" podUID="4ad2b86a-ccaf-4f73-9cff-cca6dbab5f17" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9cc4d" Jul 21 00:49:08.994842 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:08.994808 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4fba9395-dc6c-4360-8b07-b72dd0ff835c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-k9zhw\" (UID: \"4fba9395-dc6c-4360-8b07-b72dd0ff835c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-k9zhw" Jul 21 00:49:08.994842 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:08.994845 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znzkd\" (UniqueName: \"kubernetes.io/projected/4fba9395-dc6c-4360-8b07-b72dd0ff835c-kube-api-access-znzkd\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-k9zhw\" (UID: \"4fba9395-dc6c-4360-8b07-b72dd0ff835c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-k9zhw" Jul 21 00:49:08.995081 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:08.994874 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvjlt\" (UniqueName: \"kubernetes.io/projected/0b21ebd1-7350-47d0-92d8-c0429488af16-kube-api-access-nvjlt\") pod \"kuadrant-operator-controller-manager-8684c9ff49-8jbtl\" (UID: \"0b21ebd1-7350-47d0-92d8-c0429488af16\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8jbtl" Jul 21 00:49:08.995081 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:08.994906 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0b21ebd1-7350-47d0-92d8-c0429488af16-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-8jbtl\" (UID: \"0b21ebd1-7350-47d0-92d8-c0429488af16\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8jbtl" Jul 21 00:49:08.995081 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:08.994926 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fjqv\" (UniqueName: \"kubernetes.io/projected/a53c5164-d2ff-43eb-bffa-3dd49d9f0310-kube-api-access-7fjqv\") pod \"limitador-operator-controller-manager-859d7b78bd-tgtqn\" (UID: \"a53c5164-d2ff-43eb-bffa-3dd49d9f0310\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tgtqn" Jul 21 00:49:08.995269 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:08.995225 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0b21ebd1-7350-47d0-92d8-c0429488af16-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-8jbtl\" (UID: \"0b21ebd1-7350-47d0-92d8-c0429488af16\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8jbtl" Jul 21 00:49:08.995340 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:08.995296 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4fba9395-dc6c-4360-8b07-b72dd0ff835c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-k9zhw\" (UID: \"4fba9395-dc6c-4360-8b07-b72dd0ff835c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-k9zhw" Jul 21 00:49:09.004791 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:09.004495 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fjqv\" (UniqueName: \"kubernetes.io/projected/a53c5164-d2ff-43eb-bffa-3dd49d9f0310-kube-api-access-7fjqv\") pod \"limitador-operator-controller-manager-859d7b78bd-tgtqn\" (UID: \"a53c5164-d2ff-43eb-bffa-3dd49d9f0310\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tgtqn" Jul 21 00:49:09.004791 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:09.004637 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvjlt\" (UniqueName: \"kubernetes.io/projected/0b21ebd1-7350-47d0-92d8-c0429488af16-kube-api-access-nvjlt\") pod \"kuadrant-operator-controller-manager-8684c9ff49-8jbtl\" (UID: \"0b21ebd1-7350-47d0-92d8-c0429488af16\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8jbtl" Jul 21 00:49:09.004791 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:09.004691 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znzkd\" (UniqueName: \"kubernetes.io/projected/4fba9395-dc6c-4360-8b07-b72dd0ff835c-kube-api-access-znzkd\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-k9zhw\" (UID: \"4fba9395-dc6c-4360-8b07-b72dd0ff835c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-k9zhw" Jul 21 00:49:09.071740 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:09.071716 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9cc4d" Jul 21 00:49:09.078146 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:09.078124 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zf9br" Jul 21 00:49:09.095550 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:09.095520 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ggkt7\" (UniqueName: \"kubernetes.io/projected/b2da871b-e678-48d6-b871-2f505ac0e278-kube-api-access-ggkt7\") pod \"b2da871b-e678-48d6-b871-2f505ac0e278\" (UID: \"b2da871b-e678-48d6-b871-2f505ac0e278\") " Jul 21 00:49:09.095926 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:09.095574 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4ad2b86a-ccaf-4f73-9cff-cca6dbab5f17-extensions-socket-volume\") pod \"4ad2b86a-ccaf-4f73-9cff-cca6dbab5f17\" (UID: \"4ad2b86a-ccaf-4f73-9cff-cca6dbab5f17\") " Jul 21 00:49:09.095926 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:09.095598 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4g8c7\" (UniqueName: \"kubernetes.io/projected/4ad2b86a-ccaf-4f73-9cff-cca6dbab5f17-kube-api-access-4g8c7\") pod \"4ad2b86a-ccaf-4f73-9cff-cca6dbab5f17\" (UID: \"4ad2b86a-ccaf-4f73-9cff-cca6dbab5f17\") " Jul 21 00:49:09.096038 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:09.096014 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ad2b86a-ccaf-4f73-9cff-cca6dbab5f17-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "4ad2b86a-ccaf-4f73-9cff-cca6dbab5f17" (UID: "4ad2b86a-ccaf-4f73-9cff-cca6dbab5f17"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 00:49:09.097540 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:09.097519 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ad2b86a-ccaf-4f73-9cff-cca6dbab5f17-kube-api-access-4g8c7" (OuterVolumeSpecName: "kube-api-access-4g8c7") pod "4ad2b86a-ccaf-4f73-9cff-cca6dbab5f17" (UID: "4ad2b86a-ccaf-4f73-9cff-cca6dbab5f17"). InnerVolumeSpecName "kube-api-access-4g8c7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 00:49:09.097633 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:09.097567 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2da871b-e678-48d6-b871-2f505ac0e278-kube-api-access-ggkt7" (OuterVolumeSpecName: "kube-api-access-ggkt7") pod "b2da871b-e678-48d6-b871-2f505ac0e278" (UID: "b2da871b-e678-48d6-b871-2f505ac0e278"). InnerVolumeSpecName "kube-api-access-ggkt7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 00:49:09.117347 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:09.117320 2580 generic.go:356] "Generic (PLEG): container finished" podID="b2da871b-e678-48d6-b871-2f505ac0e278" containerID="58cea4007cfbc6cebe6b2b4690c185cd9e16add9acf18a83c27bc5b903ddd73f" exitCode=0 Jul 21 00:49:09.117459 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:09.117360 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zf9br" Jul 21 00:49:09.117459 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:09.117419 2580 scope.go:117] "RemoveContainer" containerID="58cea4007cfbc6cebe6b2b4690c185cd9e16add9acf18a83c27bc5b903ddd73f" Jul 21 00:49:09.119375 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:49:09.119356 2580 kuberuntime_manager.go:649] "Missing actuated resource record" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2flrf" container="manager" Jul 21 00:49:09.120642 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:09.120618 2580 generic.go:356] "Generic (PLEG): container finished" podID="4ad2b86a-ccaf-4f73-9cff-cca6dbab5f17" containerID="c025ced330de95f1f9d379b58c6d78362be332e59aaa089ae479ec43e3cc05d8" exitCode=0 Jul 21 00:49:09.120726 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:09.120653 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9cc4d" Jul 21 00:49:09.127840 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:09.127656 2580 scope.go:117] "RemoveContainer" containerID="58cea4007cfbc6cebe6b2b4690c185cd9e16add9acf18a83c27bc5b903ddd73f" Jul 21 00:49:09.127943 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:49:09.127925 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58cea4007cfbc6cebe6b2b4690c185cd9e16add9acf18a83c27bc5b903ddd73f\": container with ID starting with 58cea4007cfbc6cebe6b2b4690c185cd9e16add9acf18a83c27bc5b903ddd73f not found: ID does not exist" containerID="58cea4007cfbc6cebe6b2b4690c185cd9e16add9acf18a83c27bc5b903ddd73f" Jul 21 00:49:09.127980 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:09.127953 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58cea4007cfbc6cebe6b2b4690c185cd9e16add9acf18a83c27bc5b903ddd73f"} err="failed to get container status \"58cea4007cfbc6cebe6b2b4690c185cd9e16add9acf18a83c27bc5b903ddd73f\": rpc error: code = NotFound desc = could not find container \"58cea4007cfbc6cebe6b2b4690c185cd9e16add9acf18a83c27bc5b903ddd73f\": container with ID starting with 58cea4007cfbc6cebe6b2b4690c185cd9e16add9acf18a83c27bc5b903ddd73f not found: ID does not exist" Jul 21 00:49:09.127980 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:09.127971 2580 scope.go:117] "RemoveContainer" containerID="c025ced330de95f1f9d379b58c6d78362be332e59aaa089ae479ec43e3cc05d8" Jul 21 00:49:09.136564 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:09.136481 2580 scope.go:117] "RemoveContainer" containerID="c025ced330de95f1f9d379b58c6d78362be332e59aaa089ae479ec43e3cc05d8" Jul 21 00:49:09.136852 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:49:09.136778 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c025ced330de95f1f9d379b58c6d78362be332e59aaa089ae479ec43e3cc05d8\": container with ID starting with c025ced330de95f1f9d379b58c6d78362be332e59aaa089ae479ec43e3cc05d8 not found: ID does not exist" containerID="c025ced330de95f1f9d379b58c6d78362be332e59aaa089ae479ec43e3cc05d8" Jul 21 00:49:09.136852 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:09.136812 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c025ced330de95f1f9d379b58c6d78362be332e59aaa089ae479ec43e3cc05d8"} err="failed to get container status \"c025ced330de95f1f9d379b58c6d78362be332e59aaa089ae479ec43e3cc05d8\": rpc error: code = NotFound desc = could not find container \"c025ced330de95f1f9d379b58c6d78362be332e59aaa089ae479ec43e3cc05d8\": container with ID starting with c025ced330de95f1f9d379b58c6d78362be332e59aaa089ae479ec43e3cc05d8 not found: ID does not exist" Jul 21 00:49:09.196285 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:09.196255 2580 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4ad2b86a-ccaf-4f73-9cff-cca6dbab5f17-extensions-socket-volume\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 21 00:49:09.196285 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:09.196281 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4g8c7\" (UniqueName: \"kubernetes.io/projected/4ad2b86a-ccaf-4f73-9cff-cca6dbab5f17-kube-api-access-4g8c7\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 21 00:49:09.196422 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:09.196294 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggkt7\" (UniqueName: \"kubernetes.io/projected/b2da871b-e678-48d6-b871-2f505ac0e278-kube-api-access-ggkt7\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 21 00:49:09.208761 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:09.208734 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-k9zhw" Jul 21 00:49:09.216717 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:09.216637 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8jbtl" Jul 21 00:49:09.222495 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:09.222474 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tgtqn" Jul 21 00:49:09.388726 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:09.388696 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-k9zhw"] Jul 21 00:49:09.392764 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:49:09.392708 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fba9395_dc6c_4360_8b07_b72dd0ff835c.slice/crio-21dd0e0685d913363f7bc06091f3553bdeca2f30d4339c8712ca99534f635f9f WatchSource:0}: Error finding container 21dd0e0685d913363f7bc06091f3553bdeca2f30d4339c8712ca99534f635f9f: Status 404 returned error can't find the container with id 21dd0e0685d913363f7bc06091f3553bdeca2f30d4339c8712ca99534f635f9f Jul 21 00:49:09.617868 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:09.617804 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tgtqn"] Jul 21 00:49:09.620218 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:49:09.620191 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda53c5164_d2ff_43eb_bffa_3dd49d9f0310.slice/crio-f4ac61e9af45fcc697e8dbacbca5777982a9976b8f9b45ad38069a8afa0a51c6 WatchSource:0}: Error finding container f4ac61e9af45fcc697e8dbacbca5777982a9976b8f9b45ad38069a8afa0a51c6: Status 404 returned error can't find the container with id f4ac61e9af45fcc697e8dbacbca5777982a9976b8f9b45ad38069a8afa0a51c6 Jul 21 00:49:09.625143 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:09.625042 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8jbtl"] Jul 21 00:49:09.627603 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:49:09.627483 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b21ebd1_7350_47d0_92d8_c0429488af16.slice/crio-8754bacda9b2fb844a1241e387128dcce6a47b3db56074dfe022e1179e371353 WatchSource:0}: Error finding container 8754bacda9b2fb844a1241e387128dcce6a47b3db56074dfe022e1179e371353: Status 404 returned error can't find the container with id 8754bacda9b2fb844a1241e387128dcce6a47b3db56074dfe022e1179e371353 Jul 21 00:49:10.129304 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:10.129271 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-k9zhw" event={"ID":"4fba9395-dc6c-4360-8b07-b72dd0ff835c","Type":"ContainerStarted","Data":"53b2feafb7f8fa2a72d05357f0eaf1d13edc4bd2c6472a0e5d2b00f010dc9e59"} Jul 21 00:49:10.129757 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:10.129336 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-k9zhw" Jul 21 00:49:10.129757 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:10.129363 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-k9zhw" event={"ID":"4fba9395-dc6c-4360-8b07-b72dd0ff835c","Type":"ContainerStarted","Data":"21dd0e0685d913363f7bc06091f3553bdeca2f30d4339c8712ca99534f635f9f"} Jul 21 00:49:10.131611 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:10.131586 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tgtqn" event={"ID":"a53c5164-d2ff-43eb-bffa-3dd49d9f0310","Type":"ContainerStarted","Data":"c86ecbef2c17420dfd1e09b40a58d0d82d9f991b8125dee539b0c4700a00f2ff"} Jul 21 00:49:10.131729 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:10.131624 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tgtqn" event={"ID":"a53c5164-d2ff-43eb-bffa-3dd49d9f0310","Type":"ContainerStarted","Data":"f4ac61e9af45fcc697e8dbacbca5777982a9976b8f9b45ad38069a8afa0a51c6"} Jul 21 00:49:10.131729 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:10.131650 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tgtqn" Jul 21 00:49:10.133158 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:10.133134 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2flrf" podUID="f34779a0-f034-4195-a5ec-9145f78cea55" containerName="manager" containerID="cri-o://dd4ccc78c61b61085ba897528892203806c5de4ed9f5939bfb5ba8622fb65f34" gracePeriod=2 Jul 21 00:49:10.134720 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:10.134697 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8jbtl" event={"ID":"0b21ebd1-7350-47d0-92d8-c0429488af16","Type":"ContainerStarted","Data":"59b85a114d51996f14d75c53a4d9a65ff62d74e0ad859d739e7cccb138e94f47"} Jul 21 00:49:10.134819 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:10.134737 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8jbtl" event={"ID":"0b21ebd1-7350-47d0-92d8-c0429488af16","Type":"ContainerStarted","Data":"8754bacda9b2fb844a1241e387128dcce6a47b3db56074dfe022e1179e371353"} Jul 21 00:49:10.134819 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:10.134765 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8jbtl" Jul 21 00:49:10.153890 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:10.153846 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-k9zhw" podStartSLOduration=2.153831936 podStartE2EDuration="2.153831936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:49:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:49:10.150672005 +0000 UTC m=+497.756781045" watchObservedRunningTime="2026-07-21 00:49:10.153831936 +0000 UTC m=+497.759940976" Jul 21 00:49:10.176143 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:10.176092 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tgtqn" podStartSLOduration=2.176078386 podStartE2EDuration="2.176078386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:49:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:49:10.1740376 +0000 UTC m=+497.780146640" watchObservedRunningTime="2026-07-21 00:49:10.176078386 +0000 UTC m=+497.782187426" Jul 21 00:49:10.176282 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:49:10.176186 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-2flrf\" is forbidden: User \"system:node:ip-10-0-129-137.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-137.ec2.internal' and this object" podUID="f34779a0-f034-4195-a5ec-9145f78cea55" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2flrf" Jul 21 00:49:10.195835 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:10.195794 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8jbtl" podStartSLOduration=2.195778233 podStartE2EDuration="2.195778233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:49:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:49:10.195078253 +0000 UTC m=+497.801187293" watchObservedRunningTime="2026-07-21 00:49:10.195778233 +0000 UTC m=+497.801887273" Jul 21 00:49:10.382481 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:10.382423 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2flrf" Jul 21 00:49:10.384779 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:49:10.384752 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-2flrf\" is forbidden: User \"system:node:ip-10-0-129-137.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-137.ec2.internal' and this object" podUID="f34779a0-f034-4195-a5ec-9145f78cea55" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2flrf" Jul 21 00:49:10.407886 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:10.407860 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f34779a0-f034-4195-a5ec-9145f78cea55-extensions-socket-volume\") pod \"f34779a0-f034-4195-a5ec-9145f78cea55\" (UID: \"f34779a0-f034-4195-a5ec-9145f78cea55\") " Jul 21 00:49:10.408028 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:10.407966 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t9jwf\" (UniqueName: \"kubernetes.io/projected/f34779a0-f034-4195-a5ec-9145f78cea55-kube-api-access-t9jwf\") pod \"f34779a0-f034-4195-a5ec-9145f78cea55\" (UID: \"f34779a0-f034-4195-a5ec-9145f78cea55\") " Jul 21 00:49:10.408263 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:10.408222 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f34779a0-f034-4195-a5ec-9145f78cea55-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "f34779a0-f034-4195-a5ec-9145f78cea55" (UID: "f34779a0-f034-4195-a5ec-9145f78cea55"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 00:49:10.410003 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:10.409980 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f34779a0-f034-4195-a5ec-9145f78cea55-kube-api-access-t9jwf" (OuterVolumeSpecName: "kube-api-access-t9jwf") pod "f34779a0-f034-4195-a5ec-9145f78cea55" (UID: "f34779a0-f034-4195-a5ec-9145f78cea55"). InnerVolumeSpecName "kube-api-access-t9jwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 00:49:10.509460 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:10.509421 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9jwf\" (UniqueName: \"kubernetes.io/projected/f34779a0-f034-4195-a5ec-9145f78cea55-kube-api-access-t9jwf\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 21 00:49:10.509460 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:10.509452 2580 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f34779a0-f034-4195-a5ec-9145f78cea55-extensions-socket-volume\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 21 00:49:10.912448 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:10.912408 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ad2b86a-ccaf-4f73-9cff-cca6dbab5f17" path="/var/lib/kubelet/pods/4ad2b86a-ccaf-4f73-9cff-cca6dbab5f17/volumes" Jul 21 00:49:10.912794 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:10.912774 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2da871b-e678-48d6-b871-2f505ac0e278" path="/var/lib/kubelet/pods/b2da871b-e678-48d6-b871-2f505ac0e278/volumes" Jul 21 00:49:10.913131 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:10.913114 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f34779a0-f034-4195-a5ec-9145f78cea55" path="/var/lib/kubelet/pods/f34779a0-f034-4195-a5ec-9145f78cea55/volumes" Jul 21 00:49:11.140500 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:11.140464 2580 generic.go:356] "Generic (PLEG): container finished" podID="f34779a0-f034-4195-a5ec-9145f78cea55" containerID="dd4ccc78c61b61085ba897528892203806c5de4ed9f5939bfb5ba8622fb65f34" exitCode=0 Jul 21 00:49:11.140950 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:11.140506 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2flrf" Jul 21 00:49:11.140950 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:11.140568 2580 scope.go:117] "RemoveContainer" containerID="dd4ccc78c61b61085ba897528892203806c5de4ed9f5939bfb5ba8622fb65f34" Jul 21 00:49:11.145691 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:49:11.145661 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-2flrf\" is forbidden: User \"system:node:ip-10-0-129-137.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-137.ec2.internal' and this object" podUID="f34779a0-f034-4195-a5ec-9145f78cea55" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2flrf" Jul 21 00:49:11.148936 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:11.148923 2580 scope.go:117] "RemoveContainer" containerID="dd4ccc78c61b61085ba897528892203806c5de4ed9f5939bfb5ba8622fb65f34" Jul 21 00:49:11.149181 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:49:11.149164 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd4ccc78c61b61085ba897528892203806c5de4ed9f5939bfb5ba8622fb65f34\": container with ID starting with dd4ccc78c61b61085ba897528892203806c5de4ed9f5939bfb5ba8622fb65f34 not found: ID does not exist" containerID="dd4ccc78c61b61085ba897528892203806c5de4ed9f5939bfb5ba8622fb65f34" Jul 21 00:49:11.149258 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:11.149189 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd4ccc78c61b61085ba897528892203806c5de4ed9f5939bfb5ba8622fb65f34"} err="failed to get container status \"dd4ccc78c61b61085ba897528892203806c5de4ed9f5939bfb5ba8622fb65f34\": rpc error: code = NotFound desc = could not find container \"dd4ccc78c61b61085ba897528892203806c5de4ed9f5939bfb5ba8622fb65f34\": container with ID starting with dd4ccc78c61b61085ba897528892203806c5de4ed9f5939bfb5ba8622fb65f34 not found: ID does not exist" Jul 21 00:49:12.911836 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:49:12.911778 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-2flrf\" is forbidden: User \"system:node:ip-10-0-129-137.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-137.ec2.internal' and this object" podUID="f34779a0-f034-4195-a5ec-9145f78cea55" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2flrf" Jul 21 00:49:21.143093 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:21.143057 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tgtqn" Jul 21 00:49:21.143533 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:21.143144 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-k9zhw" Jul 21 00:49:21.143533 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:21.143189 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8jbtl" Jul 21 00:49:21.233478 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:21.233445 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8jbtl"] Jul 21 00:49:21.233685 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:21.233659 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8jbtl" podUID="0b21ebd1-7350-47d0-92d8-c0429488af16" containerName="manager" containerID="cri-o://59b85a114d51996f14d75c53a4d9a65ff62d74e0ad859d739e7cccb138e94f47" gracePeriod=10 Jul 21 00:49:21.472367 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:21.472344 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8jbtl" Jul 21 00:49:21.477539 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:21.477515 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mvc9h"] Jul 21 00:49:21.477923 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:21.477901 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b21ebd1-7350-47d0-92d8-c0429488af16" containerName="manager" Jul 21 00:49:21.477923 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:21.477917 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b21ebd1-7350-47d0-92d8-c0429488af16" containerName="manager" Jul 21 00:49:21.478059 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:21.477987 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b21ebd1-7350-47d0-92d8-c0429488af16" containerName="manager" Jul 21 00:49:21.479768 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:21.479753 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mvc9h" Jul 21 00:49:21.493820 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:21.493795 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mvc9h"] Jul 21 00:49:21.508983 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:21.508959 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0b21ebd1-7350-47d0-92d8-c0429488af16-extensions-socket-volume\") pod \"0b21ebd1-7350-47d0-92d8-c0429488af16\" (UID: \"0b21ebd1-7350-47d0-92d8-c0429488af16\") " Jul 21 00:49:21.509120 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:21.508993 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nvjlt\" (UniqueName: \"kubernetes.io/projected/0b21ebd1-7350-47d0-92d8-c0429488af16-kube-api-access-nvjlt\") pod \"0b21ebd1-7350-47d0-92d8-c0429488af16\" (UID: \"0b21ebd1-7350-47d0-92d8-c0429488af16\") " Jul 21 00:49:21.509316 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:21.509296 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b21ebd1-7350-47d0-92d8-c0429488af16-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "0b21ebd1-7350-47d0-92d8-c0429488af16" (UID: "0b21ebd1-7350-47d0-92d8-c0429488af16"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 00:49:21.511091 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:21.511064 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b21ebd1-7350-47d0-92d8-c0429488af16-kube-api-access-nvjlt" (OuterVolumeSpecName: "kube-api-access-nvjlt") pod "0b21ebd1-7350-47d0-92d8-c0429488af16" (UID: "0b21ebd1-7350-47d0-92d8-c0429488af16"). InnerVolumeSpecName "kube-api-access-nvjlt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 00:49:21.609902 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:21.609864 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/328266ea-0fd8-46ae-a2d1-5653eaa17b4d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-mvc9h\" (UID: \"328266ea-0fd8-46ae-a2d1-5653eaa17b4d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mvc9h" Jul 21 00:49:21.610104 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:21.609989 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqtr4\" (UniqueName: \"kubernetes.io/projected/328266ea-0fd8-46ae-a2d1-5653eaa17b4d-kube-api-access-tqtr4\") pod \"kuadrant-operator-controller-manager-86878c769f-mvc9h\" (UID: \"328266ea-0fd8-46ae-a2d1-5653eaa17b4d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mvc9h" Jul 21 00:49:21.610104 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:21.610087 2580 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0b21ebd1-7350-47d0-92d8-c0429488af16-extensions-socket-volume\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 21 00:49:21.610192 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:21.610111 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvjlt\" (UniqueName: \"kubernetes.io/projected/0b21ebd1-7350-47d0-92d8-c0429488af16-kube-api-access-nvjlt\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 21 00:49:21.711538 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:21.711447 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqtr4\" (UniqueName: \"kubernetes.io/projected/328266ea-0fd8-46ae-a2d1-5653eaa17b4d-kube-api-access-tqtr4\") pod \"kuadrant-operator-controller-manager-86878c769f-mvc9h\" (UID: \"328266ea-0fd8-46ae-a2d1-5653eaa17b4d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mvc9h" Jul 21 00:49:21.711538 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:21.711498 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/328266ea-0fd8-46ae-a2d1-5653eaa17b4d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-mvc9h\" (UID: \"328266ea-0fd8-46ae-a2d1-5653eaa17b4d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mvc9h" Jul 21 00:49:21.711825 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:21.711808 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/328266ea-0fd8-46ae-a2d1-5653eaa17b4d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-mvc9h\" (UID: \"328266ea-0fd8-46ae-a2d1-5653eaa17b4d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mvc9h" Jul 21 00:49:21.726700 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:21.726672 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqtr4\" (UniqueName: \"kubernetes.io/projected/328266ea-0fd8-46ae-a2d1-5653eaa17b4d-kube-api-access-tqtr4\") pod \"kuadrant-operator-controller-manager-86878c769f-mvc9h\" (UID: \"328266ea-0fd8-46ae-a2d1-5653eaa17b4d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mvc9h" Jul 21 00:49:21.789085 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:21.789049 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mvc9h" Jul 21 00:49:21.922399 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:21.922371 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mvc9h"] Jul 21 00:49:21.924478 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:49:21.924455 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod328266ea_0fd8_46ae_a2d1_5653eaa17b4d.slice/crio-9c6afaa4250df859e16f1302bbde6bcf6b35154616509953025be80e10a6f674 WatchSource:0}: Error finding container 9c6afaa4250df859e16f1302bbde6bcf6b35154616509953025be80e10a6f674: Status 404 returned error can't find the container with id 9c6afaa4250df859e16f1302bbde6bcf6b35154616509953025be80e10a6f674 Jul 21 00:49:22.200377 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:22.200345 2580 generic.go:356] "Generic (PLEG): container finished" podID="0b21ebd1-7350-47d0-92d8-c0429488af16" containerID="59b85a114d51996f14d75c53a4d9a65ff62d74e0ad859d739e7cccb138e94f47" exitCode=0 Jul 21 00:49:22.200799 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:22.200391 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8jbtl" Jul 21 00:49:22.200799 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:22.200427 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8jbtl" event={"ID":"0b21ebd1-7350-47d0-92d8-c0429488af16","Type":"ContainerDied","Data":"59b85a114d51996f14d75c53a4d9a65ff62d74e0ad859d739e7cccb138e94f47"} Jul 21 00:49:22.200799 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:22.200474 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8jbtl" event={"ID":"0b21ebd1-7350-47d0-92d8-c0429488af16","Type":"ContainerDied","Data":"8754bacda9b2fb844a1241e387128dcce6a47b3db56074dfe022e1179e371353"} Jul 21 00:49:22.200799 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:22.200500 2580 scope.go:117] "RemoveContainer" containerID="59b85a114d51996f14d75c53a4d9a65ff62d74e0ad859d739e7cccb138e94f47" Jul 21 00:49:22.202399 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:22.202378 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mvc9h" event={"ID":"328266ea-0fd8-46ae-a2d1-5653eaa17b4d","Type":"ContainerStarted","Data":"1dfa6365dcac917081cba504d04307d599dc23c4508517e4a9db85bcd22dcbd2"} Jul 21 00:49:22.202520 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:22.202412 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mvc9h" event={"ID":"328266ea-0fd8-46ae-a2d1-5653eaa17b4d","Type":"ContainerStarted","Data":"9c6afaa4250df859e16f1302bbde6bcf6b35154616509953025be80e10a6f674"} Jul 21 00:49:22.202520 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:22.202508 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mvc9h" Jul 21 00:49:22.210096 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:22.210075 2580 scope.go:117] "RemoveContainer" containerID="59b85a114d51996f14d75c53a4d9a65ff62d74e0ad859d739e7cccb138e94f47" Jul 21 00:49:22.210376 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:49:22.210358 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59b85a114d51996f14d75c53a4d9a65ff62d74e0ad859d739e7cccb138e94f47\": container with ID starting with 59b85a114d51996f14d75c53a4d9a65ff62d74e0ad859d739e7cccb138e94f47 not found: ID does not exist" containerID="59b85a114d51996f14d75c53a4d9a65ff62d74e0ad859d739e7cccb138e94f47" Jul 21 00:49:22.210450 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:22.210384 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59b85a114d51996f14d75c53a4d9a65ff62d74e0ad859d739e7cccb138e94f47"} err="failed to get container status \"59b85a114d51996f14d75c53a4d9a65ff62d74e0ad859d739e7cccb138e94f47\": rpc error: code = NotFound desc = could not find container \"59b85a114d51996f14d75c53a4d9a65ff62d74e0ad859d739e7cccb138e94f47\": container with ID starting with 59b85a114d51996f14d75c53a4d9a65ff62d74e0ad859d739e7cccb138e94f47 not found: ID does not exist" Jul 21 00:49:22.226555 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:22.226515 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mvc9h" podStartSLOduration=1.226502665 podStartE2EDuration="1.226502665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:49:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:49:22.225089677 +0000 UTC m=+509.831198788" watchObservedRunningTime="2026-07-21 00:49:22.226502665 +0000 UTC m=+509.832611705" Jul 21 00:49:22.243758 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:22.243705 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8jbtl"] Jul 21 00:49:22.248830 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:22.248803 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8jbtl"] Jul 21 00:49:22.912510 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:22.912478 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b21ebd1-7350-47d0-92d8-c0429488af16" path="/var/lib/kubelet/pods/0b21ebd1-7350-47d0-92d8-c0429488af16/volumes" Jul 21 00:49:33.210875 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:33.210841 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mvc9h" Jul 21 00:49:33.262960 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:33.262917 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-k9zhw"] Jul 21 00:49:33.263374 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:33.263345 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-k9zhw" podUID="4fba9395-dc6c-4360-8b07-b72dd0ff835c" containerName="manager" containerID="cri-o://53b2feafb7f8fa2a72d05357f0eaf1d13edc4bd2c6472a0e5d2b00f010dc9e59" gracePeriod=10 Jul 21 00:49:33.503374 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:33.503344 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-k9zhw" Jul 21 00:49:33.613427 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:33.613403 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-znzkd\" (UniqueName: \"kubernetes.io/projected/4fba9395-dc6c-4360-8b07-b72dd0ff835c-kube-api-access-znzkd\") pod \"4fba9395-dc6c-4360-8b07-b72dd0ff835c\" (UID: \"4fba9395-dc6c-4360-8b07-b72dd0ff835c\") " Jul 21 00:49:33.613580 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:33.613443 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4fba9395-dc6c-4360-8b07-b72dd0ff835c-extensions-socket-volume\") pod \"4fba9395-dc6c-4360-8b07-b72dd0ff835c\" (UID: \"4fba9395-dc6c-4360-8b07-b72dd0ff835c\") " Jul 21 00:49:33.613815 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:33.613791 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fba9395-dc6c-4360-8b07-b72dd0ff835c-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "4fba9395-dc6c-4360-8b07-b72dd0ff835c" (UID: "4fba9395-dc6c-4360-8b07-b72dd0ff835c"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 00:49:33.615440 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:33.615413 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fba9395-dc6c-4360-8b07-b72dd0ff835c-kube-api-access-znzkd" (OuterVolumeSpecName: "kube-api-access-znzkd") pod "4fba9395-dc6c-4360-8b07-b72dd0ff835c" (UID: "4fba9395-dc6c-4360-8b07-b72dd0ff835c"). InnerVolumeSpecName "kube-api-access-znzkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 00:49:33.713991 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:33.713969 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znzkd\" (UniqueName: \"kubernetes.io/projected/4fba9395-dc6c-4360-8b07-b72dd0ff835c-kube-api-access-znzkd\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 21 00:49:33.713991 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:33.713991 2580 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4fba9395-dc6c-4360-8b07-b72dd0ff835c-extensions-socket-volume\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 21 00:49:34.267442 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:34.267405 2580 generic.go:356] "Generic (PLEG): container finished" podID="4fba9395-dc6c-4360-8b07-b72dd0ff835c" containerID="53b2feafb7f8fa2a72d05357f0eaf1d13edc4bd2c6472a0e5d2b00f010dc9e59" exitCode=0 Jul 21 00:49:34.267442 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:34.267445 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-k9zhw" Jul 21 00:49:34.267911 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:34.267480 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-k9zhw" event={"ID":"4fba9395-dc6c-4360-8b07-b72dd0ff835c","Type":"ContainerDied","Data":"53b2feafb7f8fa2a72d05357f0eaf1d13edc4bd2c6472a0e5d2b00f010dc9e59"} Jul 21 00:49:34.267911 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:34.267546 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-k9zhw" event={"ID":"4fba9395-dc6c-4360-8b07-b72dd0ff835c","Type":"ContainerDied","Data":"21dd0e0685d913363f7bc06091f3553bdeca2f30d4339c8712ca99534f635f9f"} Jul 21 00:49:34.267911 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:34.267574 2580 scope.go:117] "RemoveContainer" containerID="53b2feafb7f8fa2a72d05357f0eaf1d13edc4bd2c6472a0e5d2b00f010dc9e59" Jul 21 00:49:34.277602 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:34.277587 2580 scope.go:117] "RemoveContainer" containerID="53b2feafb7f8fa2a72d05357f0eaf1d13edc4bd2c6472a0e5d2b00f010dc9e59" Jul 21 00:49:34.277901 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:49:34.277882 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53b2feafb7f8fa2a72d05357f0eaf1d13edc4bd2c6472a0e5d2b00f010dc9e59\": container with ID starting with 53b2feafb7f8fa2a72d05357f0eaf1d13edc4bd2c6472a0e5d2b00f010dc9e59 not found: ID does not exist" containerID="53b2feafb7f8fa2a72d05357f0eaf1d13edc4bd2c6472a0e5d2b00f010dc9e59" Jul 21 00:49:34.277983 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:34.277913 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53b2feafb7f8fa2a72d05357f0eaf1d13edc4bd2c6472a0e5d2b00f010dc9e59"} err="failed to get container status \"53b2feafb7f8fa2a72d05357f0eaf1d13edc4bd2c6472a0e5d2b00f010dc9e59\": rpc error: code = NotFound desc = could not find container \"53b2feafb7f8fa2a72d05357f0eaf1d13edc4bd2c6472a0e5d2b00f010dc9e59\": container with ID starting with 53b2feafb7f8fa2a72d05357f0eaf1d13edc4bd2c6472a0e5d2b00f010dc9e59 not found: ID does not exist" Jul 21 00:49:34.305635 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:34.305611 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-k9zhw"] Jul 21 00:49:34.309790 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:34.309768 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-k9zhw"] Jul 21 00:49:34.917071 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:34.917035 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4fba9395-dc6c-4360-8b07-b72dd0ff835c" path="/var/lib/kubelet/pods/4fba9395-dc6c-4360-8b07-b72dd0ff835c/volumes" Jul 21 00:49:52.088606 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:52.088534 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-mvfdz"] Jul 21 00:49:52.089031 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:52.088881 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fba9395-dc6c-4360-8b07-b72dd0ff835c" containerName="manager" Jul 21 00:49:52.089031 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:52.088891 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fba9395-dc6c-4360-8b07-b72dd0ff835c" containerName="manager" Jul 21 00:49:52.089031 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:52.088959 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fba9395-dc6c-4360-8b07-b72dd0ff835c" containerName="manager" Jul 21 00:49:52.091029 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:52.091013 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-mvfdz" Jul 21 00:49:52.093760 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:52.093734 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-d5rtl\"" Jul 21 00:49:52.093910 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:52.093775 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 21 00:49:52.102517 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:52.102498 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-mvfdz"] Jul 21 00:49:52.187297 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:52.187257 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-mvfdz"] Jul 21 00:49:52.276657 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:52.276624 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vst49\" (UniqueName: \"kubernetes.io/projected/2a8fa0f2-17d6-4c18-aa16-dc668126826e-kube-api-access-vst49\") pod \"limitador-limitador-64c94fd64f-mvfdz\" (UID: \"2a8fa0f2-17d6-4c18-aa16-dc668126826e\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-mvfdz" Jul 21 00:49:52.276812 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:52.276667 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2a8fa0f2-17d6-4c18-aa16-dc668126826e-config-file\") pod \"limitador-limitador-64c94fd64f-mvfdz\" (UID: \"2a8fa0f2-17d6-4c18-aa16-dc668126826e\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-mvfdz" Jul 21 00:49:52.377627 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:52.377549 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vst49\" (UniqueName: \"kubernetes.io/projected/2a8fa0f2-17d6-4c18-aa16-dc668126826e-kube-api-access-vst49\") pod \"limitador-limitador-64c94fd64f-mvfdz\" (UID: \"2a8fa0f2-17d6-4c18-aa16-dc668126826e\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-mvfdz" Jul 21 00:49:52.377627 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:52.377587 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2a8fa0f2-17d6-4c18-aa16-dc668126826e-config-file\") pod \"limitador-limitador-64c94fd64f-mvfdz\" (UID: \"2a8fa0f2-17d6-4c18-aa16-dc668126826e\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-mvfdz" Jul 21 00:49:52.378167 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:52.378149 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2a8fa0f2-17d6-4c18-aa16-dc668126826e-config-file\") pod \"limitador-limitador-64c94fd64f-mvfdz\" (UID: \"2a8fa0f2-17d6-4c18-aa16-dc668126826e\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-mvfdz" Jul 21 00:49:52.389636 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:52.389600 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vst49\" (UniqueName: \"kubernetes.io/projected/2a8fa0f2-17d6-4c18-aa16-dc668126826e-kube-api-access-vst49\") pod \"limitador-limitador-64c94fd64f-mvfdz\" (UID: \"2a8fa0f2-17d6-4c18-aa16-dc668126826e\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-mvfdz" Jul 21 00:49:52.401313 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:52.401291 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-mvfdz" Jul 21 00:49:52.531092 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:52.531066 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-mvfdz"] Jul 21 00:49:52.533130 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:49:52.533103 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a8fa0f2_17d6_4c18_aa16_dc668126826e.slice/crio-8b5e3f01135b948c84c90c935ee4d8276f67888d309ed7f45cf1ac6be3634681 WatchSource:0}: Error finding container 8b5e3f01135b948c84c90c935ee4d8276f67888d309ed7f45cf1ac6be3634681: Status 404 returned error can't find the container with id 8b5e3f01135b948c84c90c935ee4d8276f67888d309ed7f45cf1ac6be3634681 Jul 21 00:49:52.866757 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:52.866724 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-x9cpf"] Jul 21 00:49:52.870544 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:52.870520 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-x9cpf" Jul 21 00:49:52.873008 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:52.872987 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-vm5xq\"" Jul 21 00:49:52.877640 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:52.877463 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-x9cpf"] Jul 21 00:49:52.972109 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:52.972081 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-k9xc2"] Jul 21 00:49:52.974772 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:52.974755 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-k9xc2" Jul 21 00:49:52.982480 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:52.982458 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-k9xc2"] Jul 21 00:49:52.983368 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:52.983345 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rksrl\" (UniqueName: \"kubernetes.io/projected/d8af90df-20c8-4d74-8d2d-2b2a824e1376-kube-api-access-rksrl\") pod \"authorino-58cd955675-x9cpf\" (UID: \"d8af90df-20c8-4d74-8d2d-2b2a824e1376\") " pod="kuadrant-system/authorino-58cd955675-x9cpf" Jul 21 00:49:53.084015 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:53.083978 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rksrl\" (UniqueName: \"kubernetes.io/projected/d8af90df-20c8-4d74-8d2d-2b2a824e1376-kube-api-access-rksrl\") pod \"authorino-58cd955675-x9cpf\" (UID: \"d8af90df-20c8-4d74-8d2d-2b2a824e1376\") " pod="kuadrant-system/authorino-58cd955675-x9cpf" Jul 21 00:49:53.084186 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:53.084092 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbbrf\" (UniqueName: \"kubernetes.io/projected/3965ce93-f413-456d-9b8d-5ce4a43a2b7b-kube-api-access-zbbrf\") pod \"authorino-75d556bd77-k9xc2\" (UID: \"3965ce93-f413-456d-9b8d-5ce4a43a2b7b\") " pod="kuadrant-system/authorino-75d556bd77-k9xc2" Jul 21 00:49:53.092631 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:53.092608 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rksrl\" (UniqueName: \"kubernetes.io/projected/d8af90df-20c8-4d74-8d2d-2b2a824e1376-kube-api-access-rksrl\") pod \"authorino-58cd955675-x9cpf\" (UID: \"d8af90df-20c8-4d74-8d2d-2b2a824e1376\") " pod="kuadrant-system/authorino-58cd955675-x9cpf" Jul 21 00:49:53.184761 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:53.184735 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbbrf\" (UniqueName: \"kubernetes.io/projected/3965ce93-f413-456d-9b8d-5ce4a43a2b7b-kube-api-access-zbbrf\") pod \"authorino-75d556bd77-k9xc2\" (UID: \"3965ce93-f413-456d-9b8d-5ce4a43a2b7b\") " pod="kuadrant-system/authorino-75d556bd77-k9xc2" Jul 21 00:49:53.189641 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:53.189616 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-x9cpf" Jul 21 00:49:53.193497 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:53.193456 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbbrf\" (UniqueName: \"kubernetes.io/projected/3965ce93-f413-456d-9b8d-5ce4a43a2b7b-kube-api-access-zbbrf\") pod \"authorino-75d556bd77-k9xc2\" (UID: \"3965ce93-f413-456d-9b8d-5ce4a43a2b7b\") " pod="kuadrant-system/authorino-75d556bd77-k9xc2" Jul 21 00:49:53.284421 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:53.284174 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-k9xc2" Jul 21 00:49:53.360387 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:53.359832 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-x9cpf"] Jul 21 00:49:53.363312 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:49:53.363272 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd8af90df_20c8_4d74_8d2d_2b2a824e1376.slice/crio-1efa5e9ac54d18a67578c11a8d8ada9e4b19466801c8f84457cdf2c85e4c398a WatchSource:0}: Error finding container 1efa5e9ac54d18a67578c11a8d8ada9e4b19466801c8f84457cdf2c85e4c398a: Status 404 returned error can't find the container with id 1efa5e9ac54d18a67578c11a8d8ada9e4b19466801c8f84457cdf2c85e4c398a Jul 21 00:49:53.369284 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:53.369253 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-mvfdz" event={"ID":"2a8fa0f2-17d6-4c18-aa16-dc668126826e","Type":"ContainerStarted","Data":"8b5e3f01135b948c84c90c935ee4d8276f67888d309ed7f45cf1ac6be3634681"} Jul 21 00:49:53.494609 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:53.494575 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-k9xc2"] Jul 21 00:49:54.375811 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:54.375770 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-x9cpf" event={"ID":"d8af90df-20c8-4d74-8d2d-2b2a824e1376","Type":"ContainerStarted","Data":"1efa5e9ac54d18a67578c11a8d8ada9e4b19466801c8f84457cdf2c85e4c398a"} Jul 21 00:49:54.377089 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:54.377060 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-k9xc2" event={"ID":"3965ce93-f413-456d-9b8d-5ce4a43a2b7b","Type":"ContainerStarted","Data":"610a30c7265b2b66911ceba07ef030882cbf7a0d023cab4a3f669629a85145e8"} Jul 21 00:49:56.395926 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:56.395888 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-mvfdz" event={"ID":"2a8fa0f2-17d6-4c18-aa16-dc668126826e","Type":"ContainerStarted","Data":"0c77868a469597c3a272e351ca46f8267c43d3e7d10495719806923b1eccc559"} Jul 21 00:49:56.396420 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:56.395994 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-mvfdz" Jul 21 00:49:56.414379 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:56.414330 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-mvfdz" podStartSLOduration=1.402882157 podStartE2EDuration="4.414315457s" totalImagesPullingTime="3.0114333s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:49:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:49:56.412425239 +0000 UTC m=+544.018534278" watchObservedRunningTime="2026-07-21 00:49:56.414315457 +0000 UTC m=+544.020424545" Jul 21 00:49:58.408340 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:58.408306 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-x9cpf" event={"ID":"d8af90df-20c8-4d74-8d2d-2b2a824e1376","Type":"ContainerStarted","Data":"186e30c13f7ea62b206f72b4651ce3a60c1229af3a96f4ec20dc6b04ddffbf2b"} Jul 21 00:49:58.409732 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:58.409708 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-k9xc2" event={"ID":"3965ce93-f413-456d-9b8d-5ce4a43a2b7b","Type":"ContainerStarted","Data":"e78f96cbe340acd7adc5eb7adda1ad927b061b95a4f7521be1bd1018948c9386"} Jul 21 00:49:58.423311 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:58.423266 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-x9cpf" podStartSLOduration=2.064417191 podStartE2EDuration="6.42325449s" totalImagesPullingTime="4.358837299s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:49:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:49:58.422026657 +0000 UTC m=+546.028135720" watchObservedRunningTime="2026-07-21 00:49:58.42325449 +0000 UTC m=+546.029363527" Jul 21 00:49:58.458063 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:58.458016 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-k9xc2" podStartSLOduration=2.222321667 podStartE2EDuration="6.45800028s" totalImagesPullingTime="4.235678613s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:49:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:49:58.434739124 +0000 UTC m=+546.040848164" watchObservedRunningTime="2026-07-21 00:49:58.45800028 +0000 UTC m=+546.064109319" Jul 21 00:49:58.458371 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:49:58.458353 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-x9cpf"] Jul 21 00:50:00.419138 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:00.419101 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-x9cpf" podUID="d8af90df-20c8-4d74-8d2d-2b2a824e1376" containerName="authorino" containerID="cri-o://186e30c13f7ea62b206f72b4651ce3a60c1229af3a96f4ec20dc6b04ddffbf2b" gracePeriod=30 Jul 21 00:50:00.659903 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:00.659881 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-x9cpf" Jul 21 00:50:00.755136 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:00.755056 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rksrl\" (UniqueName: \"kubernetes.io/projected/d8af90df-20c8-4d74-8d2d-2b2a824e1376-kube-api-access-rksrl\") pod \"d8af90df-20c8-4d74-8d2d-2b2a824e1376\" (UID: \"d8af90df-20c8-4d74-8d2d-2b2a824e1376\") " Jul 21 00:50:00.756980 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:00.756957 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8af90df-20c8-4d74-8d2d-2b2a824e1376-kube-api-access-rksrl" (OuterVolumeSpecName: "kube-api-access-rksrl") pod "d8af90df-20c8-4d74-8d2d-2b2a824e1376" (UID: "d8af90df-20c8-4d74-8d2d-2b2a824e1376"). InnerVolumeSpecName "kube-api-access-rksrl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 00:50:00.856754 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:00.856717 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rksrl\" (UniqueName: \"kubernetes.io/projected/d8af90df-20c8-4d74-8d2d-2b2a824e1376-kube-api-access-rksrl\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 21 00:50:01.425110 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:01.425076 2580 generic.go:356] "Generic (PLEG): container finished" podID="d8af90df-20c8-4d74-8d2d-2b2a824e1376" containerID="186e30c13f7ea62b206f72b4651ce3a60c1229af3a96f4ec20dc6b04ddffbf2b" exitCode=0 Jul 21 00:50:01.425667 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:01.425118 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-x9cpf" Jul 21 00:50:01.425667 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:01.425170 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-x9cpf" event={"ID":"d8af90df-20c8-4d74-8d2d-2b2a824e1376","Type":"ContainerDied","Data":"186e30c13f7ea62b206f72b4651ce3a60c1229af3a96f4ec20dc6b04ddffbf2b"} Jul 21 00:50:01.425667 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:01.425224 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-x9cpf" event={"ID":"d8af90df-20c8-4d74-8d2d-2b2a824e1376","Type":"ContainerDied","Data":"1efa5e9ac54d18a67578c11a8d8ada9e4b19466801c8f84457cdf2c85e4c398a"} Jul 21 00:50:01.425667 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:01.425266 2580 scope.go:117] "RemoveContainer" containerID="186e30c13f7ea62b206f72b4651ce3a60c1229af3a96f4ec20dc6b04ddffbf2b" Jul 21 00:50:01.433996 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:01.433978 2580 scope.go:117] "RemoveContainer" containerID="186e30c13f7ea62b206f72b4651ce3a60c1229af3a96f4ec20dc6b04ddffbf2b" Jul 21 00:50:01.434294 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:50:01.434276 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"186e30c13f7ea62b206f72b4651ce3a60c1229af3a96f4ec20dc6b04ddffbf2b\": container with ID starting with 186e30c13f7ea62b206f72b4651ce3a60c1229af3a96f4ec20dc6b04ddffbf2b not found: ID does not exist" containerID="186e30c13f7ea62b206f72b4651ce3a60c1229af3a96f4ec20dc6b04ddffbf2b" Jul 21 00:50:01.434446 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:01.434303 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"186e30c13f7ea62b206f72b4651ce3a60c1229af3a96f4ec20dc6b04ddffbf2b"} err="failed to get container status \"186e30c13f7ea62b206f72b4651ce3a60c1229af3a96f4ec20dc6b04ddffbf2b\": rpc error: code = NotFound desc = could not find container \"186e30c13f7ea62b206f72b4651ce3a60c1229af3a96f4ec20dc6b04ddffbf2b\": container with ID starting with 186e30c13f7ea62b206f72b4651ce3a60c1229af3a96f4ec20dc6b04ddffbf2b not found: ID does not exist" Jul 21 00:50:01.445545 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:01.445523 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-x9cpf"] Jul 21 00:50:01.450787 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:01.450768 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-x9cpf"] Jul 21 00:50:02.912370 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:02.912337 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8af90df-20c8-4d74-8d2d-2b2a824e1376" path="/var/lib/kubelet/pods/d8af90df-20c8-4d74-8d2d-2b2a824e1376/volumes" Jul 21 00:50:07.016147 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:07.016114 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-mvfdz"] Jul 21 00:50:07.016698 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:07.016569 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-mvfdz" podUID="2a8fa0f2-17d6-4c18-aa16-dc668126826e" containerName="limitador" containerID="cri-o://0c77868a469597c3a272e351ca46f8267c43d3e7d10495719806923b1eccc559" gracePeriod=30 Jul 21 00:50:07.017320 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:07.017300 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-mvfdz" Jul 21 00:50:07.454848 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:07.454812 2580 generic.go:356] "Generic (PLEG): container finished" podID="2a8fa0f2-17d6-4c18-aa16-dc668126826e" containerID="0c77868a469597c3a272e351ca46f8267c43d3e7d10495719806923b1eccc559" exitCode=0 Jul 21 00:50:07.454961 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:07.454883 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-mvfdz" event={"ID":"2a8fa0f2-17d6-4c18-aa16-dc668126826e","Type":"ContainerDied","Data":"0c77868a469597c3a272e351ca46f8267c43d3e7d10495719806923b1eccc559"} Jul 21 00:50:07.975513 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:07.975487 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-mvfdz" Jul 21 00:50:08.022665 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:08.022640 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2a8fa0f2-17d6-4c18-aa16-dc668126826e-config-file\") pod \"2a8fa0f2-17d6-4c18-aa16-dc668126826e\" (UID: \"2a8fa0f2-17d6-4c18-aa16-dc668126826e\") " Jul 21 00:50:08.023077 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:08.022703 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vst49\" (UniqueName: \"kubernetes.io/projected/2a8fa0f2-17d6-4c18-aa16-dc668126826e-kube-api-access-vst49\") pod \"2a8fa0f2-17d6-4c18-aa16-dc668126826e\" (UID: \"2a8fa0f2-17d6-4c18-aa16-dc668126826e\") " Jul 21 00:50:08.023077 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:08.023049 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a8fa0f2-17d6-4c18-aa16-dc668126826e-config-file" (OuterVolumeSpecName: "config-file") pod "2a8fa0f2-17d6-4c18-aa16-dc668126826e" (UID: "2a8fa0f2-17d6-4c18-aa16-dc668126826e"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 00:50:08.024739 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:08.024714 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a8fa0f2-17d6-4c18-aa16-dc668126826e-kube-api-access-vst49" (OuterVolumeSpecName: "kube-api-access-vst49") pod "2a8fa0f2-17d6-4c18-aa16-dc668126826e" (UID: "2a8fa0f2-17d6-4c18-aa16-dc668126826e"). InnerVolumeSpecName "kube-api-access-vst49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 00:50:08.123420 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:08.123360 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vst49\" (UniqueName: \"kubernetes.io/projected/2a8fa0f2-17d6-4c18-aa16-dc668126826e-kube-api-access-vst49\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 21 00:50:08.123420 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:08.123381 2580 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2a8fa0f2-17d6-4c18-aa16-dc668126826e-config-file\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 21 00:50:08.461078 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:08.461010 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-mvfdz" Jul 21 00:50:08.461078 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:08.461055 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-mvfdz" event={"ID":"2a8fa0f2-17d6-4c18-aa16-dc668126826e","Type":"ContainerDied","Data":"8b5e3f01135b948c84c90c935ee4d8276f67888d309ed7f45cf1ac6be3634681"} Jul 21 00:50:08.461282 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:08.461112 2580 scope.go:117] "RemoveContainer" containerID="0c77868a469597c3a272e351ca46f8267c43d3e7d10495719806923b1eccc559" Jul 21 00:50:08.481625 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:08.481599 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-mvfdz"] Jul 21 00:50:08.485125 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:08.485104 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-mvfdz"] Jul 21 00:50:08.912193 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:08.912158 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a8fa0f2-17d6-4c18-aa16-dc668126826e" path="/var/lib/kubelet/pods/2a8fa0f2-17d6-4c18-aa16-dc668126826e/volumes" Jul 21 00:50:11.674141 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:11.674105 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-vkmt7"] Jul 21 00:50:11.674656 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:11.674637 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a8fa0f2-17d6-4c18-aa16-dc668126826e" containerName="limitador" Jul 21 00:50:11.675205 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:11.674659 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a8fa0f2-17d6-4c18-aa16-dc668126826e" containerName="limitador" Jul 21 00:50:11.675205 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:11.674673 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8af90df-20c8-4d74-8d2d-2b2a824e1376" containerName="authorino" Jul 21 00:50:11.675205 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:11.674683 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8af90df-20c8-4d74-8d2d-2b2a824e1376" containerName="authorino" Jul 21 00:50:11.675205 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:11.674778 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a8fa0f2-17d6-4c18-aa16-dc668126826e" containerName="limitador" Jul 21 00:50:11.675205 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:11.674790 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8af90df-20c8-4d74-8d2d-2b2a824e1376" containerName="authorino" Jul 21 00:50:11.679128 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:11.679108 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-vkmt7" Jul 21 00:50:11.681669 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:11.681647 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-zcq4w\"" Jul 21 00:50:11.681669 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:11.681664 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 21 00:50:11.682938 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:11.682914 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 21 00:50:11.683030 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:11.682942 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 21 00:50:11.687773 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:11.687752 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-vkmt7"] Jul 21 00:50:11.757092 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:11.757059 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggc8z\" (UniqueName: \"kubernetes.io/projected/983b7ef4-755f-467b-a0da-f5683974deb4-kube-api-access-ggc8z\") pod \"postgres-69fb4594cf-vkmt7\" (UID: \"983b7ef4-755f-467b-a0da-f5683974deb4\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-vkmt7" Jul 21 00:50:11.757092 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:11.757092 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/983b7ef4-755f-467b-a0da-f5683974deb4-data\") pod \"postgres-69fb4594cf-vkmt7\" (UID: \"983b7ef4-755f-467b-a0da-f5683974deb4\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-vkmt7" Jul 21 00:50:11.858334 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:11.858300 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggc8z\" (UniqueName: \"kubernetes.io/projected/983b7ef4-755f-467b-a0da-f5683974deb4-kube-api-access-ggc8z\") pod \"postgres-69fb4594cf-vkmt7\" (UID: \"983b7ef4-755f-467b-a0da-f5683974deb4\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-vkmt7" Jul 21 00:50:11.858334 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:11.858338 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/983b7ef4-755f-467b-a0da-f5683974deb4-data\") pod \"postgres-69fb4594cf-vkmt7\" (UID: \"983b7ef4-755f-467b-a0da-f5683974deb4\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-vkmt7" Jul 21 00:50:11.858721 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:11.858706 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/983b7ef4-755f-467b-a0da-f5683974deb4-data\") pod \"postgres-69fb4594cf-vkmt7\" (UID: \"983b7ef4-755f-467b-a0da-f5683974deb4\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-vkmt7" Jul 21 00:50:11.867042 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:11.867023 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggc8z\" (UniqueName: \"kubernetes.io/projected/983b7ef4-755f-467b-a0da-f5683974deb4-kube-api-access-ggc8z\") pod \"postgres-69fb4594cf-vkmt7\" (UID: \"983b7ef4-755f-467b-a0da-f5683974deb4\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-vkmt7" Jul 21 00:50:11.990287 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:11.990181 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-vkmt7" Jul 21 00:50:12.119278 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:12.119253 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-vkmt7"] Jul 21 00:50:12.121122 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:50:12.121090 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod983b7ef4_755f_467b_a0da_f5683974deb4.slice/crio-4d9c63a40b209f7fbc80b4609daa6a795b2df8d44ec2964ec67f87ac545e1939 WatchSource:0}: Error finding container 4d9c63a40b209f7fbc80b4609daa6a795b2df8d44ec2964ec67f87ac545e1939: Status 404 returned error can't find the container with id 4d9c63a40b209f7fbc80b4609daa6a795b2df8d44ec2964ec67f87ac545e1939 Jul 21 00:50:12.481288 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:12.481226 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-vkmt7" event={"ID":"983b7ef4-755f-467b-a0da-f5683974deb4","Type":"ContainerStarted","Data":"4d9c63a40b209f7fbc80b4609daa6a795b2df8d44ec2964ec67f87ac545e1939"} Jul 21 00:50:19.522916 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:19.522829 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-vkmt7" event={"ID":"983b7ef4-755f-467b-a0da-f5683974deb4","Type":"ContainerStarted","Data":"8dda067a73ed73d1432eba9bbf4db6ee2ba3b721460bc67fa257e1fbb7d0f80f"} Jul 21 00:50:19.523330 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:19.522964 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-vkmt7" Jul 21 00:50:19.539393 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:19.539351 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-vkmt7" podStartSLOduration=1.526774359 podStartE2EDuration="8.539336535s" totalImagesPullingTime="7.012562176s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:50:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:50:19.537275746 +0000 UTC m=+567.143384788" watchObservedRunningTime="2026-07-21 00:50:19.539336535 +0000 UTC m=+567.145445574" Jul 21 00:50:25.557396 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:25.557366 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-vkmt7" Jul 21 00:50:26.167577 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:26.167542 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-7dwxq"] Jul 21 00:50:26.172994 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:26.172970 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-7dwxq" Jul 21 00:50:26.175579 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:26.175555 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-7dwxq"] Jul 21 00:50:26.283605 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:26.283576 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lcbt\" (UniqueName: \"kubernetes.io/projected/16b04187-4174-470c-b497-90c41305ea64-kube-api-access-5lcbt\") pod \"authorino-5d4b67c446-7dwxq\" (UID: \"16b04187-4174-470c-b497-90c41305ea64\") " pod="kuadrant-system/authorino-5d4b67c446-7dwxq" Jul 21 00:50:26.294366 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:26.294330 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-7dwxq"] Jul 21 00:50:26.294617 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:50:26.294592 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-5lcbt], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5d4b67c446-7dwxq" podUID="16b04187-4174-470c-b497-90c41305ea64" Jul 21 00:50:26.315825 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:26.315796 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-76877994-64hqh"] Jul 21 00:50:26.319281 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:26.319263 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-64hqh" Jul 21 00:50:26.321759 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:26.321738 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 21 00:50:26.328634 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:26.328611 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-76877994-64hqh"] Jul 21 00:50:26.384338 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:26.384305 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f5ace73b-999c-4930-8b0a-d58df058503d-tls-cert\") pod \"authorino-76877994-64hqh\" (UID: \"f5ace73b-999c-4930-8b0a-d58df058503d\") " pod="kuadrant-system/authorino-76877994-64hqh" Jul 21 00:50:26.384496 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:26.384361 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lcbt\" (UniqueName: \"kubernetes.io/projected/16b04187-4174-470c-b497-90c41305ea64-kube-api-access-5lcbt\") pod \"authorino-5d4b67c446-7dwxq\" (UID: \"16b04187-4174-470c-b497-90c41305ea64\") " pod="kuadrant-system/authorino-5d4b67c446-7dwxq" Jul 21 00:50:26.384561 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:26.384490 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjj6t\" (UniqueName: \"kubernetes.io/projected/f5ace73b-999c-4930-8b0a-d58df058503d-kube-api-access-fjj6t\") pod \"authorino-76877994-64hqh\" (UID: \"f5ace73b-999c-4930-8b0a-d58df058503d\") " pod="kuadrant-system/authorino-76877994-64hqh" Jul 21 00:50:26.392331 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:26.392306 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lcbt\" (UniqueName: \"kubernetes.io/projected/16b04187-4174-470c-b497-90c41305ea64-kube-api-access-5lcbt\") pod \"authorino-5d4b67c446-7dwxq\" (UID: \"16b04187-4174-470c-b497-90c41305ea64\") " pod="kuadrant-system/authorino-5d4b67c446-7dwxq" Jul 21 00:50:26.481790 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:26.481718 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-64hqh"] Jul 21 00:50:26.482003 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:50:26.481985 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-fjj6t tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-76877994-64hqh" podUID="f5ace73b-999c-4930-8b0a-d58df058503d" Jul 21 00:50:26.485136 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:26.485105 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f5ace73b-999c-4930-8b0a-d58df058503d-tls-cert\") pod \"authorino-76877994-64hqh\" (UID: \"f5ace73b-999c-4930-8b0a-d58df058503d\") " pod="kuadrant-system/authorino-76877994-64hqh" Jul 21 00:50:26.485282 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:26.485183 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjj6t\" (UniqueName: \"kubernetes.io/projected/f5ace73b-999c-4930-8b0a-d58df058503d-kube-api-access-fjj6t\") pod \"authorino-76877994-64hqh\" (UID: \"f5ace73b-999c-4930-8b0a-d58df058503d\") " pod="kuadrant-system/authorino-76877994-64hqh" Jul 21 00:50:26.487445 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:26.487425 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f5ace73b-999c-4930-8b0a-d58df058503d-tls-cert\") pod \"authorino-76877994-64hqh\" (UID: \"f5ace73b-999c-4930-8b0a-d58df058503d\") " pod="kuadrant-system/authorino-76877994-64hqh" Jul 21 00:50:26.495562 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:26.495541 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjj6t\" (UniqueName: \"kubernetes.io/projected/f5ace73b-999c-4930-8b0a-d58df058503d-kube-api-access-fjj6t\") pod \"authorino-76877994-64hqh\" (UID: \"f5ace73b-999c-4930-8b0a-d58df058503d\") " pod="kuadrant-system/authorino-76877994-64hqh" Jul 21 00:50:26.505718 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:26.505694 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-769465ddbd-v46sv"] Jul 21 00:50:26.512488 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:26.510921 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-769465ddbd-v46sv" Jul 21 00:50:26.515871 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:26.515844 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-769465ddbd-v46sv"] Jul 21 00:50:26.557453 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:26.557425 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-64hqh" Jul 21 00:50:26.557875 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:26.557432 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-7dwxq" Jul 21 00:50:26.564589 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:26.564568 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-64hqh" Jul 21 00:50:26.567124 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:26.567107 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-7dwxq" Jul 21 00:50:26.585653 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:26.585625 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a23ae0de-9373-4024-aa25-4aeb5405243d-tls-cert\") pod \"authorino-769465ddbd-v46sv\" (UID: \"a23ae0de-9373-4024-aa25-4aeb5405243d\") " pod="kuadrant-system/authorino-769465ddbd-v46sv" Jul 21 00:50:26.585809 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:26.585791 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7st7\" (UniqueName: \"kubernetes.io/projected/a23ae0de-9373-4024-aa25-4aeb5405243d-kube-api-access-p7st7\") pod \"authorino-769465ddbd-v46sv\" (UID: \"a23ae0de-9373-4024-aa25-4aeb5405243d\") " pod="kuadrant-system/authorino-769465ddbd-v46sv" Jul 21 00:50:26.686763 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:26.686737 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5lcbt\" (UniqueName: \"kubernetes.io/projected/16b04187-4174-470c-b497-90c41305ea64-kube-api-access-5lcbt\") pod \"16b04187-4174-470c-b497-90c41305ea64\" (UID: \"16b04187-4174-470c-b497-90c41305ea64\") " Jul 21 00:50:26.686894 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:26.686770 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f5ace73b-999c-4930-8b0a-d58df058503d-tls-cert\") pod \"f5ace73b-999c-4930-8b0a-d58df058503d\" (UID: \"f5ace73b-999c-4930-8b0a-d58df058503d\") " Jul 21 00:50:26.686938 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:26.686905 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fjj6t\" (UniqueName: \"kubernetes.io/projected/f5ace73b-999c-4930-8b0a-d58df058503d-kube-api-access-fjj6t\") pod \"f5ace73b-999c-4930-8b0a-d58df058503d\" (UID: \"f5ace73b-999c-4930-8b0a-d58df058503d\") " Jul 21 00:50:26.687159 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:26.687132 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7st7\" (UniqueName: \"kubernetes.io/projected/a23ae0de-9373-4024-aa25-4aeb5405243d-kube-api-access-p7st7\") pod \"authorino-769465ddbd-v46sv\" (UID: \"a23ae0de-9373-4024-aa25-4aeb5405243d\") " pod="kuadrant-system/authorino-769465ddbd-v46sv" Jul 21 00:50:26.687263 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:26.687179 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a23ae0de-9373-4024-aa25-4aeb5405243d-tls-cert\") pod \"authorino-769465ddbd-v46sv\" (UID: \"a23ae0de-9373-4024-aa25-4aeb5405243d\") " pod="kuadrant-system/authorino-769465ddbd-v46sv" Jul 21 00:50:26.688916 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:26.688882 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16b04187-4174-470c-b497-90c41305ea64-kube-api-access-5lcbt" (OuterVolumeSpecName: "kube-api-access-5lcbt") pod "16b04187-4174-470c-b497-90c41305ea64" (UID: "16b04187-4174-470c-b497-90c41305ea64"). InnerVolumeSpecName "kube-api-access-5lcbt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 00:50:26.689026 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:26.688914 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5ace73b-999c-4930-8b0a-d58df058503d-kube-api-access-fjj6t" (OuterVolumeSpecName: "kube-api-access-fjj6t") pod "f5ace73b-999c-4930-8b0a-d58df058503d" (UID: "f5ace73b-999c-4930-8b0a-d58df058503d"). InnerVolumeSpecName "kube-api-access-fjj6t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 00:50:26.689026 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:26.688946 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5ace73b-999c-4930-8b0a-d58df058503d-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "f5ace73b-999c-4930-8b0a-d58df058503d" (UID: "f5ace73b-999c-4930-8b0a-d58df058503d"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 00:50:26.689447 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:26.689431 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a23ae0de-9373-4024-aa25-4aeb5405243d-tls-cert\") pod \"authorino-769465ddbd-v46sv\" (UID: \"a23ae0de-9373-4024-aa25-4aeb5405243d\") " pod="kuadrant-system/authorino-769465ddbd-v46sv" Jul 21 00:50:26.694598 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:26.694578 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7st7\" (UniqueName: \"kubernetes.io/projected/a23ae0de-9373-4024-aa25-4aeb5405243d-kube-api-access-p7st7\") pod \"authorino-769465ddbd-v46sv\" (UID: \"a23ae0de-9373-4024-aa25-4aeb5405243d\") " pod="kuadrant-system/authorino-769465ddbd-v46sv" Jul 21 00:50:26.788111 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:26.788049 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lcbt\" (UniqueName: \"kubernetes.io/projected/16b04187-4174-470c-b497-90c41305ea64-kube-api-access-5lcbt\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 21 00:50:26.788111 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:26.788073 2580 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f5ace73b-999c-4930-8b0a-d58df058503d-tls-cert\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 21 00:50:26.788111 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:26.788083 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjj6t\" (UniqueName: \"kubernetes.io/projected/f5ace73b-999c-4930-8b0a-d58df058503d-kube-api-access-fjj6t\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 21 00:50:26.824373 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:26.824351 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-769465ddbd-v46sv" Jul 21 00:50:26.949326 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:26.949297 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-769465ddbd-v46sv"] Jul 21 00:50:26.951270 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:50:26.951245 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda23ae0de_9373_4024_aa25_4aeb5405243d.slice/crio-0558d4d39651518f60dff18e3dae371aea108d9269916f362faf5cab8e95fea6 WatchSource:0}: Error finding container 0558d4d39651518f60dff18e3dae371aea108d9269916f362faf5cab8e95fea6: Status 404 returned error can't find the container with id 0558d4d39651518f60dff18e3dae371aea108d9269916f362faf5cab8e95fea6 Jul 21 00:50:27.563468 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:27.563430 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-769465ddbd-v46sv" event={"ID":"a23ae0de-9373-4024-aa25-4aeb5405243d","Type":"ContainerStarted","Data":"0097f0bad971886e664e6a49402074f01c6e3d662d6d3b4a69aa571b632af484"} Jul 21 00:50:27.563468 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:27.563454 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-7dwxq" Jul 21 00:50:27.563895 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:27.563481 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-769465ddbd-v46sv" event={"ID":"a23ae0de-9373-4024-aa25-4aeb5405243d","Type":"ContainerStarted","Data":"0558d4d39651518f60dff18e3dae371aea108d9269916f362faf5cab8e95fea6"} Jul 21 00:50:27.563895 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:27.563449 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-64hqh" Jul 21 00:50:27.592300 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:27.592219 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-769465ddbd-v46sv" podStartSLOduration=1.170082141 podStartE2EDuration="1.592197298s" totalImagesPullingTime="422.115157ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:50:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:50:27.577743696 +0000 UTC m=+575.183852737" watchObservedRunningTime="2026-07-21 00:50:27.592197298 +0000 UTC m=+575.198306339" Jul 21 00:50:27.601407 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:27.601379 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-64hqh"] Jul 21 00:50:27.605969 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:27.605946 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-k9xc2"] Jul 21 00:50:27.606186 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:27.606158 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-k9xc2" podUID="3965ce93-f413-456d-9b8d-5ce4a43a2b7b" containerName="authorino" containerID="cri-o://e78f96cbe340acd7adc5eb7adda1ad927b061b95a4f7521be1bd1018948c9386" gracePeriod=30 Jul 21 00:50:27.609349 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:27.609326 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-76877994-64hqh"] Jul 21 00:50:27.633374 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:27.633312 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-7dwxq"] Jul 21 00:50:27.636804 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:27.636782 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-7dwxq"] Jul 21 00:50:27.857429 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:27.857401 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-k9xc2" Jul 21 00:50:27.997908 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:27.997834 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zbbrf\" (UniqueName: \"kubernetes.io/projected/3965ce93-f413-456d-9b8d-5ce4a43a2b7b-kube-api-access-zbbrf\") pod \"3965ce93-f413-456d-9b8d-5ce4a43a2b7b\" (UID: \"3965ce93-f413-456d-9b8d-5ce4a43a2b7b\") " Jul 21 00:50:27.999883 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:27.999855 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3965ce93-f413-456d-9b8d-5ce4a43a2b7b-kube-api-access-zbbrf" (OuterVolumeSpecName: "kube-api-access-zbbrf") pod "3965ce93-f413-456d-9b8d-5ce4a43a2b7b" (UID: "3965ce93-f413-456d-9b8d-5ce4a43a2b7b"). InnerVolumeSpecName "kube-api-access-zbbrf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 00:50:28.098970 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:28.098939 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbbrf\" (UniqueName: \"kubernetes.io/projected/3965ce93-f413-456d-9b8d-5ce4a43a2b7b-kube-api-access-zbbrf\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 21 00:50:28.569697 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:28.569660 2580 generic.go:356] "Generic (PLEG): container finished" podID="3965ce93-f413-456d-9b8d-5ce4a43a2b7b" containerID="e78f96cbe340acd7adc5eb7adda1ad927b061b95a4f7521be1bd1018948c9386" exitCode=0 Jul 21 00:50:28.570133 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:28.569702 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-k9xc2" Jul 21 00:50:28.570133 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:28.569741 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-k9xc2" event={"ID":"3965ce93-f413-456d-9b8d-5ce4a43a2b7b","Type":"ContainerDied","Data":"e78f96cbe340acd7adc5eb7adda1ad927b061b95a4f7521be1bd1018948c9386"} Jul 21 00:50:28.570133 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:28.569794 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-k9xc2" event={"ID":"3965ce93-f413-456d-9b8d-5ce4a43a2b7b","Type":"ContainerDied","Data":"610a30c7265b2b66911ceba07ef030882cbf7a0d023cab4a3f669629a85145e8"} Jul 21 00:50:28.570133 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:28.569818 2580 scope.go:117] "RemoveContainer" containerID="e78f96cbe340acd7adc5eb7adda1ad927b061b95a4f7521be1bd1018948c9386" Jul 21 00:50:28.578385 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:28.578365 2580 scope.go:117] "RemoveContainer" containerID="e78f96cbe340acd7adc5eb7adda1ad927b061b95a4f7521be1bd1018948c9386" Jul 21 00:50:28.578604 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:50:28.578587 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e78f96cbe340acd7adc5eb7adda1ad927b061b95a4f7521be1bd1018948c9386\": container with ID starting with e78f96cbe340acd7adc5eb7adda1ad927b061b95a4f7521be1bd1018948c9386 not found: ID does not exist" containerID="e78f96cbe340acd7adc5eb7adda1ad927b061b95a4f7521be1bd1018948c9386" Jul 21 00:50:28.578662 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:28.578610 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e78f96cbe340acd7adc5eb7adda1ad927b061b95a4f7521be1bd1018948c9386"} err="failed to get container status \"e78f96cbe340acd7adc5eb7adda1ad927b061b95a4f7521be1bd1018948c9386\": rpc error: code = NotFound desc = could not find container \"e78f96cbe340acd7adc5eb7adda1ad927b061b95a4f7521be1bd1018948c9386\": container with ID starting with e78f96cbe340acd7adc5eb7adda1ad927b061b95a4f7521be1bd1018948c9386 not found: ID does not exist" Jul 21 00:50:28.590110 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:28.590080 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-k9xc2"] Jul 21 00:50:28.591813 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:28.591794 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-k9xc2"] Jul 21 00:50:28.912522 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:28.912481 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16b04187-4174-470c-b497-90c41305ea64" path="/var/lib/kubelet/pods/16b04187-4174-470c-b497-90c41305ea64/volumes" Jul 21 00:50:28.912791 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:28.912771 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3965ce93-f413-456d-9b8d-5ce4a43a2b7b" path="/var/lib/kubelet/pods/3965ce93-f413-456d-9b8d-5ce4a43a2b7b/volumes" Jul 21 00:50:28.913144 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:28.913127 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5ace73b-999c-4930-8b0a-d58df058503d" path="/var/lib/kubelet/pods/f5ace73b-999c-4930-8b0a-d58df058503d/volumes" Jul 21 00:50:32.342538 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:32.342457 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-657fd578b7-ngnn8"] Jul 21 00:50:32.342883 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:32.342871 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3965ce93-f413-456d-9b8d-5ce4a43a2b7b" containerName="authorino" Jul 21 00:50:32.342921 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:32.342884 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3965ce93-f413-456d-9b8d-5ce4a43a2b7b" containerName="authorino" Jul 21 00:50:32.342977 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:32.342967 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3965ce93-f413-456d-9b8d-5ce4a43a2b7b" containerName="authorino" Jul 21 00:50:32.601272 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:32.601187 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-657fd578b7-ngnn8"] Jul 21 00:50:32.601272 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:32.601223 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7765874568-gqgjs"] Jul 21 00:50:32.601449 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:32.601325 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-657fd578b7-ngnn8" Jul 21 00:50:32.605141 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:32.605108 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-4hxcg\"" Jul 21 00:50:32.605297 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:32.605123 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 21 00:50:32.605297 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:32.605211 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 21 00:50:32.624362 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:32.624339 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7765874568-gqgjs"] Jul 21 00:50:32.624455 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:32.624427 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7765874568-gqgjs" Jul 21 00:50:32.740642 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:32.740608 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/32111504-497b-4f55-ba47-a941a37019b4-webhook-cert\") pod \"maas-controller-7765874568-gqgjs\" (UID: \"32111504-497b-4f55-ba47-a941a37019b4\") " pod="opendatahub/maas-controller-7765874568-gqgjs" Jul 21 00:50:32.740831 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:32.740656 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkmqx\" (UniqueName: \"kubernetes.io/projected/7ce060ed-baaf-4454-9a8b-fcbba6e3cda6-kube-api-access-gkmqx\") pod \"maas-controller-657fd578b7-ngnn8\" (UID: \"7ce060ed-baaf-4454-9a8b-fcbba6e3cda6\") " pod="opendatahub/maas-controller-657fd578b7-ngnn8" Jul 21 00:50:32.740831 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:32.740692 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7ce060ed-baaf-4454-9a8b-fcbba6e3cda6-webhook-cert\") pod \"maas-controller-657fd578b7-ngnn8\" (UID: \"7ce060ed-baaf-4454-9a8b-fcbba6e3cda6\") " pod="opendatahub/maas-controller-657fd578b7-ngnn8" Jul 21 00:50:32.740831 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:32.740736 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m842c\" (UniqueName: \"kubernetes.io/projected/32111504-497b-4f55-ba47-a941a37019b4-kube-api-access-m842c\") pod \"maas-controller-7765874568-gqgjs\" (UID: \"32111504-497b-4f55-ba47-a941a37019b4\") " pod="opendatahub/maas-controller-7765874568-gqgjs" Jul 21 00:50:32.841956 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:32.841922 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkmqx\" (UniqueName: \"kubernetes.io/projected/7ce060ed-baaf-4454-9a8b-fcbba6e3cda6-kube-api-access-gkmqx\") pod \"maas-controller-657fd578b7-ngnn8\" (UID: \"7ce060ed-baaf-4454-9a8b-fcbba6e3cda6\") " pod="opendatahub/maas-controller-657fd578b7-ngnn8" Jul 21 00:50:32.842155 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:32.841982 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7ce060ed-baaf-4454-9a8b-fcbba6e3cda6-webhook-cert\") pod \"maas-controller-657fd578b7-ngnn8\" (UID: \"7ce060ed-baaf-4454-9a8b-fcbba6e3cda6\") " pod="opendatahub/maas-controller-657fd578b7-ngnn8" Jul 21 00:50:32.842155 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:32.842010 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m842c\" (UniqueName: \"kubernetes.io/projected/32111504-497b-4f55-ba47-a941a37019b4-kube-api-access-m842c\") pod \"maas-controller-7765874568-gqgjs\" (UID: \"32111504-497b-4f55-ba47-a941a37019b4\") " pod="opendatahub/maas-controller-7765874568-gqgjs" Jul 21 00:50:32.842155 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:32.842112 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/32111504-497b-4f55-ba47-a941a37019b4-webhook-cert\") pod \"maas-controller-7765874568-gqgjs\" (UID: \"32111504-497b-4f55-ba47-a941a37019b4\") " pod="opendatahub/maas-controller-7765874568-gqgjs" Jul 21 00:50:32.844533 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:32.844507 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/32111504-497b-4f55-ba47-a941a37019b4-webhook-cert\") pod \"maas-controller-7765874568-gqgjs\" (UID: \"32111504-497b-4f55-ba47-a941a37019b4\") " pod="opendatahub/maas-controller-7765874568-gqgjs" Jul 21 00:50:32.844635 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:32.844509 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7ce060ed-baaf-4454-9a8b-fcbba6e3cda6-webhook-cert\") pod \"maas-controller-657fd578b7-ngnn8\" (UID: \"7ce060ed-baaf-4454-9a8b-fcbba6e3cda6\") " pod="opendatahub/maas-controller-657fd578b7-ngnn8" Jul 21 00:50:32.849995 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:32.849975 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m842c\" (UniqueName: \"kubernetes.io/projected/32111504-497b-4f55-ba47-a941a37019b4-kube-api-access-m842c\") pod \"maas-controller-7765874568-gqgjs\" (UID: \"32111504-497b-4f55-ba47-a941a37019b4\") " pod="opendatahub/maas-controller-7765874568-gqgjs" Jul 21 00:50:32.850361 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:32.850339 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkmqx\" (UniqueName: \"kubernetes.io/projected/7ce060ed-baaf-4454-9a8b-fcbba6e3cda6-kube-api-access-gkmqx\") pod \"maas-controller-657fd578b7-ngnn8\" (UID: \"7ce060ed-baaf-4454-9a8b-fcbba6e3cda6\") " pod="opendatahub/maas-controller-657fd578b7-ngnn8" Jul 21 00:50:32.913312 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:32.913255 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-657fd578b7-ngnn8" Jul 21 00:50:32.933360 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:32.933339 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7765874568-gqgjs" Jul 21 00:50:33.070982 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:33.070953 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-657fd578b7-ngnn8"] Jul 21 00:50:33.073493 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:50:33.073456 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ce060ed_baaf_4454_9a8b_fcbba6e3cda6.slice/crio-46667a24194171a75808afadc67a405b99d9873f97186b37f572e18aa69a6854 WatchSource:0}: Error finding container 46667a24194171a75808afadc67a405b99d9873f97186b37f572e18aa69a6854: Status 404 returned error can't find the container with id 46667a24194171a75808afadc67a405b99d9873f97186b37f572e18aa69a6854 Jul 21 00:50:33.099266 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:33.099219 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7765874568-gqgjs"] Jul 21 00:50:33.100876 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:50:33.100851 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32111504_497b_4f55_ba47_a941a37019b4.slice/crio-74ecc26551428e940edd5de3287d95bd8ec23542913494ab822975d49973b10e WatchSource:0}: Error finding container 74ecc26551428e940edd5de3287d95bd8ec23542913494ab822975d49973b10e: Status 404 returned error can't find the container with id 74ecc26551428e940edd5de3287d95bd8ec23542913494ab822975d49973b10e Jul 21 00:50:33.595487 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:33.595451 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-657fd578b7-ngnn8" event={"ID":"7ce060ed-baaf-4454-9a8b-fcbba6e3cda6","Type":"ContainerStarted","Data":"46667a24194171a75808afadc67a405b99d9873f97186b37f572e18aa69a6854"} Jul 21 00:50:33.596627 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:33.596597 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7765874568-gqgjs" event={"ID":"32111504-497b-4f55-ba47-a941a37019b4","Type":"ContainerStarted","Data":"74ecc26551428e940edd5de3287d95bd8ec23542913494ab822975d49973b10e"} Jul 21 00:50:35.610380 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:35.610345 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7765874568-gqgjs" event={"ID":"32111504-497b-4f55-ba47-a941a37019b4","Type":"ContainerStarted","Data":"b8f90a7691558e9bd2ae67b482cb498ef9de9f4aa82bc0f3a9dc6a03bcf6f41d"} Jul 21 00:50:35.610828 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:35.610469 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7765874568-gqgjs" Jul 21 00:50:35.612038 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:35.612009 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-657fd578b7-ngnn8" event={"ID":"7ce060ed-baaf-4454-9a8b-fcbba6e3cda6","Type":"ContainerStarted","Data":"9287dda6751cfb76dd11f88e8dc17ed600a99cd664137ff4d240c2b3ae55952d"} Jul 21 00:50:35.612158 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:35.612140 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-657fd578b7-ngnn8" Jul 21 00:50:35.640661 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:35.640607 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7765874568-gqgjs" podStartSLOduration=1.273405426 podStartE2EDuration="3.640593921s" totalImagesPullingTime="2.367188495s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:50:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:50:35.638678125 +0000 UTC m=+583.244787166" watchObservedRunningTime="2026-07-21 00:50:35.640593921 +0000 UTC m=+583.246702986" Jul 21 00:50:35.660996 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:35.660867 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-657fd578b7-ngnn8" podStartSLOduration=1.269664189 podStartE2EDuration="3.660853683s" totalImagesPullingTime="2.391189494s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:50:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:50:35.660668986 +0000 UTC m=+583.266778023" watchObservedRunningTime="2026-07-21 00:50:35.660853683 +0000 UTC m=+583.266962722" Jul 21 00:50:46.619227 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:46.619199 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7765874568-gqgjs" Jul 21 00:50:46.619678 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:46.619402 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-657fd578b7-ngnn8" Jul 21 00:50:46.677726 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:46.677696 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-657fd578b7-ngnn8"] Jul 21 00:50:46.677927 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:46.677901 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-657fd578b7-ngnn8" podUID="7ce060ed-baaf-4454-9a8b-fcbba6e3cda6" containerName="manager" containerID="cri-o://9287dda6751cfb76dd11f88e8dc17ed600a99cd664137ff4d240c2b3ae55952d" gracePeriod=30 Jul 21 00:50:46.928913 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:46.928887 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-657fd578b7-ngnn8" Jul 21 00:50:46.958824 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:46.958793 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gkmqx\" (UniqueName: \"kubernetes.io/projected/7ce060ed-baaf-4454-9a8b-fcbba6e3cda6-kube-api-access-gkmqx\") pod \"7ce060ed-baaf-4454-9a8b-fcbba6e3cda6\" (UID: \"7ce060ed-baaf-4454-9a8b-fcbba6e3cda6\") " Jul 21 00:50:46.958965 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:46.958891 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7ce060ed-baaf-4454-9a8b-fcbba6e3cda6-webhook-cert\") pod \"7ce060ed-baaf-4454-9a8b-fcbba6e3cda6\" (UID: \"7ce060ed-baaf-4454-9a8b-fcbba6e3cda6\") " Jul 21 00:50:46.961018 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:46.960985 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ce060ed-baaf-4454-9a8b-fcbba6e3cda6-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "7ce060ed-baaf-4454-9a8b-fcbba6e3cda6" (UID: "7ce060ed-baaf-4454-9a8b-fcbba6e3cda6"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 00:50:46.961108 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:46.961079 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ce060ed-baaf-4454-9a8b-fcbba6e3cda6-kube-api-access-gkmqx" (OuterVolumeSpecName: "kube-api-access-gkmqx") pod "7ce060ed-baaf-4454-9a8b-fcbba6e3cda6" (UID: "7ce060ed-baaf-4454-9a8b-fcbba6e3cda6"). InnerVolumeSpecName "kube-api-access-gkmqx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 00:50:47.059628 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:47.059596 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkmqx\" (UniqueName: \"kubernetes.io/projected/7ce060ed-baaf-4454-9a8b-fcbba6e3cda6-kube-api-access-gkmqx\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 21 00:50:47.059628 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:47.059624 2580 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7ce060ed-baaf-4454-9a8b-fcbba6e3cda6-webhook-cert\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 21 00:50:47.670107 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:47.670067 2580 generic.go:356] "Generic (PLEG): container finished" podID="7ce060ed-baaf-4454-9a8b-fcbba6e3cda6" containerID="9287dda6751cfb76dd11f88e8dc17ed600a99cd664137ff4d240c2b3ae55952d" exitCode=0 Jul 21 00:50:47.670494 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:47.670127 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-657fd578b7-ngnn8" Jul 21 00:50:47.670494 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:47.670119 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-657fd578b7-ngnn8" event={"ID":"7ce060ed-baaf-4454-9a8b-fcbba6e3cda6","Type":"ContainerDied","Data":"9287dda6751cfb76dd11f88e8dc17ed600a99cd664137ff4d240c2b3ae55952d"} Jul 21 00:50:47.670494 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:47.670262 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-657fd578b7-ngnn8" event={"ID":"7ce060ed-baaf-4454-9a8b-fcbba6e3cda6","Type":"ContainerDied","Data":"46667a24194171a75808afadc67a405b99d9873f97186b37f572e18aa69a6854"} Jul 21 00:50:47.670494 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:47.670286 2580 scope.go:117] "RemoveContainer" containerID="9287dda6751cfb76dd11f88e8dc17ed600a99cd664137ff4d240c2b3ae55952d" Jul 21 00:50:47.679394 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:47.679182 2580 scope.go:117] "RemoveContainer" containerID="9287dda6751cfb76dd11f88e8dc17ed600a99cd664137ff4d240c2b3ae55952d" Jul 21 00:50:47.679471 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:50:47.679450 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9287dda6751cfb76dd11f88e8dc17ed600a99cd664137ff4d240c2b3ae55952d\": container with ID starting with 9287dda6751cfb76dd11f88e8dc17ed600a99cd664137ff4d240c2b3ae55952d not found: ID does not exist" containerID="9287dda6751cfb76dd11f88e8dc17ed600a99cd664137ff4d240c2b3ae55952d" Jul 21 00:50:47.679509 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:47.679479 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9287dda6751cfb76dd11f88e8dc17ed600a99cd664137ff4d240c2b3ae55952d"} err="failed to get container status \"9287dda6751cfb76dd11f88e8dc17ed600a99cd664137ff4d240c2b3ae55952d\": rpc error: code = NotFound desc = could not find container \"9287dda6751cfb76dd11f88e8dc17ed600a99cd664137ff4d240c2b3ae55952d\": container with ID starting with 9287dda6751cfb76dd11f88e8dc17ed600a99cd664137ff4d240c2b3ae55952d not found: ID does not exist" Jul 21 00:50:47.692200 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:47.692180 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-657fd578b7-ngnn8"] Jul 21 00:50:47.695681 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:47.695661 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-657fd578b7-ngnn8"] Jul 21 00:50:48.912716 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:48.912682 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ce060ed-baaf-4454-9a8b-fcbba6e3cda6" path="/var/lib/kubelet/pods/7ce060ed-baaf-4454-9a8b-fcbba6e3cda6/volumes" Jul 21 00:50:52.768997 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:52.768971 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-rb4ph_3ee1dbf0-b4ff-4871-9432-284eb25f97b1/console-operator/1.log" Jul 21 00:50:52.771562 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:50:52.771544 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-rb4ph_3ee1dbf0-b4ff-4871-9432-284eb25f97b1/console-operator/1.log" Jul 21 00:51:08.891928 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:51:08.891891 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6bc4785f87-2p6gx"] Jul 21 00:51:08.892442 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:51:08.892421 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ce060ed-baaf-4454-9a8b-fcbba6e3cda6" containerName="manager" Jul 21 00:51:08.892442 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:51:08.892439 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ce060ed-baaf-4454-9a8b-fcbba6e3cda6" containerName="manager" Jul 21 00:51:08.892519 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:51:08.892508 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ce060ed-baaf-4454-9a8b-fcbba6e3cda6" containerName="manager" Jul 21 00:51:08.897001 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:51:08.896986 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6bc4785f87-2p6gx" Jul 21 00:51:08.899728 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:51:08.899709 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 21 00:51:08.899728 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:51:08.899721 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8b65c\"" Jul 21 00:51:08.904021 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:51:08.903991 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6bc4785f87-2p6gx"] Jul 21 00:51:09.050585 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:51:09.050550 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/761360e3-de73-49dd-9bf3-292a44066f1a-openshift-service-ca\") pod \"maas-api-6bc4785f87-2p6gx\" (UID: \"761360e3-de73-49dd-9bf3-292a44066f1a\") " pod="odh-ai-gateway-infra/maas-api-6bc4785f87-2p6gx" Jul 21 00:51:09.050761 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:51:09.050674 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzzf6\" (UniqueName: \"kubernetes.io/projected/761360e3-de73-49dd-9bf3-292a44066f1a-kube-api-access-wzzf6\") pod \"maas-api-6bc4785f87-2p6gx\" (UID: \"761360e3-de73-49dd-9bf3-292a44066f1a\") " pod="odh-ai-gateway-infra/maas-api-6bc4785f87-2p6gx" Jul 21 00:51:09.050761 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:51:09.050704 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/761360e3-de73-49dd-9bf3-292a44066f1a-maas-api-tls\") pod \"maas-api-6bc4785f87-2p6gx\" (UID: \"761360e3-de73-49dd-9bf3-292a44066f1a\") " pod="odh-ai-gateway-infra/maas-api-6bc4785f87-2p6gx" Jul 21 00:51:09.152111 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:51:09.152024 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzzf6\" (UniqueName: \"kubernetes.io/projected/761360e3-de73-49dd-9bf3-292a44066f1a-kube-api-access-wzzf6\") pod \"maas-api-6bc4785f87-2p6gx\" (UID: \"761360e3-de73-49dd-9bf3-292a44066f1a\") " pod="odh-ai-gateway-infra/maas-api-6bc4785f87-2p6gx" Jul 21 00:51:09.152111 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:51:09.152081 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/761360e3-de73-49dd-9bf3-292a44066f1a-maas-api-tls\") pod \"maas-api-6bc4785f87-2p6gx\" (UID: \"761360e3-de73-49dd-9bf3-292a44066f1a\") " pod="odh-ai-gateway-infra/maas-api-6bc4785f87-2p6gx" Jul 21 00:51:09.152359 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:51:09.152193 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/761360e3-de73-49dd-9bf3-292a44066f1a-openshift-service-ca\") pod \"maas-api-6bc4785f87-2p6gx\" (UID: \"761360e3-de73-49dd-9bf3-292a44066f1a\") " pod="odh-ai-gateway-infra/maas-api-6bc4785f87-2p6gx" Jul 21 00:51:09.152359 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:51:09.152255 2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found Jul 21 00:51:09.152359 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:51:09.152339 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/761360e3-de73-49dd-9bf3-292a44066f1a-maas-api-tls podName:761360e3-de73-49dd-9bf3-292a44066f1a nodeName:}" failed. No retries permitted until 2026-07-21 00:51:09.652318274 +0000 UTC m=+617.258427306 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/761360e3-de73-49dd-9bf3-292a44066f1a-maas-api-tls") pod "maas-api-6bc4785f87-2p6gx" (UID: "761360e3-de73-49dd-9bf3-292a44066f1a") : secret "maas-api-serving-cert" not found Jul 21 00:51:09.152851 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:51:09.152830 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/761360e3-de73-49dd-9bf3-292a44066f1a-openshift-service-ca\") pod \"maas-api-6bc4785f87-2p6gx\" (UID: \"761360e3-de73-49dd-9bf3-292a44066f1a\") " pod="odh-ai-gateway-infra/maas-api-6bc4785f87-2p6gx" Jul 21 00:51:09.160842 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:51:09.160817 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzzf6\" (UniqueName: \"kubernetes.io/projected/761360e3-de73-49dd-9bf3-292a44066f1a-kube-api-access-wzzf6\") pod \"maas-api-6bc4785f87-2p6gx\" (UID: \"761360e3-de73-49dd-9bf3-292a44066f1a\") " pod="odh-ai-gateway-infra/maas-api-6bc4785f87-2p6gx" Jul 21 00:51:09.368461 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:51:09.368428 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-v45mh"] Jul 21 00:51:09.375573 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:51:09.375548 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-v45mh" Jul 21 00:51:09.378611 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:51:09.378583 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-rlmt5\"" Jul 21 00:51:09.378950 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:51:09.378923 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 21 00:51:09.380124 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:51:09.380096 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-v45mh"] Jul 21 00:51:09.455992 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:51:09.455917 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae4bd066-5fdf-4c2c-883c-bee5800b0069-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-v45mh\" (UID: \"ae4bd066-5fdf-4c2c-883c-bee5800b0069\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-v45mh" Jul 21 00:51:09.456132 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:51:09.456019 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmpdr\" (UniqueName: \"kubernetes.io/projected/ae4bd066-5fdf-4c2c-883c-bee5800b0069-kube-api-access-fmpdr\") pod \"payload-pre-processing-7677b774bb-v45mh\" (UID: \"ae4bd066-5fdf-4c2c-883c-bee5800b0069\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-v45mh" Jul 21 00:51:09.557415 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:51:09.557378 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae4bd066-5fdf-4c2c-883c-bee5800b0069-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-v45mh\" (UID: \"ae4bd066-5fdf-4c2c-883c-bee5800b0069\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-v45mh" Jul 21 00:51:09.557584 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:51:09.557448 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmpdr\" (UniqueName: \"kubernetes.io/projected/ae4bd066-5fdf-4c2c-883c-bee5800b0069-kube-api-access-fmpdr\") pod \"payload-pre-processing-7677b774bb-v45mh\" (UID: \"ae4bd066-5fdf-4c2c-883c-bee5800b0069\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-v45mh" Jul 21 00:51:09.558194 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:51:09.558171 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae4bd066-5fdf-4c2c-883c-bee5800b0069-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-v45mh\" (UID: \"ae4bd066-5fdf-4c2c-883c-bee5800b0069\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-v45mh" Jul 21 00:51:09.569666 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:51:09.569641 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmpdr\" (UniqueName: \"kubernetes.io/projected/ae4bd066-5fdf-4c2c-883c-bee5800b0069-kube-api-access-fmpdr\") pod \"payload-pre-processing-7677b774bb-v45mh\" (UID: \"ae4bd066-5fdf-4c2c-883c-bee5800b0069\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-v45mh" Jul 21 00:51:09.658257 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:51:09.658213 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/761360e3-de73-49dd-9bf3-292a44066f1a-maas-api-tls\") pod \"maas-api-6bc4785f87-2p6gx\" (UID: \"761360e3-de73-49dd-9bf3-292a44066f1a\") " pod="odh-ai-gateway-infra/maas-api-6bc4785f87-2p6gx" Jul 21 00:51:09.660507 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:51:09.660487 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/761360e3-de73-49dd-9bf3-292a44066f1a-maas-api-tls\") pod \"maas-api-6bc4785f87-2p6gx\" (UID: \"761360e3-de73-49dd-9bf3-292a44066f1a\") " pod="odh-ai-gateway-infra/maas-api-6bc4785f87-2p6gx" Jul 21 00:51:09.688208 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:51:09.688183 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-v45mh" Jul 21 00:51:09.809879 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:51:09.809848 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6bc4785f87-2p6gx" Jul 21 00:51:09.823157 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:51:09.823135 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-v45mh"] Jul 21 00:51:09.825192 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:51:09.825163 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae4bd066_5fdf_4c2c_883c_bee5800b0069.slice/crio-32fcefffb9a75c92e3f7fcf5caf21dedb0b6b99ef6c6c29f0357a71c550e0d5c WatchSource:0}: Error finding container 32fcefffb9a75c92e3f7fcf5caf21dedb0b6b99ef6c6c29f0357a71c550e0d5c: Status 404 returned error can't find the container with id 32fcefffb9a75c92e3f7fcf5caf21dedb0b6b99ef6c6c29f0357a71c550e0d5c Jul 21 00:51:09.949582 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:51:09.949555 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6bc4785f87-2p6gx"] Jul 21 00:51:09.951535 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:51:09.951508 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod761360e3_de73_49dd_9bf3_292a44066f1a.slice/crio-bdb20eab10b127c336ce0e5baa4531723dbf528085acac3a727504dc8cf462c7 WatchSource:0}: Error finding container bdb20eab10b127c336ce0e5baa4531723dbf528085acac3a727504dc8cf462c7: Status 404 returned error can't find the container with id bdb20eab10b127c336ce0e5baa4531723dbf528085acac3a727504dc8cf462c7 Jul 21 00:51:10.796530 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:51:10.796447 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-v45mh" event={"ID":"ae4bd066-5fdf-4c2c-883c-bee5800b0069","Type":"ContainerStarted","Data":"32fcefffb9a75c92e3f7fcf5caf21dedb0b6b99ef6c6c29f0357a71c550e0d5c"} Jul 21 00:51:10.799278 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:51:10.799224 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6bc4785f87-2p6gx" event={"ID":"761360e3-de73-49dd-9bf3-292a44066f1a","Type":"ContainerStarted","Data":"bdb20eab10b127c336ce0e5baa4531723dbf528085acac3a727504dc8cf462c7"} Jul 21 00:51:12.811278 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:51:12.811172 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6bc4785f87-2p6gx" event={"ID":"761360e3-de73-49dd-9bf3-292a44066f1a","Type":"ContainerStarted","Data":"8beede7e8dd8a33199ff509efee9e65c05cd11bf35f78f45de782b96cc51966b"} Jul 21 00:51:12.811278 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:51:12.811228 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6bc4785f87-2p6gx" Jul 21 00:51:12.812681 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:51:12.812658 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-v45mh" event={"ID":"ae4bd066-5fdf-4c2c-883c-bee5800b0069","Type":"ContainerStarted","Data":"341af9247d7f5f174798d0cf10e4c28609c16b1ad4e50640958c6da18b965ae3"} Jul 21 00:51:12.812808 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:51:12.812704 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-v45mh" Jul 21 00:51:12.829078 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:51:12.829042 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6bc4785f87-2p6gx" podStartSLOduration=2.411282391 podStartE2EDuration="4.829029918s" totalImagesPullingTime="2.417747527s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:51:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:51:12.827910178 +0000 UTC m=+620.434019217" watchObservedRunningTime="2026-07-21 00:51:12.829029918 +0000 UTC m=+620.435138957" Jul 21 00:51:12.842364 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:51:12.842320 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7677b774bb-v45mh" podStartSLOduration=1.252543744 podStartE2EDuration="3.842305885s" totalImagesPullingTime="2.589762141s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:51:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:51:12.841670232 +0000 UTC m=+620.447779274" watchObservedRunningTime="2026-07-21 00:51:12.842305885 +0000 UTC m=+620.448414968" Jul 21 00:51:18.823525 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:51:18.823445 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6bc4785f87-2p6gx" Jul 21 00:51:23.819939 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:51:23.819910 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-v45mh" Jul 21 00:51:30.023749 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:51:30.023716 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7nhqf"] Jul 21 00:51:30.027626 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:51:30.027603 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7nhqf" Jul 21 00:51:30.031420 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:51:30.031386 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-xtkz5\"" Jul 21 00:51:30.031557 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:51:30.031445 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 21 00:51:30.031725 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:51:30.031705 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 21 00:51:30.031831 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:51:30.031807 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 21 00:51:30.034006 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:51:30.033980 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7nhqf"] Jul 21 00:51:30.138629 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:51:30.138596 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/959d2ecf-daef-4d66-b1a3-153e0f52d094-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7nhqf\" (UID: \"959d2ecf-daef-4d66-b1a3-153e0f52d094\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7nhqf" Jul 21 00:51:30.138629 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:51:30.138629 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/959d2ecf-daef-4d66-b1a3-153e0f52d094-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7nhqf\" (UID: \"959d2ecf-daef-4d66-b1a3-153e0f52d094\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7nhqf" Jul 21 00:51:30.138843 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:51:30.138648 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/959d2ecf-daef-4d66-b1a3-153e0f52d094-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7nhqf\" (UID: \"959d2ecf-daef-4d66-b1a3-153e0f52d094\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7nhqf" Jul 21 00:51:30.138843 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:51:30.138737 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/959d2ecf-daef-4d66-b1a3-153e0f52d094-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7nhqf\" (UID: \"959d2ecf-daef-4d66-b1a3-153e0f52d094\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7nhqf" Jul 21 00:51:30.138843 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:51:30.138805 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/959d2ecf-daef-4d66-b1a3-153e0f52d094-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7nhqf\" (UID: \"959d2ecf-daef-4d66-b1a3-153e0f52d094\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7nhqf" Jul 21 00:51:30.138983 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:51:30.138865 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmdf5\" (UniqueName: \"kubernetes.io/projected/959d2ecf-daef-4d66-b1a3-153e0f52d094-kube-api-access-qmdf5\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7nhqf\" (UID: \"959d2ecf-daef-4d66-b1a3-153e0f52d094\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7nhqf" Jul 21 00:51:30.240099 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:51:30.240063 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/959d2ecf-daef-4d66-b1a3-153e0f52d094-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7nhqf\" (UID: \"959d2ecf-daef-4d66-b1a3-153e0f52d094\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7nhqf" Jul 21 00:51:30.240099 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:51:30.240099 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/959d2ecf-daef-4d66-b1a3-153e0f52d094-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7nhqf\" (UID: \"959d2ecf-daef-4d66-b1a3-153e0f52d094\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7nhqf" Jul 21 00:51:30.240343 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:51:30.240115 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/959d2ecf-daef-4d66-b1a3-153e0f52d094-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7nhqf\" (UID: \"959d2ecf-daef-4d66-b1a3-153e0f52d094\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7nhqf" Jul 21 00:51:30.240343 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:51:30.240147 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/959d2ecf-daef-4d66-b1a3-153e0f52d094-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7nhqf\" (UID: \"959d2ecf-daef-4d66-b1a3-153e0f52d094\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7nhqf" Jul 21 00:51:30.240343 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:51:30.240174 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/959d2ecf-daef-4d66-b1a3-153e0f52d094-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7nhqf\" (UID: \"959d2ecf-daef-4d66-b1a3-153e0f52d094\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7nhqf" Jul 21 00:51:30.240343 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:51:30.240216 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmdf5\" (UniqueName: \"kubernetes.io/projected/959d2ecf-daef-4d66-b1a3-153e0f52d094-kube-api-access-qmdf5\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7nhqf\" (UID: \"959d2ecf-daef-4d66-b1a3-153e0f52d094\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7nhqf" Jul 21 00:51:30.240984 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:51:30.240540 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/959d2ecf-daef-4d66-b1a3-153e0f52d094-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7nhqf\" (UID: \"959d2ecf-daef-4d66-b1a3-153e0f52d094\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7nhqf" Jul 21 00:51:30.240984 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:51:30.240628 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/959d2ecf-daef-4d66-b1a3-153e0f52d094-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7nhqf\" (UID: \"959d2ecf-daef-4d66-b1a3-153e0f52d094\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7nhqf" Jul 21 00:51:30.240984 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:51:30.240674 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/959d2ecf-daef-4d66-b1a3-153e0f52d094-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7nhqf\" (UID: \"959d2ecf-daef-4d66-b1a3-153e0f52d094\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7nhqf" Jul 21 00:51:30.242429 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:51:30.242410 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/959d2ecf-daef-4d66-b1a3-153e0f52d094-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7nhqf\" (UID: \"959d2ecf-daef-4d66-b1a3-153e0f52d094\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7nhqf" Jul 21 00:51:30.242815 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:51:30.242796 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/959d2ecf-daef-4d66-b1a3-153e0f52d094-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7nhqf\" (UID: \"959d2ecf-daef-4d66-b1a3-153e0f52d094\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7nhqf" Jul 21 00:51:30.248414 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:51:30.248385 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmdf5\" (UniqueName: \"kubernetes.io/projected/959d2ecf-daef-4d66-b1a3-153e0f52d094-kube-api-access-qmdf5\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7nhqf\" (UID: \"959d2ecf-daef-4d66-b1a3-153e0f52d094\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7nhqf" Jul 21 00:51:30.339534 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:51:30.339505 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7nhqf" Jul 21 00:51:30.477973 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:51:30.477944 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7nhqf"] Jul 21 00:51:30.480014 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:51:30.479978 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod959d2ecf_daef_4d66_b1a3_153e0f52d094.slice/crio-13c858743289894e665b82a7e215d0fed81a1e520d30f0699a0493f52382fc89 WatchSource:0}: Error finding container 13c858743289894e665b82a7e215d0fed81a1e520d30f0699a0493f52382fc89: Status 404 returned error can't find the container with id 13c858743289894e665b82a7e215d0fed81a1e520d30f0699a0493f52382fc89 Jul 21 00:51:30.916494 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:51:30.916428 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7nhqf" event={"ID":"959d2ecf-daef-4d66-b1a3-153e0f52d094","Type":"ContainerStarted","Data":"13c858743289894e665b82a7e215d0fed81a1e520d30f0699a0493f52382fc89"} Jul 21 00:51:31.915011 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:51:31.914978 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7nhqf" event={"ID":"959d2ecf-daef-4d66-b1a3-153e0f52d094","Type":"ContainerStarted","Data":"740a9f1ecf2236e5f32bbbe372f63684f8a63c9e92cd55eadb0d64bc4af953dd"} Jul 21 00:51:31.932038 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:51:31.931984 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7nhqf" podStartSLOduration=1.701430885 podStartE2EDuration="2.931968222s" totalImagesPullingTime="1.230537337s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:51:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:51:31.931614639 +0000 UTC m=+639.537723691" watchObservedRunningTime="2026-07-21 00:51:31.931968222 +0000 UTC m=+639.538077262" Jul 21 00:51:40.340349 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:51:40.340311 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7nhqf" Jul 21 00:51:40.340792 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:51:40.340376 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7nhqf" Jul 21 00:51:40.352653 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:51:40.352629 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7nhqf" Jul 21 00:51:40.966404 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:51:40.966376 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7nhqf" Jul 21 00:52:14.315165 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:52:14.315131 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-s4b8b"] Jul 21 00:52:14.322878 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:52:14.322843 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s4b8b" Jul 21 00:52:14.326483 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:52:14.326450 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 21 00:52:14.330760 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:52:14.330733 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-s4b8b"] Jul 21 00:52:14.431617 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:52:14.431586 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgftm\" (UniqueName: \"kubernetes.io/projected/58f90218-920c-48a6-b0d8-3373396703c0-kube-api-access-kgftm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s4b8b\" (UID: \"58f90218-920c-48a6-b0d8-3373396703c0\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s4b8b" Jul 21 00:52:14.431796 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:52:14.431635 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58f90218-920c-48a6-b0d8-3373396703c0-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s4b8b\" (UID: \"58f90218-920c-48a6-b0d8-3373396703c0\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s4b8b" Jul 21 00:52:14.431796 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:52:14.431656 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58f90218-920c-48a6-b0d8-3373396703c0-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s4b8b\" (UID: \"58f90218-920c-48a6-b0d8-3373396703c0\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s4b8b" Jul 21 00:52:14.431796 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:52:14.431680 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58f90218-920c-48a6-b0d8-3373396703c0-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s4b8b\" (UID: \"58f90218-920c-48a6-b0d8-3373396703c0\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s4b8b" Jul 21 00:52:14.431796 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:52:14.431717 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58f90218-920c-48a6-b0d8-3373396703c0-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s4b8b\" (UID: \"58f90218-920c-48a6-b0d8-3373396703c0\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s4b8b" Jul 21 00:52:14.431796 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:52:14.431740 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58f90218-920c-48a6-b0d8-3373396703c0-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s4b8b\" (UID: \"58f90218-920c-48a6-b0d8-3373396703c0\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s4b8b" Jul 21 00:52:14.532321 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:52:14.532284 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58f90218-920c-48a6-b0d8-3373396703c0-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s4b8b\" (UID: \"58f90218-920c-48a6-b0d8-3373396703c0\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s4b8b" Jul 21 00:52:14.532321 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:52:14.532324 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58f90218-920c-48a6-b0d8-3373396703c0-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s4b8b\" (UID: \"58f90218-920c-48a6-b0d8-3373396703c0\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s4b8b" Jul 21 00:52:14.532564 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:52:14.532343 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58f90218-920c-48a6-b0d8-3373396703c0-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s4b8b\" (UID: \"58f90218-920c-48a6-b0d8-3373396703c0\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s4b8b" Jul 21 00:52:14.532564 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:52:14.532470 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58f90218-920c-48a6-b0d8-3373396703c0-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s4b8b\" (UID: \"58f90218-920c-48a6-b0d8-3373396703c0\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s4b8b" Jul 21 00:52:14.532564 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:52:14.532522 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58f90218-920c-48a6-b0d8-3373396703c0-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s4b8b\" (UID: \"58f90218-920c-48a6-b0d8-3373396703c0\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s4b8b" Jul 21 00:52:14.532731 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:52:14.532653 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgftm\" (UniqueName: \"kubernetes.io/projected/58f90218-920c-48a6-b0d8-3373396703c0-kube-api-access-kgftm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s4b8b\" (UID: \"58f90218-920c-48a6-b0d8-3373396703c0\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s4b8b" Jul 21 00:52:14.532791 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:52:14.532740 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58f90218-920c-48a6-b0d8-3373396703c0-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s4b8b\" (UID: \"58f90218-920c-48a6-b0d8-3373396703c0\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s4b8b" Jul 21 00:52:14.532830 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:52:14.532798 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58f90218-920c-48a6-b0d8-3373396703c0-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s4b8b\" (UID: \"58f90218-920c-48a6-b0d8-3373396703c0\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s4b8b" Jul 21 00:52:14.532944 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:52:14.532918 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58f90218-920c-48a6-b0d8-3373396703c0-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s4b8b\" (UID: \"58f90218-920c-48a6-b0d8-3373396703c0\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s4b8b" Jul 21 00:52:14.534576 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:52:14.534550 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58f90218-920c-48a6-b0d8-3373396703c0-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s4b8b\" (UID: \"58f90218-920c-48a6-b0d8-3373396703c0\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s4b8b" Jul 21 00:52:14.534821 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:52:14.534806 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58f90218-920c-48a6-b0d8-3373396703c0-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s4b8b\" (UID: \"58f90218-920c-48a6-b0d8-3373396703c0\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s4b8b" Jul 21 00:52:14.540104 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:52:14.540086 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgftm\" (UniqueName: \"kubernetes.io/projected/58f90218-920c-48a6-b0d8-3373396703c0-kube-api-access-kgftm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-s4b8b\" (UID: \"58f90218-920c-48a6-b0d8-3373396703c0\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s4b8b" Jul 21 00:52:14.636615 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:52:14.636578 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s4b8b" Jul 21 00:52:14.771868 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:52:14.771825 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-s4b8b"] Jul 21 00:52:14.774630 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:52:14.774602 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58f90218_920c_48a6_b0d8_3373396703c0.slice/crio-4e9100d4e9bdeebda89ddb85a0394105c2fae475e55557bacd7471338f33470d WatchSource:0}: Error finding container 4e9100d4e9bdeebda89ddb85a0394105c2fae475e55557bacd7471338f33470d: Status 404 returned error can't find the container with id 4e9100d4e9bdeebda89ddb85a0394105c2fae475e55557bacd7471338f33470d Jul 21 00:52:14.776259 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:52:14.776225 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 00:52:15.125247 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:52:15.125207 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s4b8b" event={"ID":"58f90218-920c-48a6-b0d8-3373396703c0","Type":"ContainerStarted","Data":"aaab99323196da4315b0b25e20714a07f2fe272fc533f8d5579c21e11c4c5a52"} Jul 21 00:52:15.125449 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:52:15.125279 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s4b8b" event={"ID":"58f90218-920c-48a6-b0d8-3373396703c0","Type":"ContainerStarted","Data":"4e9100d4e9bdeebda89ddb85a0394105c2fae475e55557bacd7471338f33470d"} Jul 21 00:52:24.637536 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:52:24.637499 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s4b8b" Jul 21 00:52:24.638003 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:52:24.637583 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s4b8b" Jul 21 00:52:24.649556 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:52:24.649533 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s4b8b" Jul 21 00:52:24.668848 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:52:24.668804 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s4b8b" podStartSLOduration=10.506486446 podStartE2EDuration="10.66878992s" totalImagesPullingTime="162.303474ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:52:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:52:15.140696649 +0000 UTC m=+682.746805691" watchObservedRunningTime="2026-07-21 00:52:24.66878992 +0000 UTC m=+692.274898958" Jul 21 00:52:25.188197 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:52:25.188167 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-s4b8b" Jul 21 00:52:45.799734 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:52:45.799649 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-hw24q"] Jul 21 00:52:45.803773 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:52:45.803753 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hw24q" Jul 21 00:52:45.806434 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:52:45.806411 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 21 00:52:45.812713 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:52:45.812523 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-hw24q"] Jul 21 00:52:45.915797 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:52:45.915763 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db171be2-aedf-4c85-b387-c90902e082b9-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hw24q\" (UID: \"db171be2-aedf-4c85-b387-c90902e082b9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hw24q" Jul 21 00:52:45.915971 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:52:45.915817 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-462wj\" (UniqueName: \"kubernetes.io/projected/db171be2-aedf-4c85-b387-c90902e082b9-kube-api-access-462wj\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hw24q\" (UID: \"db171be2-aedf-4c85-b387-c90902e082b9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hw24q" Jul 21 00:52:45.915971 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:52:45.915856 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db171be2-aedf-4c85-b387-c90902e082b9-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hw24q\" (UID: \"db171be2-aedf-4c85-b387-c90902e082b9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hw24q" Jul 21 00:52:45.915971 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:52:45.915937 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db171be2-aedf-4c85-b387-c90902e082b9-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hw24q\" (UID: \"db171be2-aedf-4c85-b387-c90902e082b9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hw24q" Jul 21 00:52:45.916114 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:52:45.915985 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db171be2-aedf-4c85-b387-c90902e082b9-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hw24q\" (UID: \"db171be2-aedf-4c85-b387-c90902e082b9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hw24q" Jul 21 00:52:45.916114 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:52:45.916015 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db171be2-aedf-4c85-b387-c90902e082b9-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hw24q\" (UID: \"db171be2-aedf-4c85-b387-c90902e082b9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hw24q" Jul 21 00:52:46.017072 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:52:46.017037 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db171be2-aedf-4c85-b387-c90902e082b9-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hw24q\" (UID: \"db171be2-aedf-4c85-b387-c90902e082b9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hw24q" Jul 21 00:52:46.017282 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:52:46.017095 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-462wj\" (UniqueName: \"kubernetes.io/projected/db171be2-aedf-4c85-b387-c90902e082b9-kube-api-access-462wj\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hw24q\" (UID: \"db171be2-aedf-4c85-b387-c90902e082b9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hw24q" Jul 21 00:52:46.017282 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:52:46.017135 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db171be2-aedf-4c85-b387-c90902e082b9-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hw24q\" (UID: \"db171be2-aedf-4c85-b387-c90902e082b9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hw24q" Jul 21 00:52:46.017421 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:52:46.017345 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db171be2-aedf-4c85-b387-c90902e082b9-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hw24q\" (UID: \"db171be2-aedf-4c85-b387-c90902e082b9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hw24q" Jul 21 00:52:46.017421 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:52:46.017386 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db171be2-aedf-4c85-b387-c90902e082b9-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hw24q\" (UID: \"db171be2-aedf-4c85-b387-c90902e082b9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hw24q" Jul 21 00:52:46.017535 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:52:46.017458 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db171be2-aedf-4c85-b387-c90902e082b9-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hw24q\" (UID: \"db171be2-aedf-4c85-b387-c90902e082b9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hw24q" Jul 21 00:52:46.017535 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:52:46.017467 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db171be2-aedf-4c85-b387-c90902e082b9-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hw24q\" (UID: \"db171be2-aedf-4c85-b387-c90902e082b9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hw24q" Jul 21 00:52:46.017634 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:52:46.017563 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db171be2-aedf-4c85-b387-c90902e082b9-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hw24q\" (UID: \"db171be2-aedf-4c85-b387-c90902e082b9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hw24q" Jul 21 00:52:46.017684 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:52:46.017668 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db171be2-aedf-4c85-b387-c90902e082b9-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hw24q\" (UID: \"db171be2-aedf-4c85-b387-c90902e082b9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hw24q" Jul 21 00:52:46.019615 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:52:46.019593 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db171be2-aedf-4c85-b387-c90902e082b9-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hw24q\" (UID: \"db171be2-aedf-4c85-b387-c90902e082b9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hw24q" Jul 21 00:52:46.019855 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:52:46.019840 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db171be2-aedf-4c85-b387-c90902e082b9-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hw24q\" (UID: \"db171be2-aedf-4c85-b387-c90902e082b9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hw24q" Jul 21 00:52:46.025327 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:52:46.025304 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-462wj\" (UniqueName: \"kubernetes.io/projected/db171be2-aedf-4c85-b387-c90902e082b9-kube-api-access-462wj\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-hw24q\" (UID: \"db171be2-aedf-4c85-b387-c90902e082b9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hw24q" Jul 21 00:52:46.116763 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:52:46.116730 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hw24q" Jul 21 00:52:46.262058 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:52:46.262031 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-hw24q"] Jul 21 00:52:46.264445 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:52:46.264419 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb171be2_aedf_4c85_b387_c90902e082b9.slice/crio-1141939afb8c746462a2c32af3525ba38afcac8b2f8af10b4c9ea727073ac683 WatchSource:0}: Error finding container 1141939afb8c746462a2c32af3525ba38afcac8b2f8af10b4c9ea727073ac683: Status 404 returned error can't find the container with id 1141939afb8c746462a2c32af3525ba38afcac8b2f8af10b4c9ea727073ac683 Jul 21 00:52:46.284419 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:52:46.284389 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hw24q" event={"ID":"db171be2-aedf-4c85-b387-c90902e082b9","Type":"ContainerStarted","Data":"1141939afb8c746462a2c32af3525ba38afcac8b2f8af10b4c9ea727073ac683"} Jul 21 00:52:47.290836 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:52:47.290798 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hw24q" event={"ID":"db171be2-aedf-4c85-b387-c90902e082b9","Type":"ContainerStarted","Data":"b38386e745fe7ea20d9c1527bf742bc5a472f7a7155790d66650f8774b4d5c0e"} Jul 21 00:52:47.313745 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:52:47.313700 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hw24q" podStartSLOduration=2.1394967 podStartE2EDuration="2.313684713s" totalImagesPullingTime="174.188013ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:52:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:52:47.312690746 +0000 UTC m=+714.918799786" watchObservedRunningTime="2026-07-21 00:52:47.313684713 +0000 UTC m=+714.919793766" Jul 21 00:52:56.117174 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:52:56.117127 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hw24q" Jul 21 00:52:56.117174 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:52:56.117191 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hw24q" Jul 21 00:52:56.129479 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:52:56.129452 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hw24q" Jul 21 00:52:56.347107 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:52:56.347080 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-hw24q" Jul 21 00:54:52.774958 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:54:52.774920 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7b9456ddfc-nvvzj"] Jul 21 00:54:52.778443 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:54:52.778425 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7b9456ddfc-nvvzj" Jul 21 00:54:52.792712 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:54:52.792687 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7b9456ddfc-nvvzj"] Jul 21 00:54:52.879293 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:54:52.879256 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zstjp\" (UniqueName: \"kubernetes.io/projected/b48d40a2-bf25-42bb-95df-b088de4ac80a-kube-api-access-zstjp\") pod \"authorino-7b9456ddfc-nvvzj\" (UID: \"b48d40a2-bf25-42bb-95df-b088de4ac80a\") " pod="kuadrant-system/authorino-7b9456ddfc-nvvzj" Jul 21 00:54:52.879496 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:54:52.879365 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b48d40a2-bf25-42bb-95df-b088de4ac80a-tls-cert\") pod \"authorino-7b9456ddfc-nvvzj\" (UID: \"b48d40a2-bf25-42bb-95df-b088de4ac80a\") " pod="kuadrant-system/authorino-7b9456ddfc-nvvzj" Jul 21 00:54:52.980869 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:54:52.980835 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b48d40a2-bf25-42bb-95df-b088de4ac80a-tls-cert\") pod \"authorino-7b9456ddfc-nvvzj\" (UID: \"b48d40a2-bf25-42bb-95df-b088de4ac80a\") " pod="kuadrant-system/authorino-7b9456ddfc-nvvzj" Jul 21 00:54:52.981035 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:54:52.980887 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zstjp\" (UniqueName: \"kubernetes.io/projected/b48d40a2-bf25-42bb-95df-b088de4ac80a-kube-api-access-zstjp\") pod \"authorino-7b9456ddfc-nvvzj\" (UID: \"b48d40a2-bf25-42bb-95df-b088de4ac80a\") " pod="kuadrant-system/authorino-7b9456ddfc-nvvzj" Jul 21 00:54:52.983212 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:54:52.983194 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b48d40a2-bf25-42bb-95df-b088de4ac80a-tls-cert\") pod \"authorino-7b9456ddfc-nvvzj\" (UID: \"b48d40a2-bf25-42bb-95df-b088de4ac80a\") " pod="kuadrant-system/authorino-7b9456ddfc-nvvzj" Jul 21 00:54:52.988553 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:54:52.988531 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zstjp\" (UniqueName: \"kubernetes.io/projected/b48d40a2-bf25-42bb-95df-b088de4ac80a-kube-api-access-zstjp\") pod \"authorino-7b9456ddfc-nvvzj\" (UID: \"b48d40a2-bf25-42bb-95df-b088de4ac80a\") " pod="kuadrant-system/authorino-7b9456ddfc-nvvzj" Jul 21 00:54:53.087616 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:54:53.087580 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7b9456ddfc-nvvzj" Jul 21 00:54:53.228134 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:54:53.228105 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7b9456ddfc-nvvzj"] Jul 21 00:54:53.229761 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:54:53.229736 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb48d40a2_bf25_42bb_95df_b088de4ac80a.slice/crio-84ce4ee021262f3da6aa82b55b0a32380577e6710a5f89ccbef4d6fe5cbe8ec6 WatchSource:0}: Error finding container 84ce4ee021262f3da6aa82b55b0a32380577e6710a5f89ccbef4d6fe5cbe8ec6: Status 404 returned error can't find the container with id 84ce4ee021262f3da6aa82b55b0a32380577e6710a5f89ccbef4d6fe5cbe8ec6 Jul 21 00:54:53.927874 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:54:53.927778 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7b9456ddfc-nvvzj" event={"ID":"b48d40a2-bf25-42bb-95df-b088de4ac80a","Type":"ContainerStarted","Data":"97b477d6386deea8807e38fa4dee0923b0d7c94e94c4a4990038c31ca9417cdd"} Jul 21 00:54:53.927874 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:54:53.927831 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7b9456ddfc-nvvzj" event={"ID":"b48d40a2-bf25-42bb-95df-b088de4ac80a","Type":"ContainerStarted","Data":"84ce4ee021262f3da6aa82b55b0a32380577e6710a5f89ccbef4d6fe5cbe8ec6"} Jul 21 00:54:53.954101 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:54:53.954025 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7b9456ddfc-nvvzj" podStartSLOduration=1.5334655659999998 podStartE2EDuration="1.954000722s" totalImagesPullingTime="420.535156ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:54:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:54:53.951886552 +0000 UTC m=+841.557995593" watchObservedRunningTime="2026-07-21 00:54:53.954000722 +0000 UTC m=+841.560109763" Jul 21 00:54:53.985709 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:54:53.985606 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-769465ddbd-v46sv"] Jul 21 00:54:53.985890 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:54:53.985842 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-769465ddbd-v46sv" podUID="a23ae0de-9373-4024-aa25-4aeb5405243d" containerName="authorino" containerID="cri-o://0097f0bad971886e664e6a49402074f01c6e3d662d6d3b4a69aa571b632af484" gracePeriod=30 Jul 21 00:54:54.264791 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:54:54.264759 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-769465ddbd-v46sv" Jul 21 00:54:54.393297 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:54:54.393262 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a23ae0de-9373-4024-aa25-4aeb5405243d-tls-cert\") pod \"a23ae0de-9373-4024-aa25-4aeb5405243d\" (UID: \"a23ae0de-9373-4024-aa25-4aeb5405243d\") " Jul 21 00:54:54.393513 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:54:54.393323 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p7st7\" (UniqueName: \"kubernetes.io/projected/a23ae0de-9373-4024-aa25-4aeb5405243d-kube-api-access-p7st7\") pod \"a23ae0de-9373-4024-aa25-4aeb5405243d\" (UID: \"a23ae0de-9373-4024-aa25-4aeb5405243d\") " Jul 21 00:54:54.395407 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:54:54.395369 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a23ae0de-9373-4024-aa25-4aeb5405243d-kube-api-access-p7st7" (OuterVolumeSpecName: "kube-api-access-p7st7") pod "a23ae0de-9373-4024-aa25-4aeb5405243d" (UID: "a23ae0de-9373-4024-aa25-4aeb5405243d"). InnerVolumeSpecName "kube-api-access-p7st7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 00:54:54.404221 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:54:54.404189 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a23ae0de-9373-4024-aa25-4aeb5405243d-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "a23ae0de-9373-4024-aa25-4aeb5405243d" (UID: "a23ae0de-9373-4024-aa25-4aeb5405243d"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 00:54:54.494339 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:54:54.494252 2580 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a23ae0de-9373-4024-aa25-4aeb5405243d-tls-cert\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 21 00:54:54.494339 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:54:54.494286 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7st7\" (UniqueName: \"kubernetes.io/projected/a23ae0de-9373-4024-aa25-4aeb5405243d-kube-api-access-p7st7\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 21 00:54:54.934443 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:54:54.934408 2580 generic.go:356] "Generic (PLEG): container finished" podID="a23ae0de-9373-4024-aa25-4aeb5405243d" containerID="0097f0bad971886e664e6a49402074f01c6e3d662d6d3b4a69aa571b632af484" exitCode=0 Jul 21 00:54:54.934895 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:54:54.934470 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-769465ddbd-v46sv" Jul 21 00:54:54.934895 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:54:54.934503 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-769465ddbd-v46sv" event={"ID":"a23ae0de-9373-4024-aa25-4aeb5405243d","Type":"ContainerDied","Data":"0097f0bad971886e664e6a49402074f01c6e3d662d6d3b4a69aa571b632af484"} Jul 21 00:54:54.934895 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:54:54.934551 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-769465ddbd-v46sv" event={"ID":"a23ae0de-9373-4024-aa25-4aeb5405243d","Type":"ContainerDied","Data":"0558d4d39651518f60dff18e3dae371aea108d9269916f362faf5cab8e95fea6"} Jul 21 00:54:54.934895 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:54:54.934578 2580 scope.go:117] "RemoveContainer" containerID="0097f0bad971886e664e6a49402074f01c6e3d662d6d3b4a69aa571b632af484" Jul 21 00:54:54.944454 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:54:54.944429 2580 scope.go:117] "RemoveContainer" containerID="0097f0bad971886e664e6a49402074f01c6e3d662d6d3b4a69aa571b632af484" Jul 21 00:54:54.944760 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:54:54.944736 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0097f0bad971886e664e6a49402074f01c6e3d662d6d3b4a69aa571b632af484\": container with ID starting with 0097f0bad971886e664e6a49402074f01c6e3d662d6d3b4a69aa571b632af484 not found: ID does not exist" containerID="0097f0bad971886e664e6a49402074f01c6e3d662d6d3b4a69aa571b632af484" Jul 21 00:54:54.944859 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:54:54.944771 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0097f0bad971886e664e6a49402074f01c6e3d662d6d3b4a69aa571b632af484"} err="failed to get container status \"0097f0bad971886e664e6a49402074f01c6e3d662d6d3b4a69aa571b632af484\": rpc error: code = NotFound desc = could not find container \"0097f0bad971886e664e6a49402074f01c6e3d662d6d3b4a69aa571b632af484\": container with ID starting with 0097f0bad971886e664e6a49402074f01c6e3d662d6d3b4a69aa571b632af484 not found: ID does not exist" Jul 21 00:54:54.954179 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:54:54.954151 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-769465ddbd-v46sv"] Jul 21 00:54:54.957101 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:54:54.957080 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-769465ddbd-v46sv"] Jul 21 00:54:56.912707 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:54:56.912671 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a23ae0de-9373-4024-aa25-4aeb5405243d" path="/var/lib/kubelet/pods/a23ae0de-9373-4024-aa25-4aeb5405243d/volumes" Jul 21 00:55:40.448150 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:55:40.448114 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7765874568-gqgjs"] Jul 21 00:55:40.448645 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:55:40.448370 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7765874568-gqgjs" podUID="32111504-497b-4f55-ba47-a941a37019b4" containerName="manager" containerID="cri-o://b8f90a7691558e9bd2ae67b482cb498ef9de9f4aa82bc0f3a9dc6a03bcf6f41d" gracePeriod=30 Jul 21 00:55:40.719851 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:55:40.719827 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7765874568-gqgjs" Jul 21 00:55:40.838793 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:55:40.838763 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/32111504-497b-4f55-ba47-a941a37019b4-webhook-cert\") pod \"32111504-497b-4f55-ba47-a941a37019b4\" (UID: \"32111504-497b-4f55-ba47-a941a37019b4\") " Jul 21 00:55:40.838961 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:55:40.838817 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m842c\" (UniqueName: \"kubernetes.io/projected/32111504-497b-4f55-ba47-a941a37019b4-kube-api-access-m842c\") pod \"32111504-497b-4f55-ba47-a941a37019b4\" (UID: \"32111504-497b-4f55-ba47-a941a37019b4\") " Jul 21 00:55:40.840894 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:55:40.840865 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32111504-497b-4f55-ba47-a941a37019b4-kube-api-access-m842c" (OuterVolumeSpecName: "kube-api-access-m842c") pod "32111504-497b-4f55-ba47-a941a37019b4" (UID: "32111504-497b-4f55-ba47-a941a37019b4"). InnerVolumeSpecName "kube-api-access-m842c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 00:55:40.841003 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:55:40.840920 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32111504-497b-4f55-ba47-a941a37019b4-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "32111504-497b-4f55-ba47-a941a37019b4" (UID: "32111504-497b-4f55-ba47-a941a37019b4"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 00:55:40.940132 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:55:40.940106 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m842c\" (UniqueName: \"kubernetes.io/projected/32111504-497b-4f55-ba47-a941a37019b4-kube-api-access-m842c\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 21 00:55:40.940132 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:55:40.940130 2580 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/32111504-497b-4f55-ba47-a941a37019b4-webhook-cert\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 21 00:55:41.179130 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:55:41.179092 2580 generic.go:356] "Generic (PLEG): container finished" podID="32111504-497b-4f55-ba47-a941a37019b4" containerID="b8f90a7691558e9bd2ae67b482cb498ef9de9f4aa82bc0f3a9dc6a03bcf6f41d" exitCode=0 Jul 21 00:55:41.179355 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:55:41.179137 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7765874568-gqgjs" Jul 21 00:55:41.179355 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:55:41.179161 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7765874568-gqgjs" event={"ID":"32111504-497b-4f55-ba47-a941a37019b4","Type":"ContainerDied","Data":"b8f90a7691558e9bd2ae67b482cb498ef9de9f4aa82bc0f3a9dc6a03bcf6f41d"} Jul 21 00:55:41.179355 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:55:41.179213 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7765874568-gqgjs" event={"ID":"32111504-497b-4f55-ba47-a941a37019b4","Type":"ContainerDied","Data":"74ecc26551428e940edd5de3287d95bd8ec23542913494ab822975d49973b10e"} Jul 21 00:55:41.179355 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:55:41.179269 2580 scope.go:117] "RemoveContainer" containerID="b8f90a7691558e9bd2ae67b482cb498ef9de9f4aa82bc0f3a9dc6a03bcf6f41d" Jul 21 00:55:41.188360 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:55:41.188259 2580 scope.go:117] "RemoveContainer" containerID="b8f90a7691558e9bd2ae67b482cb498ef9de9f4aa82bc0f3a9dc6a03bcf6f41d" Jul 21 00:55:41.188564 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:55:41.188545 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8f90a7691558e9bd2ae67b482cb498ef9de9f4aa82bc0f3a9dc6a03bcf6f41d\": container with ID starting with b8f90a7691558e9bd2ae67b482cb498ef9de9f4aa82bc0f3a9dc6a03bcf6f41d not found: ID does not exist" containerID="b8f90a7691558e9bd2ae67b482cb498ef9de9f4aa82bc0f3a9dc6a03bcf6f41d" Jul 21 00:55:41.188614 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:55:41.188574 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8f90a7691558e9bd2ae67b482cb498ef9de9f4aa82bc0f3a9dc6a03bcf6f41d"} err="failed to get container status \"b8f90a7691558e9bd2ae67b482cb498ef9de9f4aa82bc0f3a9dc6a03bcf6f41d\": rpc error: code = NotFound desc = could not find container \"b8f90a7691558e9bd2ae67b482cb498ef9de9f4aa82bc0f3a9dc6a03bcf6f41d\": container with ID starting with b8f90a7691558e9bd2ae67b482cb498ef9de9f4aa82bc0f3a9dc6a03bcf6f41d not found: ID does not exist" Jul 21 00:55:41.195941 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:55:41.195920 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7765874568-gqgjs"] Jul 21 00:55:41.199345 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:55:41.199323 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7765874568-gqgjs"] Jul 21 00:55:42.912079 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:55:42.912047 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32111504-497b-4f55-ba47-a941a37019b4" path="/var/lib/kubelet/pods/32111504-497b-4f55-ba47-a941a37019b4/volumes" Jul 21 00:55:52.802137 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:55:52.802062 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-rb4ph_3ee1dbf0-b4ff-4871-9432-284eb25f97b1/console-operator/1.log" Jul 21 00:55:52.802137 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:55:52.802062 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-rb4ph_3ee1dbf0-b4ff-4871-9432-284eb25f97b1/console-operator/1.log" Jul 21 00:56:41.992850 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:56:41.992814 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7765874568-fw2xb"] Jul 21 00:56:41.995464 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:56:41.993224 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32111504-497b-4f55-ba47-a941a37019b4" containerName="manager" Jul 21 00:56:41.995464 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:56:41.993247 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="32111504-497b-4f55-ba47-a941a37019b4" containerName="manager" Jul 21 00:56:41.995464 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:56:41.993268 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a23ae0de-9373-4024-aa25-4aeb5405243d" containerName="authorino" Jul 21 00:56:41.995464 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:56:41.993273 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a23ae0de-9373-4024-aa25-4aeb5405243d" containerName="authorino" Jul 21 00:56:41.995464 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:56:41.993336 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a23ae0de-9373-4024-aa25-4aeb5405243d" containerName="authorino" Jul 21 00:56:41.995464 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:56:41.993345 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="32111504-497b-4f55-ba47-a941a37019b4" containerName="manager" Jul 21 00:56:41.996282 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:56:41.996265 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7765874568-fw2xb" Jul 21 00:56:41.998850 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:56:41.998828 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-4hxcg\"" Jul 21 00:56:41.998850 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:56:41.998844 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 21 00:56:41.999975 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:56:41.999959 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 21 00:56:42.007843 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:56:42.007817 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7765874568-fw2xb"] Jul 21 00:56:42.095474 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:56:42.095443 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5feba319-6017-4b67-bb5c-624a0027c50f-webhook-cert\") pod \"maas-controller-7765874568-fw2xb\" (UID: \"5feba319-6017-4b67-bb5c-624a0027c50f\") " pod="opendatahub/maas-controller-7765874568-fw2xb" Jul 21 00:56:42.095639 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:56:42.095518 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dnth\" (UniqueName: \"kubernetes.io/projected/5feba319-6017-4b67-bb5c-624a0027c50f-kube-api-access-6dnth\") pod \"maas-controller-7765874568-fw2xb\" (UID: \"5feba319-6017-4b67-bb5c-624a0027c50f\") " pod="opendatahub/maas-controller-7765874568-fw2xb" Jul 21 00:56:42.196105 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:56:42.196068 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5feba319-6017-4b67-bb5c-624a0027c50f-webhook-cert\") pod \"maas-controller-7765874568-fw2xb\" (UID: \"5feba319-6017-4b67-bb5c-624a0027c50f\") " pod="opendatahub/maas-controller-7765874568-fw2xb" Jul 21 00:56:42.196284 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:56:42.196147 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dnth\" (UniqueName: \"kubernetes.io/projected/5feba319-6017-4b67-bb5c-624a0027c50f-kube-api-access-6dnth\") pod \"maas-controller-7765874568-fw2xb\" (UID: \"5feba319-6017-4b67-bb5c-624a0027c50f\") " pod="opendatahub/maas-controller-7765874568-fw2xb" Jul 21 00:56:42.198558 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:56:42.198529 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5feba319-6017-4b67-bb5c-624a0027c50f-webhook-cert\") pod \"maas-controller-7765874568-fw2xb\" (UID: \"5feba319-6017-4b67-bb5c-624a0027c50f\") " pod="opendatahub/maas-controller-7765874568-fw2xb" Jul 21 00:56:42.205062 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:56:42.205035 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dnth\" (UniqueName: \"kubernetes.io/projected/5feba319-6017-4b67-bb5c-624a0027c50f-kube-api-access-6dnth\") pod \"maas-controller-7765874568-fw2xb\" (UID: \"5feba319-6017-4b67-bb5c-624a0027c50f\") " pod="opendatahub/maas-controller-7765874568-fw2xb" Jul 21 00:56:42.308061 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:56:42.307962 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7765874568-fw2xb" Jul 21 00:56:42.647324 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:56:42.647181 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7765874568-fw2xb"] Jul 21 00:56:42.650223 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:56:42.650182 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5feba319_6017_4b67_bb5c_624a0027c50f.slice/crio-7f496d5498fe36aef14c45033013ddaca1340559edc73cfbe7cddcc412966ea3 WatchSource:0}: Error finding container 7f496d5498fe36aef14c45033013ddaca1340559edc73cfbe7cddcc412966ea3: Status 404 returned error can't find the container with id 7f496d5498fe36aef14c45033013ddaca1340559edc73cfbe7cddcc412966ea3 Jul 21 00:56:43.501707 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:56:43.501625 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7765874568-fw2xb" event={"ID":"5feba319-6017-4b67-bb5c-624a0027c50f","Type":"ContainerStarted","Data":"167ef1cd33a2577c30a2376a8d9063aa594076d96f8819eedcd2aae43bc3173d"} Jul 21 00:56:43.501707 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:56:43.501686 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7765874568-fw2xb" Jul 21 00:56:43.501707 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:56:43.501701 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7765874568-fw2xb" event={"ID":"5feba319-6017-4b67-bb5c-624a0027c50f","Type":"ContainerStarted","Data":"7f496d5498fe36aef14c45033013ddaca1340559edc73cfbe7cddcc412966ea3"} Jul 21 00:56:43.518756 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:56:43.518707 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7765874568-fw2xb" podStartSLOduration=2.058462842 podStartE2EDuration="2.518692696s" totalImagesPullingTime="460.229854ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:56:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:56:43.517044006 +0000 UTC m=+951.123153047" watchObservedRunningTime="2026-07-21 00:56:43.518692696 +0000 UTC m=+951.124801772" Jul 21 00:56:54.509098 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:56:54.509069 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7765874568-fw2xb" Jul 21 00:56:59.139228 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:56:59.139183 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7765874568-fw2xb"] Jul 21 00:56:59.139694 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:56:59.139471 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7765874568-fw2xb" podUID="5feba319-6017-4b67-bb5c-624a0027c50f" containerName="manager" containerID="cri-o://167ef1cd33a2577c30a2376a8d9063aa594076d96f8819eedcd2aae43bc3173d" gracePeriod=30 Jul 21 00:56:59.402074 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:56:59.402009 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7765874568-fw2xb" Jul 21 00:56:59.442354 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:56:59.442322 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6dnth\" (UniqueName: \"kubernetes.io/projected/5feba319-6017-4b67-bb5c-624a0027c50f-kube-api-access-6dnth\") pod \"5feba319-6017-4b67-bb5c-624a0027c50f\" (UID: \"5feba319-6017-4b67-bb5c-624a0027c50f\") " Jul 21 00:56:59.442523 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:56:59.442443 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5feba319-6017-4b67-bb5c-624a0027c50f-webhook-cert\") pod \"5feba319-6017-4b67-bb5c-624a0027c50f\" (UID: \"5feba319-6017-4b67-bb5c-624a0027c50f\") " Jul 21 00:56:59.444529 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:56:59.444499 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5feba319-6017-4b67-bb5c-624a0027c50f-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "5feba319-6017-4b67-bb5c-624a0027c50f" (UID: "5feba319-6017-4b67-bb5c-624a0027c50f"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 00:56:59.444827 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:56:59.444796 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5feba319-6017-4b67-bb5c-624a0027c50f-kube-api-access-6dnth" (OuterVolumeSpecName: "kube-api-access-6dnth") pod "5feba319-6017-4b67-bb5c-624a0027c50f" (UID: "5feba319-6017-4b67-bb5c-624a0027c50f"). InnerVolumeSpecName "kube-api-access-6dnth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 00:56:59.543830 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:56:59.543796 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dnth\" (UniqueName: \"kubernetes.io/projected/5feba319-6017-4b67-bb5c-624a0027c50f-kube-api-access-6dnth\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 21 00:56:59.543830 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:56:59.543824 2580 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5feba319-6017-4b67-bb5c-624a0027c50f-webhook-cert\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 21 00:56:59.583742 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:56:59.583711 2580 generic.go:356] "Generic (PLEG): container finished" podID="5feba319-6017-4b67-bb5c-624a0027c50f" containerID="167ef1cd33a2577c30a2376a8d9063aa594076d96f8819eedcd2aae43bc3173d" exitCode=0 Jul 21 00:56:59.583887 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:56:59.583751 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7765874568-fw2xb" Jul 21 00:56:59.583887 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:56:59.583793 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7765874568-fw2xb" event={"ID":"5feba319-6017-4b67-bb5c-624a0027c50f","Type":"ContainerDied","Data":"167ef1cd33a2577c30a2376a8d9063aa594076d96f8819eedcd2aae43bc3173d"} Jul 21 00:56:59.583887 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:56:59.583839 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7765874568-fw2xb" event={"ID":"5feba319-6017-4b67-bb5c-624a0027c50f","Type":"ContainerDied","Data":"7f496d5498fe36aef14c45033013ddaca1340559edc73cfbe7cddcc412966ea3"} Jul 21 00:56:59.583887 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:56:59.583861 2580 scope.go:117] "RemoveContainer" containerID="167ef1cd33a2577c30a2376a8d9063aa594076d96f8819eedcd2aae43bc3173d" Jul 21 00:56:59.593191 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:56:59.593174 2580 scope.go:117] "RemoveContainer" containerID="167ef1cd33a2577c30a2376a8d9063aa594076d96f8819eedcd2aae43bc3173d" Jul 21 00:56:59.593483 ip-10-0-129-137 kubenswrapper[2580]: E0721 00:56:59.593457 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"167ef1cd33a2577c30a2376a8d9063aa594076d96f8819eedcd2aae43bc3173d\": container with ID starting with 167ef1cd33a2577c30a2376a8d9063aa594076d96f8819eedcd2aae43bc3173d not found: ID does not exist" containerID="167ef1cd33a2577c30a2376a8d9063aa594076d96f8819eedcd2aae43bc3173d" Jul 21 00:56:59.593542 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:56:59.593493 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"167ef1cd33a2577c30a2376a8d9063aa594076d96f8819eedcd2aae43bc3173d"} err="failed to get container status \"167ef1cd33a2577c30a2376a8d9063aa594076d96f8819eedcd2aae43bc3173d\": rpc error: code = NotFound desc = could not find container \"167ef1cd33a2577c30a2376a8d9063aa594076d96f8819eedcd2aae43bc3173d\": container with ID starting with 167ef1cd33a2577c30a2376a8d9063aa594076d96f8819eedcd2aae43bc3173d not found: ID does not exist" Jul 21 00:56:59.605981 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:56:59.605953 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7765874568-fw2xb"] Jul 21 00:56:59.611043 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:56:59.611023 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7765874568-fw2xb"] Jul 21 00:57:00.408716 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:57:00.408686 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7765874568-dn64s"] Jul 21 00:57:00.409329 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:57:00.409309 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5feba319-6017-4b67-bb5c-624a0027c50f" containerName="manager" Jul 21 00:57:00.409391 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:57:00.409333 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5feba319-6017-4b67-bb5c-624a0027c50f" containerName="manager" Jul 21 00:57:00.409444 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:57:00.409433 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5feba319-6017-4b67-bb5c-624a0027c50f" containerName="manager" Jul 21 00:57:00.413837 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:57:00.413821 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7765874568-dn64s" Jul 21 00:57:00.417666 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:57:00.417646 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-4hxcg\"" Jul 21 00:57:00.417790 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:57:00.417657 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 21 00:57:00.417790 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:57:00.417705 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 21 00:57:00.424598 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:57:00.423130 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7765874568-dn64s"] Jul 21 00:57:00.453494 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:57:00.453468 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb49f\" (UniqueName: \"kubernetes.io/projected/017bf68e-1d4d-4145-b90b-ee6ad6d0100f-kube-api-access-hb49f\") pod \"maas-controller-7765874568-dn64s\" (UID: \"017bf68e-1d4d-4145-b90b-ee6ad6d0100f\") " pod="opendatahub/maas-controller-7765874568-dn64s" Jul 21 00:57:00.453626 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:57:00.453505 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/017bf68e-1d4d-4145-b90b-ee6ad6d0100f-webhook-cert\") pod \"maas-controller-7765874568-dn64s\" (UID: \"017bf68e-1d4d-4145-b90b-ee6ad6d0100f\") " pod="opendatahub/maas-controller-7765874568-dn64s" Jul 21 00:57:00.554202 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:57:00.554152 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb49f\" (UniqueName: \"kubernetes.io/projected/017bf68e-1d4d-4145-b90b-ee6ad6d0100f-kube-api-access-hb49f\") pod \"maas-controller-7765874568-dn64s\" (UID: \"017bf68e-1d4d-4145-b90b-ee6ad6d0100f\") " pod="opendatahub/maas-controller-7765874568-dn64s" Jul 21 00:57:00.554402 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:57:00.554210 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/017bf68e-1d4d-4145-b90b-ee6ad6d0100f-webhook-cert\") pod \"maas-controller-7765874568-dn64s\" (UID: \"017bf68e-1d4d-4145-b90b-ee6ad6d0100f\") " pod="opendatahub/maas-controller-7765874568-dn64s" Jul 21 00:57:00.556902 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:57:00.556867 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/017bf68e-1d4d-4145-b90b-ee6ad6d0100f-webhook-cert\") pod \"maas-controller-7765874568-dn64s\" (UID: \"017bf68e-1d4d-4145-b90b-ee6ad6d0100f\") " pod="opendatahub/maas-controller-7765874568-dn64s" Jul 21 00:57:00.562597 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:57:00.562572 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb49f\" (UniqueName: \"kubernetes.io/projected/017bf68e-1d4d-4145-b90b-ee6ad6d0100f-kube-api-access-hb49f\") pod \"maas-controller-7765874568-dn64s\" (UID: \"017bf68e-1d4d-4145-b90b-ee6ad6d0100f\") " pod="opendatahub/maas-controller-7765874568-dn64s" Jul 21 00:57:00.729968 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:57:00.729883 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7765874568-dn64s" Jul 21 00:57:00.861901 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:57:00.861875 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7765874568-dn64s"] Jul 21 00:57:00.864555 ip-10-0-129-137 kubenswrapper[2580]: W0721 00:57:00.864515 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod017bf68e_1d4d_4145_b90b_ee6ad6d0100f.slice/crio-3c5f27d4fdbc2259130f77b16e920d1079357db09aa4a39cb61b3bc2555a38cc WatchSource:0}: Error finding container 3c5f27d4fdbc2259130f77b16e920d1079357db09aa4a39cb61b3bc2555a38cc: Status 404 returned error can't find the container with id 3c5f27d4fdbc2259130f77b16e920d1079357db09aa4a39cb61b3bc2555a38cc Jul 21 00:57:00.912162 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:57:00.912134 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5feba319-6017-4b67-bb5c-624a0027c50f" path="/var/lib/kubelet/pods/5feba319-6017-4b67-bb5c-624a0027c50f/volumes" Jul 21 00:57:01.595873 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:57:01.595837 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7765874568-dn64s" event={"ID":"017bf68e-1d4d-4145-b90b-ee6ad6d0100f","Type":"ContainerStarted","Data":"45c3097aabd1ff2a151f72063c2e7b70f05149324490abbf2e019ae1493d7a12"} Jul 21 00:57:01.596278 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:57:01.595898 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7765874568-dn64s" event={"ID":"017bf68e-1d4d-4145-b90b-ee6ad6d0100f","Type":"ContainerStarted","Data":"3c5f27d4fdbc2259130f77b16e920d1079357db09aa4a39cb61b3bc2555a38cc"} Jul 21 00:57:01.596278 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:57:01.596014 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7765874568-dn64s" Jul 21 00:57:01.615202 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:57:01.615017 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7765874568-dn64s" podStartSLOduration=1.171673354 podStartE2EDuration="1.615002276s" totalImagesPullingTime="443.328922ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:57:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:57:01.614844301 +0000 UTC m=+969.220953399" watchObservedRunningTime="2026-07-21 00:57:01.615002276 +0000 UTC m=+969.221111316" Jul 21 00:57:12.604045 ip-10-0-129-137 kubenswrapper[2580]: I0721 00:57:12.604011 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7765874568-dn64s" Jul 21 01:00:52.832086 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:00:52.832054 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-rb4ph_3ee1dbf0-b4ff-4871-9432-284eb25f97b1/console-operator/1.log" Jul 21 01:00:52.834611 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:00:52.834591 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-rb4ph_3ee1dbf0-b4ff-4871-9432-284eb25f97b1/console-operator/1.log" Jul 21 01:01:39.920567 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:01:39.920534 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mvc9h"] Jul 21 01:01:39.921100 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:01:39.920728 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mvc9h" podUID="328266ea-0fd8-46ae-a2d1-5653eaa17b4d" containerName="manager" containerID="cri-o://1dfa6365dcac917081cba504d04307d599dc23c4508517e4a9db85bcd22dcbd2" gracePeriod=10 Jul 21 01:01:43.209253 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:01:43.209203 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mvc9h" podUID="328266ea-0fd8-46ae-a2d1-5653eaa17b4d" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.40:8081/readyz\": dial tcp 10.132.0.40:8081: connect: connection refused" Jul 21 01:01:43.779560 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:01:43.779535 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mvc9h" Jul 21 01:01:43.920277 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:01:43.920144 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/328266ea-0fd8-46ae-a2d1-5653eaa17b4d-extensions-socket-volume\") pod \"328266ea-0fd8-46ae-a2d1-5653eaa17b4d\" (UID: \"328266ea-0fd8-46ae-a2d1-5653eaa17b4d\") " Jul 21 01:01:43.920277 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:01:43.920268 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tqtr4\" (UniqueName: \"kubernetes.io/projected/328266ea-0fd8-46ae-a2d1-5653eaa17b4d-kube-api-access-tqtr4\") pod \"328266ea-0fd8-46ae-a2d1-5653eaa17b4d\" (UID: \"328266ea-0fd8-46ae-a2d1-5653eaa17b4d\") " Jul 21 01:01:43.920623 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:01:43.920599 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/328266ea-0fd8-46ae-a2d1-5653eaa17b4d-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "328266ea-0fd8-46ae-a2d1-5653eaa17b4d" (UID: "328266ea-0fd8-46ae-a2d1-5653eaa17b4d"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 01:01:43.922374 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:01:43.922345 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/328266ea-0fd8-46ae-a2d1-5653eaa17b4d-kube-api-access-tqtr4" (OuterVolumeSpecName: "kube-api-access-tqtr4") pod "328266ea-0fd8-46ae-a2d1-5653eaa17b4d" (UID: "328266ea-0fd8-46ae-a2d1-5653eaa17b4d"). InnerVolumeSpecName "kube-api-access-tqtr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 01:01:44.021097 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:01:44.021060 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqtr4\" (UniqueName: \"kubernetes.io/projected/328266ea-0fd8-46ae-a2d1-5653eaa17b4d-kube-api-access-tqtr4\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 21 01:01:44.021097 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:01:44.021090 2580 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/328266ea-0fd8-46ae-a2d1-5653eaa17b4d-extensions-socket-volume\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 21 01:01:44.057145 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:01:44.057107 2580 generic.go:356] "Generic (PLEG): container finished" podID="328266ea-0fd8-46ae-a2d1-5653eaa17b4d" containerID="1dfa6365dcac917081cba504d04307d599dc23c4508517e4a9db85bcd22dcbd2" exitCode=0 Jul 21 01:01:44.057351 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:01:44.057153 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mvc9h" Jul 21 01:01:44.057351 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:01:44.057201 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mvc9h" event={"ID":"328266ea-0fd8-46ae-a2d1-5653eaa17b4d","Type":"ContainerDied","Data":"1dfa6365dcac917081cba504d04307d599dc23c4508517e4a9db85bcd22dcbd2"} Jul 21 01:01:44.057351 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:01:44.057285 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mvc9h" event={"ID":"328266ea-0fd8-46ae-a2d1-5653eaa17b4d","Type":"ContainerDied","Data":"9c6afaa4250df859e16f1302bbde6bcf6b35154616509953025be80e10a6f674"} Jul 21 01:01:44.057351 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:01:44.057321 2580 scope.go:117] "RemoveContainer" containerID="1dfa6365dcac917081cba504d04307d599dc23c4508517e4a9db85bcd22dcbd2" Jul 21 01:01:44.067695 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:01:44.067675 2580 scope.go:117] "RemoveContainer" containerID="1dfa6365dcac917081cba504d04307d599dc23c4508517e4a9db85bcd22dcbd2" Jul 21 01:01:44.067948 ip-10-0-129-137 kubenswrapper[2580]: E0721 01:01:44.067929 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dfa6365dcac917081cba504d04307d599dc23c4508517e4a9db85bcd22dcbd2\": container with ID starting with 1dfa6365dcac917081cba504d04307d599dc23c4508517e4a9db85bcd22dcbd2 not found: ID does not exist" containerID="1dfa6365dcac917081cba504d04307d599dc23c4508517e4a9db85bcd22dcbd2" Jul 21 01:01:44.068000 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:01:44.067957 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dfa6365dcac917081cba504d04307d599dc23c4508517e4a9db85bcd22dcbd2"} err="failed to get container status \"1dfa6365dcac917081cba504d04307d599dc23c4508517e4a9db85bcd22dcbd2\": rpc error: code = NotFound desc = could not find container \"1dfa6365dcac917081cba504d04307d599dc23c4508517e4a9db85bcd22dcbd2\": container with ID starting with 1dfa6365dcac917081cba504d04307d599dc23c4508517e4a9db85bcd22dcbd2 not found: ID does not exist" Jul 21 01:01:44.078607 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:01:44.078582 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mvc9h"] Jul 21 01:01:44.082208 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:01:44.082189 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mvc9h"] Jul 21 01:01:44.913374 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:01:44.913340 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="328266ea-0fd8-46ae-a2d1-5653eaa17b4d" path="/var/lib/kubelet/pods/328266ea-0fd8-46ae-a2d1-5653eaa17b4d/volumes" Jul 21 01:02:41.049902 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:02:41.049868 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jmp7l"] Jul 21 01:02:41.050417 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:02:41.050316 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="328266ea-0fd8-46ae-a2d1-5653eaa17b4d" containerName="manager" Jul 21 01:02:41.050417 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:02:41.050329 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="328266ea-0fd8-46ae-a2d1-5653eaa17b4d" containerName="manager" Jul 21 01:02:41.050417 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:02:41.050391 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="328266ea-0fd8-46ae-a2d1-5653eaa17b4d" containerName="manager" Jul 21 01:02:41.053725 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:02:41.053709 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jmp7l" Jul 21 01:02:41.056618 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:02:41.056597 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-72g52\"" Jul 21 01:02:41.065795 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:02:41.065773 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jmp7l"] Jul 21 01:02:41.124964 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:02:41.124934 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkmml\" (UniqueName: \"kubernetes.io/projected/90ffe3aa-c193-4a69-a632-e2d1cb0013f7-kube-api-access-wkmml\") pod \"kuadrant-operator-controller-manager-86878c769f-jmp7l\" (UID: \"90ffe3aa-c193-4a69-a632-e2d1cb0013f7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jmp7l" Jul 21 01:02:41.125117 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:02:41.125028 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/90ffe3aa-c193-4a69-a632-e2d1cb0013f7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-jmp7l\" (UID: \"90ffe3aa-c193-4a69-a632-e2d1cb0013f7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jmp7l" Jul 21 01:02:41.226556 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:02:41.226519 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkmml\" (UniqueName: \"kubernetes.io/projected/90ffe3aa-c193-4a69-a632-e2d1cb0013f7-kube-api-access-wkmml\") pod \"kuadrant-operator-controller-manager-86878c769f-jmp7l\" (UID: \"90ffe3aa-c193-4a69-a632-e2d1cb0013f7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jmp7l" Jul 21 01:02:41.226723 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:02:41.226592 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/90ffe3aa-c193-4a69-a632-e2d1cb0013f7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-jmp7l\" (UID: \"90ffe3aa-c193-4a69-a632-e2d1cb0013f7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jmp7l" Jul 21 01:02:41.226939 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:02:41.226923 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/90ffe3aa-c193-4a69-a632-e2d1cb0013f7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-jmp7l\" (UID: \"90ffe3aa-c193-4a69-a632-e2d1cb0013f7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jmp7l" Jul 21 01:02:41.240122 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:02:41.240101 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkmml\" (UniqueName: \"kubernetes.io/projected/90ffe3aa-c193-4a69-a632-e2d1cb0013f7-kube-api-access-wkmml\") pod \"kuadrant-operator-controller-manager-86878c769f-jmp7l\" (UID: \"90ffe3aa-c193-4a69-a632-e2d1cb0013f7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jmp7l" Jul 21 01:02:41.363725 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:02:41.363696 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jmp7l" Jul 21 01:02:41.512474 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:02:41.512449 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jmp7l"] Jul 21 01:02:41.514834 ip-10-0-129-137 kubenswrapper[2580]: W0721 01:02:41.514808 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90ffe3aa_c193_4a69_a632_e2d1cb0013f7.slice/crio-09addc8fb567a17107725057cfab212c6d3caa78f1f75b19b10365f414bdf618 WatchSource:0}: Error finding container 09addc8fb567a17107725057cfab212c6d3caa78f1f75b19b10365f414bdf618: Status 404 returned error can't find the container with id 09addc8fb567a17107725057cfab212c6d3caa78f1f75b19b10365f414bdf618 Jul 21 01:02:42.357154 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:02:42.357115 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jmp7l" event={"ID":"90ffe3aa-c193-4a69-a632-e2d1cb0013f7","Type":"ContainerStarted","Data":"b72bbe894e222f9b77e5f52feec83e8fe43b4c3cad6610193a4b6a550f1275d8"} Jul 21 01:02:42.357567 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:02:42.357181 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jmp7l" event={"ID":"90ffe3aa-c193-4a69-a632-e2d1cb0013f7","Type":"ContainerStarted","Data":"09addc8fb567a17107725057cfab212c6d3caa78f1f75b19b10365f414bdf618"} Jul 21 01:02:42.357567 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:02:42.357221 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jmp7l" Jul 21 01:02:42.397007 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:02:42.396913 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jmp7l" podStartSLOduration=1.396892061 podStartE2EDuration="1.396892061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 01:02:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:02:42.394991269 +0000 UTC m=+1310.001100310" watchObservedRunningTime="2026-07-21 01:02:42.396892061 +0000 UTC m=+1310.003001102" Jul 21 01:02:53.364810 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:02:53.364770 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jmp7l" Jul 21 01:05:52.865959 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:05:52.865926 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-rb4ph_3ee1dbf0-b4ff-4871-9432-284eb25f97b1/console-operator/1.log" Jul 21 01:05:52.868849 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:05:52.868828 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-rb4ph_3ee1dbf0-b4ff-4871-9432-284eb25f97b1/console-operator/1.log" Jul 21 01:10:52.901916 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:10:52.901885 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-rb4ph_3ee1dbf0-b4ff-4871-9432-284eb25f97b1/console-operator/1.log" Jul 21 01:10:52.906171 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:10:52.906144 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-rb4ph_3ee1dbf0-b4ff-4871-9432-284eb25f97b1/console-operator/1.log" Jul 21 01:13:53.556591 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:13:53.556544 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/odh-model-controller-7998c67dc4-zmq6v" podUID="115ede03-1e4e-432b-a874-579f0fa66f73" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.25:8081/readyz\": dial tcp 10.132.0.25:8081: connect: connection refused" Jul 21 01:13:53.874989 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:13:53.874955 2580 generic.go:356] "Generic (PLEG): container finished" podID="115ede03-1e4e-432b-a874-579f0fa66f73" containerID="6600331fd86fe758c151924d5e52eaa02b2bf93279cd620816988f6b7c5716ca" exitCode=1 Jul 21 01:13:53.875163 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:13:53.875027 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-zmq6v" event={"ID":"115ede03-1e4e-432b-a874-579f0fa66f73","Type":"ContainerDied","Data":"6600331fd86fe758c151924d5e52eaa02b2bf93279cd620816988f6b7c5716ca"} Jul 21 01:13:53.875354 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:13:53.875342 2580 scope.go:117] "RemoveContainer" containerID="6600331fd86fe758c151924d5e52eaa02b2bf93279cd620816988f6b7c5716ca" Jul 21 01:13:53.875657 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:13:53.875646 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 01:13:54.882017 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:13:54.881987 2580 generic.go:356] "Generic (PLEG): container finished" podID="017bf68e-1d4d-4145-b90b-ee6ad6d0100f" containerID="45c3097aabd1ff2a151f72063c2e7b70f05149324490abbf2e019ae1493d7a12" exitCode=1 Jul 21 01:13:54.882493 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:13:54.882056 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7765874568-dn64s" event={"ID":"017bf68e-1d4d-4145-b90b-ee6ad6d0100f","Type":"ContainerDied","Data":"45c3097aabd1ff2a151f72063c2e7b70f05149324490abbf2e019ae1493d7a12"} Jul 21 01:13:54.882493 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:13:54.882416 2580 scope.go:117] "RemoveContainer" containerID="45c3097aabd1ff2a151f72063c2e7b70f05149324490abbf2e019ae1493d7a12" Jul 21 01:13:54.884397 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:13:54.884376 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-zmq6v" event={"ID":"115ede03-1e4e-432b-a874-579f0fa66f73","Type":"ContainerStarted","Data":"fb43058da5c435211f5e3e88bbaabe892b092c9705250aaada335faf6b281c80"} Jul 21 01:13:54.884610 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:13:54.884591 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-zmq6v" Jul 21 01:13:55.427190 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:13:55.427165 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 21 01:13:55.891384 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:13:55.891345 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7765874568-dn64s" event={"ID":"017bf68e-1d4d-4145-b90b-ee6ad6d0100f","Type":"ContainerStarted","Data":"67e1e923fbb21695658b6483166a3b2c90d6827a3f112bcc928b9d7f663496c3"} Jul 21 01:13:55.891852 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:13:55.891529 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7765874568-dn64s" Jul 21 01:14:05.893391 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:14:05.893356 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-zmq6v" Jul 21 01:14:05.943136 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:14:05.943091 2580 generic.go:356] "Generic (PLEG): container finished" podID="017bf68e-1d4d-4145-b90b-ee6ad6d0100f" containerID="67e1e923fbb21695658b6483166a3b2c90d6827a3f112bcc928b9d7f663496c3" exitCode=1 Jul 21 01:14:05.943307 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:14:05.943173 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7765874568-dn64s" event={"ID":"017bf68e-1d4d-4145-b90b-ee6ad6d0100f","Type":"ContainerDied","Data":"67e1e923fbb21695658b6483166a3b2c90d6827a3f112bcc928b9d7f663496c3"} Jul 21 01:14:05.943307 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:14:05.943267 2580 scope.go:117] "RemoveContainer" containerID="45c3097aabd1ff2a151f72063c2e7b70f05149324490abbf2e019ae1493d7a12" Jul 21 01:14:05.943674 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:14:05.943652 2580 scope.go:117] "RemoveContainer" containerID="67e1e923fbb21695658b6483166a3b2c90d6827a3f112bcc928b9d7f663496c3" Jul 21 01:14:05.943918 ip-10-0-129-137 kubenswrapper[2580]: E0721 01:14:05.943893 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=maas-controller-7765874568-dn64s_opendatahub(017bf68e-1d4d-4145-b90b-ee6ad6d0100f)\"" pod="opendatahub/maas-controller-7765874568-dn64s" podUID="017bf68e-1d4d-4145-b90b-ee6ad6d0100f" Jul 21 01:14:18.912615 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:14:18.912581 2580 scope.go:117] "RemoveContainer" containerID="67e1e923fbb21695658b6483166a3b2c90d6827a3f112bcc928b9d7f663496c3" Jul 21 01:14:20.019887 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:14:20.019853 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7765874568-dn64s" event={"ID":"017bf68e-1d4d-4145-b90b-ee6ad6d0100f","Type":"ContainerStarted","Data":"648f6f3fd237446492612c109a4290c6f33b97cac135a490e93edfb7a7119ed6"} Jul 21 01:14:20.020325 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:14:20.020060 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7765874568-dn64s" Jul 21 01:14:31.026606 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:14:31.026571 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7765874568-dn64s" Jul 21 01:15:52.932566 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:15:52.932538 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-rb4ph_3ee1dbf0-b4ff-4871-9432-284eb25f97b1/console-operator/1.log" Jul 21 01:15:52.944977 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:15:52.944957 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-rb4ph_3ee1dbf0-b4ff-4871-9432-284eb25f97b1/console-operator/1.log" Jul 21 01:18:41.109506 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:18:41.109469 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-6bfe4a/test-model-6bfe4a-kserve-57f5785485-sts7s"] Jul 21 01:18:41.113418 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:18:41.113393 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6bfe4a/test-model-6bfe4a-kserve-57f5785485-sts7s" Jul 21 01:18:41.117506 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:18:41.117459 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-6bfe4a\"/\"test-model-6bfe4a-kserve-self-signed-certs\"" Jul 21 01:18:41.117506 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:18:41.117488 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-6bfe4a\"/\"openshift-service-ca.crt\"" Jul 21 01:18:41.117688 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:18:41.117514 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-6bfe4a\"/\"kube-root-ca.crt\"" Jul 21 01:18:41.117688 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:18:41.117463 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-6bfe4a\"/\"default-dockercfg-7mtbd\"" Jul 21 01:18:41.123430 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:18:41.123405 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-6bfe4a/test-model-6bfe4a-kserve-57f5785485-sts7s"] Jul 21 01:18:41.165748 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:18:41.165714 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5911503-d27e-4924-90f8-19253daedb1d-dshm\") pod \"test-model-6bfe4a-kserve-57f5785485-sts7s\" (UID: \"f5911503-d27e-4924-90f8-19253daedb1d\") " pod="ai-tenant-e2e-inf-b-6bfe4a/test-model-6bfe4a-kserve-57f5785485-sts7s" Jul 21 01:18:41.165748 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:18:41.165751 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5911503-d27e-4924-90f8-19253daedb1d-home\") pod \"test-model-6bfe4a-kserve-57f5785485-sts7s\" (UID: \"f5911503-d27e-4924-90f8-19253daedb1d\") " pod="ai-tenant-e2e-inf-b-6bfe4a/test-model-6bfe4a-kserve-57f5785485-sts7s" Jul 21 01:18:41.165956 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:18:41.165809 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5911503-d27e-4924-90f8-19253daedb1d-tls-certs\") pod \"test-model-6bfe4a-kserve-57f5785485-sts7s\" (UID: \"f5911503-d27e-4924-90f8-19253daedb1d\") " pod="ai-tenant-e2e-inf-b-6bfe4a/test-model-6bfe4a-kserve-57f5785485-sts7s" Jul 21 01:18:41.165956 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:18:41.165838 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5911503-d27e-4924-90f8-19253daedb1d-model-cache\") pod \"test-model-6bfe4a-kserve-57f5785485-sts7s\" (UID: \"f5911503-d27e-4924-90f8-19253daedb1d\") " pod="ai-tenant-e2e-inf-b-6bfe4a/test-model-6bfe4a-kserve-57f5785485-sts7s" Jul 21 01:18:41.165956 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:18:41.165858 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5911503-d27e-4924-90f8-19253daedb1d-tmp-dir\") pod \"test-model-6bfe4a-kserve-57f5785485-sts7s\" (UID: \"f5911503-d27e-4924-90f8-19253daedb1d\") " pod="ai-tenant-e2e-inf-b-6bfe4a/test-model-6bfe4a-kserve-57f5785485-sts7s" Jul 21 01:18:41.165956 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:18:41.165895 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf7qh\" (UniqueName: \"kubernetes.io/projected/f5911503-d27e-4924-90f8-19253daedb1d-kube-api-access-cf7qh\") pod \"test-model-6bfe4a-kserve-57f5785485-sts7s\" (UID: \"f5911503-d27e-4924-90f8-19253daedb1d\") " pod="ai-tenant-e2e-inf-b-6bfe4a/test-model-6bfe4a-kserve-57f5785485-sts7s" Jul 21 01:18:41.267019 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:18:41.266981 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5911503-d27e-4924-90f8-19253daedb1d-tls-certs\") pod \"test-model-6bfe4a-kserve-57f5785485-sts7s\" (UID: \"f5911503-d27e-4924-90f8-19253daedb1d\") " pod="ai-tenant-e2e-inf-b-6bfe4a/test-model-6bfe4a-kserve-57f5785485-sts7s" Jul 21 01:18:41.267200 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:18:41.267034 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5911503-d27e-4924-90f8-19253daedb1d-model-cache\") pod \"test-model-6bfe4a-kserve-57f5785485-sts7s\" (UID: \"f5911503-d27e-4924-90f8-19253daedb1d\") " pod="ai-tenant-e2e-inf-b-6bfe4a/test-model-6bfe4a-kserve-57f5785485-sts7s" Jul 21 01:18:41.267200 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:18:41.267058 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5911503-d27e-4924-90f8-19253daedb1d-tmp-dir\") pod \"test-model-6bfe4a-kserve-57f5785485-sts7s\" (UID: \"f5911503-d27e-4924-90f8-19253daedb1d\") " pod="ai-tenant-e2e-inf-b-6bfe4a/test-model-6bfe4a-kserve-57f5785485-sts7s" Jul 21 01:18:41.267306 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:18:41.267216 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf7qh\" (UniqueName: \"kubernetes.io/projected/f5911503-d27e-4924-90f8-19253daedb1d-kube-api-access-cf7qh\") pod \"test-model-6bfe4a-kserve-57f5785485-sts7s\" (UID: \"f5911503-d27e-4924-90f8-19253daedb1d\") " pod="ai-tenant-e2e-inf-b-6bfe4a/test-model-6bfe4a-kserve-57f5785485-sts7s" Jul 21 01:18:41.267407 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:18:41.267381 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5911503-d27e-4924-90f8-19253daedb1d-dshm\") pod \"test-model-6bfe4a-kserve-57f5785485-sts7s\" (UID: \"f5911503-d27e-4924-90f8-19253daedb1d\") " pod="ai-tenant-e2e-inf-b-6bfe4a/test-model-6bfe4a-kserve-57f5785485-sts7s" Jul 21 01:18:41.267561 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:18:41.267419 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5911503-d27e-4924-90f8-19253daedb1d-home\") pod \"test-model-6bfe4a-kserve-57f5785485-sts7s\" (UID: \"f5911503-d27e-4924-90f8-19253daedb1d\") " pod="ai-tenant-e2e-inf-b-6bfe4a/test-model-6bfe4a-kserve-57f5785485-sts7s" Jul 21 01:18:41.267561 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:18:41.267480 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5911503-d27e-4924-90f8-19253daedb1d-tmp-dir\") pod \"test-model-6bfe4a-kserve-57f5785485-sts7s\" (UID: \"f5911503-d27e-4924-90f8-19253daedb1d\") " pod="ai-tenant-e2e-inf-b-6bfe4a/test-model-6bfe4a-kserve-57f5785485-sts7s" Jul 21 01:18:41.267561 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:18:41.267514 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5911503-d27e-4924-90f8-19253daedb1d-model-cache\") pod \"test-model-6bfe4a-kserve-57f5785485-sts7s\" (UID: \"f5911503-d27e-4924-90f8-19253daedb1d\") " pod="ai-tenant-e2e-inf-b-6bfe4a/test-model-6bfe4a-kserve-57f5785485-sts7s" Jul 21 01:18:41.267813 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:18:41.267795 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5911503-d27e-4924-90f8-19253daedb1d-home\") pod \"test-model-6bfe4a-kserve-57f5785485-sts7s\" (UID: \"f5911503-d27e-4924-90f8-19253daedb1d\") " pod="ai-tenant-e2e-inf-b-6bfe4a/test-model-6bfe4a-kserve-57f5785485-sts7s" Jul 21 01:18:41.269707 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:18:41.269685 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5911503-d27e-4924-90f8-19253daedb1d-dshm\") pod \"test-model-6bfe4a-kserve-57f5785485-sts7s\" (UID: \"f5911503-d27e-4924-90f8-19253daedb1d\") " pod="ai-tenant-e2e-inf-b-6bfe4a/test-model-6bfe4a-kserve-57f5785485-sts7s" Jul 21 01:18:41.269819 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:18:41.269707 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5911503-d27e-4924-90f8-19253daedb1d-tls-certs\") pod \"test-model-6bfe4a-kserve-57f5785485-sts7s\" (UID: \"f5911503-d27e-4924-90f8-19253daedb1d\") " pod="ai-tenant-e2e-inf-b-6bfe4a/test-model-6bfe4a-kserve-57f5785485-sts7s" Jul 21 01:18:41.278032 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:18:41.277998 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf7qh\" (UniqueName: \"kubernetes.io/projected/f5911503-d27e-4924-90f8-19253daedb1d-kube-api-access-cf7qh\") pod \"test-model-6bfe4a-kserve-57f5785485-sts7s\" (UID: \"f5911503-d27e-4924-90f8-19253daedb1d\") " pod="ai-tenant-e2e-inf-b-6bfe4a/test-model-6bfe4a-kserve-57f5785485-sts7s" Jul 21 01:18:41.423928 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:18:41.423829 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6bfe4a/test-model-6bfe4a-kserve-57f5785485-sts7s" Jul 21 01:18:41.574839 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:18:41.574815 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-6bfe4a/test-model-6bfe4a-kserve-57f5785485-sts7s"] Jul 21 01:18:41.576545 ip-10-0-129-137 kubenswrapper[2580]: W0721 01:18:41.576518 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf5911503_d27e_4924_90f8_19253daedb1d.slice/crio-17ed12697dc679f81d82fe573f85dfeee96290143ec0d5f6bc6bc8af122d66fa WatchSource:0}: Error finding container 17ed12697dc679f81d82fe573f85dfeee96290143ec0d5f6bc6bc8af122d66fa: Status 404 returned error can't find the container with id 17ed12697dc679f81d82fe573f85dfeee96290143ec0d5f6bc6bc8af122d66fa Jul 21 01:18:42.385770 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:18:42.385735 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6bfe4a/test-model-6bfe4a-kserve-57f5785485-sts7s" event={"ID":"f5911503-d27e-4924-90f8-19253daedb1d","Type":"ContainerStarted","Data":"e82bc2461a8ab8292a226026587c197a915310908ebbb0280821bad1600d06b6"} Jul 21 01:18:42.385770 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:18:42.385777 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6bfe4a/test-model-6bfe4a-kserve-57f5785485-sts7s" event={"ID":"f5911503-d27e-4924-90f8-19253daedb1d","Type":"ContainerStarted","Data":"17ed12697dc679f81d82fe573f85dfeee96290143ec0d5f6bc6bc8af122d66fa"} Jul 21 01:18:42.406880 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:18:42.406826 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-6bfe4a/test-model-6bfe4a-kserve-57f5785485-sts7s" podStartSLOduration=1.406807706 podStartE2EDuration="1.406807706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 01:18:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:18:42.404991375 +0000 UTC m=+2270.011100416" watchObservedRunningTime="2026-07-21 01:18:42.406807706 +0000 UTC m=+2270.012916747" Jul 21 01:18:51.424844 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:18:51.424800 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-6bfe4a/test-model-6bfe4a-kserve-57f5785485-sts7s" Jul 21 01:18:51.425276 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:18:51.424888 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-6bfe4a/test-model-6bfe4a-kserve-57f5785485-sts7s" Jul 21 01:18:51.437838 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:18:51.437809 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-6bfe4a/test-model-6bfe4a-kserve-57f5785485-sts7s" Jul 21 01:18:52.456382 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:18:52.456349 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-6bfe4a/test-model-6bfe4a-kserve-57f5785485-sts7s" Jul 21 01:19:32.712010 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:19:32.711976 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-6bfe4a/test-model-6bfe4a-kserve-57f5785485-sts7s"] Jul 21 01:19:47.734708 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:19:47.734592 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-6bfe4a/test-model-6bfe4a-kserve-57f5785485-sts7s" podUID="f5911503-d27e-4924-90f8-19253daedb1d" containerName="main" containerID="cri-o://e82bc2461a8ab8292a226026587c197a915310908ebbb0280821bad1600d06b6" gracePeriod=45 Jul 21 01:19:47.983273 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:19:47.983242 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6bfe4a/test-model-6bfe4a-kserve-57f5785485-sts7s" Jul 21 01:19:48.112535 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:19:48.112502 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5911503-d27e-4924-90f8-19253daedb1d-tls-certs\") pod \"f5911503-d27e-4924-90f8-19253daedb1d\" (UID: \"f5911503-d27e-4924-90f8-19253daedb1d\") " Jul 21 01:19:48.112730 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:19:48.112552 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5911503-d27e-4924-90f8-19253daedb1d-dshm\") pod \"f5911503-d27e-4924-90f8-19253daedb1d\" (UID: \"f5911503-d27e-4924-90f8-19253daedb1d\") " Jul 21 01:19:48.112730 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:19:48.112582 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5911503-d27e-4924-90f8-19253daedb1d-tmp-dir\") pod \"f5911503-d27e-4924-90f8-19253daedb1d\" (UID: \"f5911503-d27e-4924-90f8-19253daedb1d\") " Jul 21 01:19:48.112730 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:19:48.112641 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cf7qh\" (UniqueName: \"kubernetes.io/projected/f5911503-d27e-4924-90f8-19253daedb1d-kube-api-access-cf7qh\") pod \"f5911503-d27e-4924-90f8-19253daedb1d\" (UID: \"f5911503-d27e-4924-90f8-19253daedb1d\") " Jul 21 01:19:48.112730 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:19:48.112669 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5911503-d27e-4924-90f8-19253daedb1d-home\") pod \"f5911503-d27e-4924-90f8-19253daedb1d\" (UID: \"f5911503-d27e-4924-90f8-19253daedb1d\") " Jul 21 01:19:48.112730 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:19:48.112722 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5911503-d27e-4924-90f8-19253daedb1d-model-cache\") pod \"f5911503-d27e-4924-90f8-19253daedb1d\" (UID: \"f5911503-d27e-4924-90f8-19253daedb1d\") " Jul 21 01:19:48.113017 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:19:48.112891 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5911503-d27e-4924-90f8-19253daedb1d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f5911503-d27e-4924-90f8-19253daedb1d" (UID: "f5911503-d27e-4924-90f8-19253daedb1d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 01:19:48.113017 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:19:48.112971 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5911503-d27e-4924-90f8-19253daedb1d-home" (OuterVolumeSpecName: "home") pod "f5911503-d27e-4924-90f8-19253daedb1d" (UID: "f5911503-d27e-4924-90f8-19253daedb1d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 01:19:48.113017 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:19:48.112999 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5911503-d27e-4924-90f8-19253daedb1d-model-cache" (OuterVolumeSpecName: "model-cache") pod "f5911503-d27e-4924-90f8-19253daedb1d" (UID: "f5911503-d27e-4924-90f8-19253daedb1d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 01:19:48.113168 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:19:48.113134 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5911503-d27e-4924-90f8-19253daedb1d-tmp-dir\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 21 01:19:48.113168 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:19:48.113151 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5911503-d27e-4924-90f8-19253daedb1d-home\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 21 01:19:48.113168 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:19:48.113160 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5911503-d27e-4924-90f8-19253daedb1d-model-cache\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 21 01:19:48.114888 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:19:48.114855 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5911503-d27e-4924-90f8-19253daedb1d-dshm" (OuterVolumeSpecName: "dshm") pod "f5911503-d27e-4924-90f8-19253daedb1d" (UID: "f5911503-d27e-4924-90f8-19253daedb1d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 01:19:48.114996 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:19:48.114906 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5911503-d27e-4924-90f8-19253daedb1d-kube-api-access-cf7qh" (OuterVolumeSpecName: "kube-api-access-cf7qh") pod "f5911503-d27e-4924-90f8-19253daedb1d" (UID: "f5911503-d27e-4924-90f8-19253daedb1d"). InnerVolumeSpecName "kube-api-access-cf7qh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 01:19:48.114996 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:19:48.114954 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5911503-d27e-4924-90f8-19253daedb1d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f5911503-d27e-4924-90f8-19253daedb1d" (UID: "f5911503-d27e-4924-90f8-19253daedb1d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 01:19:48.214686 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:19:48.214646 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cf7qh\" (UniqueName: \"kubernetes.io/projected/f5911503-d27e-4924-90f8-19253daedb1d-kube-api-access-cf7qh\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 21 01:19:48.214686 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:19:48.214681 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5911503-d27e-4924-90f8-19253daedb1d-tls-certs\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 21 01:19:48.214686 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:19:48.214690 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5911503-d27e-4924-90f8-19253daedb1d-dshm\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 21 01:19:48.739638 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:19:48.739604 2580 generic.go:356] "Generic (PLEG): container finished" podID="f5911503-d27e-4924-90f8-19253daedb1d" containerID="e82bc2461a8ab8292a226026587c197a915310908ebbb0280821bad1600d06b6" exitCode=0 Jul 21 01:19:48.740038 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:19:48.739650 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6bfe4a/test-model-6bfe4a-kserve-57f5785485-sts7s" Jul 21 01:19:48.740038 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:19:48.739685 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6bfe4a/test-model-6bfe4a-kserve-57f5785485-sts7s" event={"ID":"f5911503-d27e-4924-90f8-19253daedb1d","Type":"ContainerDied","Data":"e82bc2461a8ab8292a226026587c197a915310908ebbb0280821bad1600d06b6"} Jul 21 01:19:48.740038 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:19:48.739731 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6bfe4a/test-model-6bfe4a-kserve-57f5785485-sts7s" event={"ID":"f5911503-d27e-4924-90f8-19253daedb1d","Type":"ContainerDied","Data":"17ed12697dc679f81d82fe573f85dfeee96290143ec0d5f6bc6bc8af122d66fa"} Jul 21 01:19:48.740038 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:19:48.739749 2580 scope.go:117] "RemoveContainer" containerID="e82bc2461a8ab8292a226026587c197a915310908ebbb0280821bad1600d06b6" Jul 21 01:19:48.749595 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:19:48.749576 2580 scope.go:117] "RemoveContainer" containerID="e82bc2461a8ab8292a226026587c197a915310908ebbb0280821bad1600d06b6" Jul 21 01:19:48.749859 ip-10-0-129-137 kubenswrapper[2580]: E0721 01:19:48.749842 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e82bc2461a8ab8292a226026587c197a915310908ebbb0280821bad1600d06b6\": container with ID starting with e82bc2461a8ab8292a226026587c197a915310908ebbb0280821bad1600d06b6 not found: ID does not exist" containerID="e82bc2461a8ab8292a226026587c197a915310908ebbb0280821bad1600d06b6" Jul 21 01:19:48.749938 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:19:48.749869 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e82bc2461a8ab8292a226026587c197a915310908ebbb0280821bad1600d06b6"} err="failed to get container status \"e82bc2461a8ab8292a226026587c197a915310908ebbb0280821bad1600d06b6\": rpc error: code = NotFound desc = could not find container \"e82bc2461a8ab8292a226026587c197a915310908ebbb0280821bad1600d06b6\": container with ID starting with e82bc2461a8ab8292a226026587c197a915310908ebbb0280821bad1600d06b6 not found: ID does not exist" Jul 21 01:19:48.761806 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:19:48.761776 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-6bfe4a/test-model-6bfe4a-kserve-57f5785485-sts7s"] Jul 21 01:19:48.763978 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:19:48.763958 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-6bfe4a/test-model-6bfe4a-kserve-57f5785485-sts7s"] Jul 21 01:19:48.913678 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:19:48.913642 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5911503-d27e-4924-90f8-19253daedb1d" path="/var/lib/kubelet/pods/f5911503-d27e-4924-90f8-19253daedb1d/volumes" Jul 21 01:20:52.967371 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:20:52.967340 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-rb4ph_3ee1dbf0-b4ff-4871-9432-284eb25f97b1/console-operator/1.log" Jul 21 01:20:52.981803 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:20:52.981778 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-rb4ph_3ee1dbf0-b4ff-4871-9432-284eb25f97b1/console-operator/1.log" Jul 21 01:22:32.510835 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:22:32.510797 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-03c5cb/auth-test-model-03c5cb-kserve-5d4899d9d7-82rpx"] Jul 21 01:22:32.511580 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:22:32.511561 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5911503-d27e-4924-90f8-19253daedb1d" containerName="main" Jul 21 01:22:32.511580 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:22:32.511582 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5911503-d27e-4924-90f8-19253daedb1d" containerName="main" Jul 21 01:22:32.511694 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:22:32.511670 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5911503-d27e-4924-90f8-19253daedb1d" containerName="main" Jul 21 01:22:32.514822 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:22:32.514806 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-03c5cb/auth-test-model-03c5cb-kserve-5d4899d9d7-82rpx" Jul 21 01:22:32.517467 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:22:32.517438 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-03c5cb\"/\"openshift-service-ca.crt\"" Jul 21 01:22:32.518720 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:22:32.518697 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-03c5cb\"/\"auth-test-model-03c5cb-kserve-self-signed-certs\"" Jul 21 01:22:32.518720 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:22:32.518710 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-03c5cb\"/\"kube-root-ca.crt\"" Jul 21 01:22:32.518879 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:22:32.518744 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-03c5cb\"/\"default-dockercfg-24n4x\"" Jul 21 01:22:32.524204 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:22:32.524181 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-03c5cb/auth-test-model-03c5cb-kserve-5d4899d9d7-82rpx"] Jul 21 01:22:32.571896 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:22:32.571861 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmqnr\" (UniqueName: \"kubernetes.io/projected/9353f401-66d0-42f9-b7af-dfd5012f1e69-kube-api-access-bmqnr\") pod \"auth-test-model-03c5cb-kserve-5d4899d9d7-82rpx\" (UID: \"9353f401-66d0-42f9-b7af-dfd5012f1e69\") " pod="ai-tenant-e2e-shared-b-03c5cb/auth-test-model-03c5cb-kserve-5d4899d9d7-82rpx" Jul 21 01:22:32.572085 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:22:32.571910 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9353f401-66d0-42f9-b7af-dfd5012f1e69-dshm\") pod \"auth-test-model-03c5cb-kserve-5d4899d9d7-82rpx\" (UID: \"9353f401-66d0-42f9-b7af-dfd5012f1e69\") " pod="ai-tenant-e2e-shared-b-03c5cb/auth-test-model-03c5cb-kserve-5d4899d9d7-82rpx" Jul 21 01:22:32.572085 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:22:32.572031 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9353f401-66d0-42f9-b7af-dfd5012f1e69-tmp-dir\") pod \"auth-test-model-03c5cb-kserve-5d4899d9d7-82rpx\" (UID: \"9353f401-66d0-42f9-b7af-dfd5012f1e69\") " pod="ai-tenant-e2e-shared-b-03c5cb/auth-test-model-03c5cb-kserve-5d4899d9d7-82rpx" Jul 21 01:22:32.572176 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:22:32.572087 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9353f401-66d0-42f9-b7af-dfd5012f1e69-home\") pod \"auth-test-model-03c5cb-kserve-5d4899d9d7-82rpx\" (UID: \"9353f401-66d0-42f9-b7af-dfd5012f1e69\") " pod="ai-tenant-e2e-shared-b-03c5cb/auth-test-model-03c5cb-kserve-5d4899d9d7-82rpx" Jul 21 01:22:32.572176 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:22:32.572130 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9353f401-66d0-42f9-b7af-dfd5012f1e69-tls-certs\") pod \"auth-test-model-03c5cb-kserve-5d4899d9d7-82rpx\" (UID: \"9353f401-66d0-42f9-b7af-dfd5012f1e69\") " pod="ai-tenant-e2e-shared-b-03c5cb/auth-test-model-03c5cb-kserve-5d4899d9d7-82rpx" Jul 21 01:22:32.572176 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:22:32.572156 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9353f401-66d0-42f9-b7af-dfd5012f1e69-model-cache\") pod \"auth-test-model-03c5cb-kserve-5d4899d9d7-82rpx\" (UID: \"9353f401-66d0-42f9-b7af-dfd5012f1e69\") " pod="ai-tenant-e2e-shared-b-03c5cb/auth-test-model-03c5cb-kserve-5d4899d9d7-82rpx" Jul 21 01:22:32.672798 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:22:32.672762 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9353f401-66d0-42f9-b7af-dfd5012f1e69-tmp-dir\") pod \"auth-test-model-03c5cb-kserve-5d4899d9d7-82rpx\" (UID: \"9353f401-66d0-42f9-b7af-dfd5012f1e69\") " pod="ai-tenant-e2e-shared-b-03c5cb/auth-test-model-03c5cb-kserve-5d4899d9d7-82rpx" Jul 21 01:22:32.673016 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:22:32.672835 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9353f401-66d0-42f9-b7af-dfd5012f1e69-home\") pod \"auth-test-model-03c5cb-kserve-5d4899d9d7-82rpx\" (UID: \"9353f401-66d0-42f9-b7af-dfd5012f1e69\") " pod="ai-tenant-e2e-shared-b-03c5cb/auth-test-model-03c5cb-kserve-5d4899d9d7-82rpx" Jul 21 01:22:32.673016 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:22:32.672860 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9353f401-66d0-42f9-b7af-dfd5012f1e69-tls-certs\") pod \"auth-test-model-03c5cb-kserve-5d4899d9d7-82rpx\" (UID: \"9353f401-66d0-42f9-b7af-dfd5012f1e69\") " pod="ai-tenant-e2e-shared-b-03c5cb/auth-test-model-03c5cb-kserve-5d4899d9d7-82rpx" Jul 21 01:22:32.673016 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:22:32.672891 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9353f401-66d0-42f9-b7af-dfd5012f1e69-model-cache\") pod \"auth-test-model-03c5cb-kserve-5d4899d9d7-82rpx\" (UID: \"9353f401-66d0-42f9-b7af-dfd5012f1e69\") " pod="ai-tenant-e2e-shared-b-03c5cb/auth-test-model-03c5cb-kserve-5d4899d9d7-82rpx" Jul 21 01:22:32.673016 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:22:32.672934 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmqnr\" (UniqueName: \"kubernetes.io/projected/9353f401-66d0-42f9-b7af-dfd5012f1e69-kube-api-access-bmqnr\") pod \"auth-test-model-03c5cb-kserve-5d4899d9d7-82rpx\" (UID: \"9353f401-66d0-42f9-b7af-dfd5012f1e69\") " pod="ai-tenant-e2e-shared-b-03c5cb/auth-test-model-03c5cb-kserve-5d4899d9d7-82rpx" Jul 21 01:22:32.673016 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:22:32.672965 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9353f401-66d0-42f9-b7af-dfd5012f1e69-dshm\") pod \"auth-test-model-03c5cb-kserve-5d4899d9d7-82rpx\" (UID: \"9353f401-66d0-42f9-b7af-dfd5012f1e69\") " pod="ai-tenant-e2e-shared-b-03c5cb/auth-test-model-03c5cb-kserve-5d4899d9d7-82rpx" Jul 21 01:22:32.673313 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:22:32.673287 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9353f401-66d0-42f9-b7af-dfd5012f1e69-tmp-dir\") pod \"auth-test-model-03c5cb-kserve-5d4899d9d7-82rpx\" (UID: \"9353f401-66d0-42f9-b7af-dfd5012f1e69\") " pod="ai-tenant-e2e-shared-b-03c5cb/auth-test-model-03c5cb-kserve-5d4899d9d7-82rpx" Jul 21 01:22:32.673394 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:22:32.673332 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9353f401-66d0-42f9-b7af-dfd5012f1e69-model-cache\") pod \"auth-test-model-03c5cb-kserve-5d4899d9d7-82rpx\" (UID: \"9353f401-66d0-42f9-b7af-dfd5012f1e69\") " pod="ai-tenant-e2e-shared-b-03c5cb/auth-test-model-03c5cb-kserve-5d4899d9d7-82rpx" Jul 21 01:22:32.673394 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:22:32.673353 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9353f401-66d0-42f9-b7af-dfd5012f1e69-home\") pod \"auth-test-model-03c5cb-kserve-5d4899d9d7-82rpx\" (UID: \"9353f401-66d0-42f9-b7af-dfd5012f1e69\") " pod="ai-tenant-e2e-shared-b-03c5cb/auth-test-model-03c5cb-kserve-5d4899d9d7-82rpx" Jul 21 01:22:32.675433 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:22:32.675397 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9353f401-66d0-42f9-b7af-dfd5012f1e69-dshm\") pod \"auth-test-model-03c5cb-kserve-5d4899d9d7-82rpx\" (UID: \"9353f401-66d0-42f9-b7af-dfd5012f1e69\") " pod="ai-tenant-e2e-shared-b-03c5cb/auth-test-model-03c5cb-kserve-5d4899d9d7-82rpx" Jul 21 01:22:32.675568 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:22:32.675547 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9353f401-66d0-42f9-b7af-dfd5012f1e69-tls-certs\") pod \"auth-test-model-03c5cb-kserve-5d4899d9d7-82rpx\" (UID: \"9353f401-66d0-42f9-b7af-dfd5012f1e69\") " pod="ai-tenant-e2e-shared-b-03c5cb/auth-test-model-03c5cb-kserve-5d4899d9d7-82rpx" Jul 21 01:22:32.688201 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:22:32.688170 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmqnr\" (UniqueName: \"kubernetes.io/projected/9353f401-66d0-42f9-b7af-dfd5012f1e69-kube-api-access-bmqnr\") pod \"auth-test-model-03c5cb-kserve-5d4899d9d7-82rpx\" (UID: \"9353f401-66d0-42f9-b7af-dfd5012f1e69\") " pod="ai-tenant-e2e-shared-b-03c5cb/auth-test-model-03c5cb-kserve-5d4899d9d7-82rpx" Jul 21 01:22:32.824421 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:22:32.824380 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-03c5cb/auth-test-model-03c5cb-kserve-5d4899d9d7-82rpx" Jul 21 01:22:32.966561 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:22:32.966536 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-03c5cb/auth-test-model-03c5cb-kserve-5d4899d9d7-82rpx"] Jul 21 01:22:32.968257 ip-10-0-129-137 kubenswrapper[2580]: W0721 01:22:32.968217 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9353f401_66d0_42f9_b7af_dfd5012f1e69.slice/crio-bea517bd103b486a5b323bb40cbe1b57ee840e854d17d71f64e3bc81e13e829a WatchSource:0}: Error finding container bea517bd103b486a5b323bb40cbe1b57ee840e854d17d71f64e3bc81e13e829a: Status 404 returned error can't find the container with id bea517bd103b486a5b323bb40cbe1b57ee840e854d17d71f64e3bc81e13e829a Jul 21 01:22:33.601349 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:22:33.601307 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-03c5cb/auth-test-model-03c5cb-kserve-5d4899d9d7-82rpx" event={"ID":"9353f401-66d0-42f9-b7af-dfd5012f1e69","Type":"ContainerStarted","Data":"4980c8725cd7126f6b14916f46444b5320c3b1ae1f8d33f368a36961cf5a6060"} Jul 21 01:22:33.601349 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:22:33.601354 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-03c5cb/auth-test-model-03c5cb-kserve-5d4899d9d7-82rpx" event={"ID":"9353f401-66d0-42f9-b7af-dfd5012f1e69","Type":"ContainerStarted","Data":"bea517bd103b486a5b323bb40cbe1b57ee840e854d17d71f64e3bc81e13e829a"} Jul 21 01:22:33.620639 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:22:33.620577 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-03c5cb/auth-test-model-03c5cb-kserve-5d4899d9d7-82rpx" podStartSLOduration=1.6205611279999999 podStartE2EDuration="1.620561128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 01:22:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:22:33.61889665 +0000 UTC m=+2501.225005690" watchObservedRunningTime="2026-07-21 01:22:33.620561128 +0000 UTC m=+2501.226670168" Jul 21 01:22:42.824921 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:22:42.824884 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-03c5cb/auth-test-model-03c5cb-kserve-5d4899d9d7-82rpx" Jul 21 01:22:42.825577 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:22:42.824935 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-03c5cb/auth-test-model-03c5cb-kserve-5d4899d9d7-82rpx" Jul 21 01:22:42.837389 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:22:42.837364 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-03c5cb/auth-test-model-03c5cb-kserve-5d4899d9d7-82rpx" Jul 21 01:22:43.665152 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:22:43.665115 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-03c5cb/auth-test-model-03c5cb-kserve-5d4899d9d7-82rpx" Jul 21 01:24:41.709540 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:24:41.709507 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-03c5cb/auth-test-model-03c5cb-kserve-5d4899d9d7-82rpx"] Jul 21 01:24:56.730163 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:24:56.730104 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-03c5cb/auth-test-model-03c5cb-kserve-5d4899d9d7-82rpx" podUID="9353f401-66d0-42f9-b7af-dfd5012f1e69" containerName="main" containerID="cri-o://4980c8725cd7126f6b14916f46444b5320c3b1ae1f8d33f368a36961cf5a6060" gracePeriod=45 Jul 21 01:24:56.980704 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:24:56.980637 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-03c5cb/auth-test-model-03c5cb-kserve-5d4899d9d7-82rpx" Jul 21 01:24:57.147832 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:24:57.147795 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9353f401-66d0-42f9-b7af-dfd5012f1e69-home\") pod \"9353f401-66d0-42f9-b7af-dfd5012f1e69\" (UID: \"9353f401-66d0-42f9-b7af-dfd5012f1e69\") " Jul 21 01:24:57.148043 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:24:57.147863 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9353f401-66d0-42f9-b7af-dfd5012f1e69-tls-certs\") pod \"9353f401-66d0-42f9-b7af-dfd5012f1e69\" (UID: \"9353f401-66d0-42f9-b7af-dfd5012f1e69\") " Jul 21 01:24:57.148043 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:24:57.147895 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9353f401-66d0-42f9-b7af-dfd5012f1e69-model-cache\") pod \"9353f401-66d0-42f9-b7af-dfd5012f1e69\" (UID: \"9353f401-66d0-42f9-b7af-dfd5012f1e69\") " Jul 21 01:24:57.148043 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:24:57.147938 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bmqnr\" (UniqueName: \"kubernetes.io/projected/9353f401-66d0-42f9-b7af-dfd5012f1e69-kube-api-access-bmqnr\") pod \"9353f401-66d0-42f9-b7af-dfd5012f1e69\" (UID: \"9353f401-66d0-42f9-b7af-dfd5012f1e69\") " Jul 21 01:24:57.148043 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:24:57.147969 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9353f401-66d0-42f9-b7af-dfd5012f1e69-dshm\") pod \"9353f401-66d0-42f9-b7af-dfd5012f1e69\" (UID: \"9353f401-66d0-42f9-b7af-dfd5012f1e69\") " Jul 21 01:24:57.148043 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:24:57.148013 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9353f401-66d0-42f9-b7af-dfd5012f1e69-tmp-dir\") pod \"9353f401-66d0-42f9-b7af-dfd5012f1e69\" (UID: \"9353f401-66d0-42f9-b7af-dfd5012f1e69\") " Jul 21 01:24:57.148350 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:24:57.148137 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9353f401-66d0-42f9-b7af-dfd5012f1e69-home" (OuterVolumeSpecName: "home") pod "9353f401-66d0-42f9-b7af-dfd5012f1e69" (UID: "9353f401-66d0-42f9-b7af-dfd5012f1e69"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 01:24:57.148410 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:24:57.148379 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9353f401-66d0-42f9-b7af-dfd5012f1e69-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9353f401-66d0-42f9-b7af-dfd5012f1e69" (UID: "9353f401-66d0-42f9-b7af-dfd5012f1e69"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 01:24:57.148468 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:24:57.148456 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9353f401-66d0-42f9-b7af-dfd5012f1e69-tmp-dir\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 21 01:24:57.148532 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:24:57.148473 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9353f401-66d0-42f9-b7af-dfd5012f1e69-home\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 21 01:24:57.148818 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:24:57.148775 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9353f401-66d0-42f9-b7af-dfd5012f1e69-model-cache" (OuterVolumeSpecName: "model-cache") pod "9353f401-66d0-42f9-b7af-dfd5012f1e69" (UID: "9353f401-66d0-42f9-b7af-dfd5012f1e69"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 01:24:57.150814 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:24:57.150790 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9353f401-66d0-42f9-b7af-dfd5012f1e69-kube-api-access-bmqnr" (OuterVolumeSpecName: "kube-api-access-bmqnr") pod "9353f401-66d0-42f9-b7af-dfd5012f1e69" (UID: "9353f401-66d0-42f9-b7af-dfd5012f1e69"). InnerVolumeSpecName "kube-api-access-bmqnr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 01:24:57.150922 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:24:57.150821 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9353f401-66d0-42f9-b7af-dfd5012f1e69-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9353f401-66d0-42f9-b7af-dfd5012f1e69" (UID: "9353f401-66d0-42f9-b7af-dfd5012f1e69"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 01:24:57.150922 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:24:57.150905 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9353f401-66d0-42f9-b7af-dfd5012f1e69-dshm" (OuterVolumeSpecName: "dshm") pod "9353f401-66d0-42f9-b7af-dfd5012f1e69" (UID: "9353f401-66d0-42f9-b7af-dfd5012f1e69"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 01:24:57.249848 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:24:57.249751 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9353f401-66d0-42f9-b7af-dfd5012f1e69-tls-certs\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 21 01:24:57.249848 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:24:57.249786 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9353f401-66d0-42f9-b7af-dfd5012f1e69-model-cache\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 21 01:24:57.249848 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:24:57.249798 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmqnr\" (UniqueName: \"kubernetes.io/projected/9353f401-66d0-42f9-b7af-dfd5012f1e69-kube-api-access-bmqnr\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 21 01:24:57.249848 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:24:57.249811 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9353f401-66d0-42f9-b7af-dfd5012f1e69-dshm\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 21 01:24:57.345053 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:24:57.345021 2580 generic.go:356] "Generic (PLEG): container finished" podID="9353f401-66d0-42f9-b7af-dfd5012f1e69" containerID="4980c8725cd7126f6b14916f46444b5320c3b1ae1f8d33f368a36961cf5a6060" exitCode=0 Jul 21 01:24:57.345223 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:24:57.345067 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-03c5cb/auth-test-model-03c5cb-kserve-5d4899d9d7-82rpx" Jul 21 01:24:57.345223 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:24:57.345090 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-03c5cb/auth-test-model-03c5cb-kserve-5d4899d9d7-82rpx" event={"ID":"9353f401-66d0-42f9-b7af-dfd5012f1e69","Type":"ContainerDied","Data":"4980c8725cd7126f6b14916f46444b5320c3b1ae1f8d33f368a36961cf5a6060"} Jul 21 01:24:57.345223 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:24:57.345122 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-03c5cb/auth-test-model-03c5cb-kserve-5d4899d9d7-82rpx" event={"ID":"9353f401-66d0-42f9-b7af-dfd5012f1e69","Type":"ContainerDied","Data":"bea517bd103b486a5b323bb40cbe1b57ee840e854d17d71f64e3bc81e13e829a"} Jul 21 01:24:57.345223 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:24:57.345139 2580 scope.go:117] "RemoveContainer" containerID="4980c8725cd7126f6b14916f46444b5320c3b1ae1f8d33f368a36961cf5a6060" Jul 21 01:24:57.355956 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:24:57.355937 2580 scope.go:117] "RemoveContainer" containerID="4980c8725cd7126f6b14916f46444b5320c3b1ae1f8d33f368a36961cf5a6060" Jul 21 01:24:57.356213 ip-10-0-129-137 kubenswrapper[2580]: E0721 01:24:57.356181 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4980c8725cd7126f6b14916f46444b5320c3b1ae1f8d33f368a36961cf5a6060\": container with ID starting with 4980c8725cd7126f6b14916f46444b5320c3b1ae1f8d33f368a36961cf5a6060 not found: ID does not exist" containerID="4980c8725cd7126f6b14916f46444b5320c3b1ae1f8d33f368a36961cf5a6060" Jul 21 01:24:57.356309 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:24:57.356224 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4980c8725cd7126f6b14916f46444b5320c3b1ae1f8d33f368a36961cf5a6060"} err="failed to get container status \"4980c8725cd7126f6b14916f46444b5320c3b1ae1f8d33f368a36961cf5a6060\": rpc error: code = NotFound desc = could not find container \"4980c8725cd7126f6b14916f46444b5320c3b1ae1f8d33f368a36961cf5a6060\": container with ID starting with 4980c8725cd7126f6b14916f46444b5320c3b1ae1f8d33f368a36961cf5a6060 not found: ID does not exist" Jul 21 01:24:57.367896 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:24:57.367871 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-03c5cb/auth-test-model-03c5cb-kserve-5d4899d9d7-82rpx"] Jul 21 01:24:57.373478 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:24:57.373452 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-03c5cb/auth-test-model-03c5cb-kserve-5d4899d9d7-82rpx"] Jul 21 01:24:58.912490 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:24:58.912455 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9353f401-66d0-42f9-b7af-dfd5012f1e69" path="/var/lib/kubelet/pods/9353f401-66d0-42f9-b7af-dfd5012f1e69/volumes" Jul 21 01:25:53.000454 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:25:53.000352 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-rb4ph_3ee1dbf0-b4ff-4871-9432-284eb25f97b1/console-operator/1.log" Jul 21 01:25:53.013579 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:25:53.013555 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-rb4ph_3ee1dbf0-b4ff-4871-9432-284eb25f97b1/console-operator/1.log" Jul 21 01:27:50.370769 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:27:50.370737 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7496-140655670881456"] Jul 21 01:27:50.371305 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:27:50.371173 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9353f401-66d0-42f9-b7af-dfd5012f1e69" containerName="main" Jul 21 01:27:50.371305 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:27:50.371187 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9353f401-66d0-42f9-b7af-dfd5012f1e69" containerName="main" Jul 21 01:27:50.371305 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:27:50.371276 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9353f401-66d0-42f9-b7af-dfd5012f1e69" containerName="main" Jul 21 01:27:50.374121 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:27:50.374104 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7496-140655670881456" Jul 21 01:27:50.376572 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:27:50.376547 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-h46kn\"" Jul 21 01:27:50.379785 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:27:50.379764 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7496-140655670881456"] Jul 21 01:27:50.451603 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:27:50.451566 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qgzv\" (UniqueName: \"kubernetes.io/projected/77afdb8c-fcdb-41f3-ae5f-41063d58f032-kube-api-access-8qgzv\") pod \"test-curl-7496-140655670881456\" (UID: \"77afdb8c-fcdb-41f3-ae5f-41063d58f032\") " pod="opendatahub/test-curl-7496-140655670881456" Jul 21 01:27:50.552980 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:27:50.552945 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qgzv\" (UniqueName: \"kubernetes.io/projected/77afdb8c-fcdb-41f3-ae5f-41063d58f032-kube-api-access-8qgzv\") pod \"test-curl-7496-140655670881456\" (UID: \"77afdb8c-fcdb-41f3-ae5f-41063d58f032\") " pod="opendatahub/test-curl-7496-140655670881456" Jul 21 01:27:50.561008 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:27:50.560980 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qgzv\" (UniqueName: \"kubernetes.io/projected/77afdb8c-fcdb-41f3-ae5f-41063d58f032-kube-api-access-8qgzv\") pod \"test-curl-7496-140655670881456\" (UID: \"77afdb8c-fcdb-41f3-ae5f-41063d58f032\") " pod="opendatahub/test-curl-7496-140655670881456" Jul 21 01:27:50.685476 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:27:50.685398 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7496-140655670881456" Jul 21 01:27:50.818710 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:27:50.818684 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7496-140655670881456"] Jul 21 01:27:50.820599 ip-10-0-129-137 kubenswrapper[2580]: W0721 01:27:50.820557 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod77afdb8c_fcdb_41f3_ae5f_41063d58f032.slice/crio-a0b21d8866e74c449a790d8a83cee1db09fb5bf0c25636c77b636537af8278f0 WatchSource:0}: Error finding container a0b21d8866e74c449a790d8a83cee1db09fb5bf0c25636c77b636537af8278f0: Status 404 returned error can't find the container with id a0b21d8866e74c449a790d8a83cee1db09fb5bf0c25636c77b636537af8278f0 Jul 21 01:27:50.821779 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:27:50.821761 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 01:27:51.241942 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:27:51.241904 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7496-140655670881456" event={"ID":"77afdb8c-fcdb-41f3-ae5f-41063d58f032","Type":"ContainerStarted","Data":"a0b21d8866e74c449a790d8a83cee1db09fb5bf0c25636c77b636537af8278f0"} Jul 21 01:27:52.248640 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:27:52.248608 2580 generic.go:356] "Generic (PLEG): container finished" podID="77afdb8c-fcdb-41f3-ae5f-41063d58f032" containerID="3752f07ae4483ee417e8f6dd0e88b7331d16fb50f09fffe65f7325965e1cfe02" exitCode=0 Jul 21 01:27:52.249122 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:27:52.248669 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7496-140655670881456" event={"ID":"77afdb8c-fcdb-41f3-ae5f-41063d58f032","Type":"ContainerDied","Data":"3752f07ae4483ee417e8f6dd0e88b7331d16fb50f09fffe65f7325965e1cfe02"} Jul 21 01:27:53.398613 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:27:53.398584 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7496-140655670881456" Jul 21 01:27:53.435202 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:27:53.435170 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7496-140655670881456_77afdb8c-fcdb-41f3-ae5f-41063d58f032/test-curl-7496-140655670881456/0.log" Jul 21 01:27:53.460858 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:27:53.460823 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7496-140655670881456"] Jul 21 01:27:53.465702 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:27:53.465671 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7496-140655670881456"] Jul 21 01:27:53.477762 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:27:53.477736 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8qgzv\" (UniqueName: \"kubernetes.io/projected/77afdb8c-fcdb-41f3-ae5f-41063d58f032-kube-api-access-8qgzv\") pod \"77afdb8c-fcdb-41f3-ae5f-41063d58f032\" (UID: \"77afdb8c-fcdb-41f3-ae5f-41063d58f032\") " Jul 21 01:27:53.479789 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:27:53.479763 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77afdb8c-fcdb-41f3-ae5f-41063d58f032-kube-api-access-8qgzv" (OuterVolumeSpecName: "kube-api-access-8qgzv") pod "77afdb8c-fcdb-41f3-ae5f-41063d58f032" (UID: "77afdb8c-fcdb-41f3-ae5f-41063d58f032"). InnerVolumeSpecName "kube-api-access-8qgzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 01:27:53.578831 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:27:53.578798 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qgzv\" (UniqueName: \"kubernetes.io/projected/77afdb8c-fcdb-41f3-ae5f-41063d58f032-kube-api-access-8qgzv\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 21 01:27:53.590129 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:27:53.590095 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7496-140655670833456"] Jul 21 01:27:53.590603 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:27:53.590584 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77afdb8c-fcdb-41f3-ae5f-41063d58f032" containerName="test-curl-7496-140655670881456" Jul 21 01:27:53.590686 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:27:53.590605 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="77afdb8c-fcdb-41f3-ae5f-41063d58f032" containerName="test-curl-7496-140655670881456" Jul 21 01:27:53.590740 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:27:53.590695 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="77afdb8c-fcdb-41f3-ae5f-41063d58f032" containerName="test-curl-7496-140655670881456" Jul 21 01:27:53.593724 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:27:53.593706 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7496-140655670833456" Jul 21 01:27:53.607065 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:27:53.607039 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7496-140655670833456"] Jul 21 01:27:53.679815 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:27:53.679777 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtzcb\" (UniqueName: \"kubernetes.io/projected/bc214094-5e3b-4c52-9e78-ec8596dfaa68-kube-api-access-qtzcb\") pod \"test-curl-7496-140655670833456\" (UID: \"bc214094-5e3b-4c52-9e78-ec8596dfaa68\") " pod="opendatahub/test-curl-7496-140655670833456" Jul 21 01:27:53.781030 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:27:53.780993 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtzcb\" (UniqueName: \"kubernetes.io/projected/bc214094-5e3b-4c52-9e78-ec8596dfaa68-kube-api-access-qtzcb\") pod \"test-curl-7496-140655670833456\" (UID: \"bc214094-5e3b-4c52-9e78-ec8596dfaa68\") " pod="opendatahub/test-curl-7496-140655670833456" Jul 21 01:27:53.789776 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:27:53.789748 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtzcb\" (UniqueName: \"kubernetes.io/projected/bc214094-5e3b-4c52-9e78-ec8596dfaa68-kube-api-access-qtzcb\") pod \"test-curl-7496-140655670833456\" (UID: \"bc214094-5e3b-4c52-9e78-ec8596dfaa68\") " pod="opendatahub/test-curl-7496-140655670833456" Jul 21 01:27:53.913841 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:27:53.913753 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7496-140655670833456" Jul 21 01:27:54.048447 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:27:54.048416 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7496-140655670833456"] Jul 21 01:27:54.050175 ip-10-0-129-137 kubenswrapper[2580]: W0721 01:27:54.050135 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbc214094_5e3b_4c52_9e78_ec8596dfaa68.slice/crio-08b8bead5fa77ab0392aad6e880e2a03cc1600f0ab9899055df1d9dd2b43dd1c WatchSource:0}: Error finding container 08b8bead5fa77ab0392aad6e880e2a03cc1600f0ab9899055df1d9dd2b43dd1c: Status 404 returned error can't find the container with id 08b8bead5fa77ab0392aad6e880e2a03cc1600f0ab9899055df1d9dd2b43dd1c Jul 21 01:27:54.260190 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:27:54.260100 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7496-140655670833456" event={"ID":"bc214094-5e3b-4c52-9e78-ec8596dfaa68","Type":"ContainerStarted","Data":"08b8bead5fa77ab0392aad6e880e2a03cc1600f0ab9899055df1d9dd2b43dd1c"} Jul 21 01:27:54.261247 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:27:54.261212 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0b21d8866e74c449a790d8a83cee1db09fb5bf0c25636c77b636537af8278f0" Jul 21 01:27:54.261439 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:27:54.261254 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7496-140655670881456" Jul 21 01:27:54.272153 ip-10-0-129-137 kubenswrapper[2580]: E0721 01:27:54.272128 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7496-140655670881456\" is forbidden: User \"system:node:ip-10-0-129-137.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-137.ec2.internal' and this object" podUID="77afdb8c-fcdb-41f3-ae5f-41063d58f032" pod="opendatahub/test-curl-7496-140655670881456" Jul 21 01:27:54.913527 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:27:54.913492 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77afdb8c-fcdb-41f3-ae5f-41063d58f032" path="/var/lib/kubelet/pods/77afdb8c-fcdb-41f3-ae5f-41063d58f032/volumes" Jul 21 01:27:55.268119 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:27:55.268038 2580 generic.go:356] "Generic (PLEG): container finished" podID="bc214094-5e3b-4c52-9e78-ec8596dfaa68" containerID="78ed58b9ba253598beca30fba716b9816a289707b014ab371f6e2b3f1358a5d0" exitCode=0 Jul 21 01:27:55.268273 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:27:55.268131 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7496-140655670833456" event={"ID":"bc214094-5e3b-4c52-9e78-ec8596dfaa68","Type":"ContainerDied","Data":"78ed58b9ba253598beca30fba716b9816a289707b014ab371f6e2b3f1358a5d0"} Jul 21 01:27:56.412810 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:27:56.412787 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7496-140655670833456" Jul 21 01:27:56.432017 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:27:56.431989 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7496-140655670833456_bc214094-5e3b-4c52-9e78-ec8596dfaa68/test-curl-7496-140655670833456/0.log" Jul 21 01:27:56.459934 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:27:56.459907 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7496-140655670833456"] Jul 21 01:27:56.463556 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:27:56.463533 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7496-140655670833456"] Jul 21 01:27:56.502968 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:27:56.502942 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qtzcb\" (UniqueName: \"kubernetes.io/projected/bc214094-5e3b-4c52-9e78-ec8596dfaa68-kube-api-access-qtzcb\") pod \"bc214094-5e3b-4c52-9e78-ec8596dfaa68\" (UID: \"bc214094-5e3b-4c52-9e78-ec8596dfaa68\") " Jul 21 01:27:56.504970 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:27:56.504944 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc214094-5e3b-4c52-9e78-ec8596dfaa68-kube-api-access-qtzcb" (OuterVolumeSpecName: "kube-api-access-qtzcb") pod "bc214094-5e3b-4c52-9e78-ec8596dfaa68" (UID: "bc214094-5e3b-4c52-9e78-ec8596dfaa68"). InnerVolumeSpecName "kube-api-access-qtzcb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 01:27:56.604102 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:27:56.604065 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtzcb\" (UniqueName: \"kubernetes.io/projected/bc214094-5e3b-4c52-9e78-ec8596dfaa68-kube-api-access-qtzcb\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 21 01:27:56.625630 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:27:56.625602 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7496-140655676106624"] Jul 21 01:27:56.626031 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:27:56.626019 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc214094-5e3b-4c52-9e78-ec8596dfaa68" containerName="test-curl-7496-140655670833456" Jul 21 01:27:56.626081 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:27:56.626033 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc214094-5e3b-4c52-9e78-ec8596dfaa68" containerName="test-curl-7496-140655670833456" Jul 21 01:27:56.626125 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:27:56.626107 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc214094-5e3b-4c52-9e78-ec8596dfaa68" containerName="test-curl-7496-140655670833456" Jul 21 01:27:56.629195 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:27:56.629178 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7496-140655676106624" Jul 21 01:27:56.635295 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:27:56.635273 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7496-140655676106624"] Jul 21 01:27:56.704997 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:27:56.704970 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxnhl\" (UniqueName: \"kubernetes.io/projected/525e0333-162b-4e2a-8800-84e61e204120-kube-api-access-lxnhl\") pod \"test-curl-7496-140655676106624\" (UID: \"525e0333-162b-4e2a-8800-84e61e204120\") " pod="opendatahub/test-curl-7496-140655676106624" Jul 21 01:27:56.806325 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:27:56.806298 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxnhl\" (UniqueName: \"kubernetes.io/projected/525e0333-162b-4e2a-8800-84e61e204120-kube-api-access-lxnhl\") pod \"test-curl-7496-140655676106624\" (UID: \"525e0333-162b-4e2a-8800-84e61e204120\") " pod="opendatahub/test-curl-7496-140655676106624" Jul 21 01:27:56.814644 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:27:56.814621 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxnhl\" (UniqueName: \"kubernetes.io/projected/525e0333-162b-4e2a-8800-84e61e204120-kube-api-access-lxnhl\") pod \"test-curl-7496-140655676106624\" (UID: \"525e0333-162b-4e2a-8800-84e61e204120\") " pod="opendatahub/test-curl-7496-140655676106624" Jul 21 01:27:56.913276 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:27:56.913209 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc214094-5e3b-4c52-9e78-ec8596dfaa68" path="/var/lib/kubelet/pods/bc214094-5e3b-4c52-9e78-ec8596dfaa68/volumes" Jul 21 01:27:56.940338 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:27:56.940317 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7496-140655676106624" Jul 21 01:27:57.077049 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:27:57.077023 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7496-140655676106624"] Jul 21 01:27:57.079246 ip-10-0-129-137 kubenswrapper[2580]: W0721 01:27:57.079209 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod525e0333_162b_4e2a_8800_84e61e204120.slice/crio-8c450056eb82cfece8a7f3e0e75afdfd69efd2f67c4c191521391f40b5997a3d WatchSource:0}: Error finding container 8c450056eb82cfece8a7f3e0e75afdfd69efd2f67c4c191521391f40b5997a3d: Status 404 returned error can't find the container with id 8c450056eb82cfece8a7f3e0e75afdfd69efd2f67c4c191521391f40b5997a3d Jul 21 01:27:57.280107 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:27:57.280010 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7496-140655670833456" Jul 21 01:27:57.280107 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:27:57.280030 2580 scope.go:117] "RemoveContainer" containerID="78ed58b9ba253598beca30fba716b9816a289707b014ab371f6e2b3f1358a5d0" Jul 21 01:27:57.281384 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:27:57.281276 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7496-140655676106624" event={"ID":"525e0333-162b-4e2a-8800-84e61e204120","Type":"ContainerStarted","Data":"8c450056eb82cfece8a7f3e0e75afdfd69efd2f67c4c191521391f40b5997a3d"} Jul 21 01:27:58.289338 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:27:58.289305 2580 generic.go:356] "Generic (PLEG): container finished" podID="525e0333-162b-4e2a-8800-84e61e204120" containerID="7376567238b0bf67011b5293f14259760f8f151b08f79d088859f4a7f5b64a86" exitCode=0 Jul 21 01:27:58.289727 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:27:58.289346 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7496-140655676106624" event={"ID":"525e0333-162b-4e2a-8800-84e61e204120","Type":"ContainerDied","Data":"7376567238b0bf67011b5293f14259760f8f151b08f79d088859f4a7f5b64a86"} Jul 21 01:27:59.437714 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:27:59.437691 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7496-140655676106624" Jul 21 01:27:59.457164 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:27:59.457139 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7496-140655676106624_525e0333-162b-4e2a-8800-84e61e204120/test-curl-7496-140655676106624/0.log" Jul 21 01:27:59.484046 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:27:59.484019 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7496-140655676106624"] Jul 21 01:27:59.486001 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:27:59.485977 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7496-140655676106624"] Jul 21 01:27:59.529983 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:27:59.529960 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lxnhl\" (UniqueName: \"kubernetes.io/projected/525e0333-162b-4e2a-8800-84e61e204120-kube-api-access-lxnhl\") pod \"525e0333-162b-4e2a-8800-84e61e204120\" (UID: \"525e0333-162b-4e2a-8800-84e61e204120\") " Jul 21 01:27:59.531962 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:27:59.531939 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/525e0333-162b-4e2a-8800-84e61e204120-kube-api-access-lxnhl" (OuterVolumeSpecName: "kube-api-access-lxnhl") pod "525e0333-162b-4e2a-8800-84e61e204120" (UID: "525e0333-162b-4e2a-8800-84e61e204120"). InnerVolumeSpecName "kube-api-access-lxnhl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 01:27:59.605237 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:27:59.605211 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7496-140655676106624"] Jul 21 01:27:59.605653 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:27:59.605640 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="525e0333-162b-4e2a-8800-84e61e204120" containerName="test-curl-7496-140655676106624" Jul 21 01:27:59.605699 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:27:59.605655 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="525e0333-162b-4e2a-8800-84e61e204120" containerName="test-curl-7496-140655676106624" Jul 21 01:27:59.605734 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:27:59.605723 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="525e0333-162b-4e2a-8800-84e61e204120" containerName="test-curl-7496-140655676106624" Jul 21 01:27:59.608982 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:27:59.608966 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7496-140655676106624" Jul 21 01:27:59.614800 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:27:59.614780 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7496-140655676106624"] Jul 21 01:27:59.641403 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:27:59.635794 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxnhl\" (UniqueName: \"kubernetes.io/projected/525e0333-162b-4e2a-8800-84e61e204120-kube-api-access-lxnhl\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 21 01:27:59.737060 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:27:59.737010 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bk86\" (UniqueName: \"kubernetes.io/projected/127c9c0d-53e3-42c7-8827-f3dbdc28356b-kube-api-access-6bk86\") pod \"test-curl-7496-140655676106624\" (UID: \"127c9c0d-53e3-42c7-8827-f3dbdc28356b\") " pod="opendatahub/test-curl-7496-140655676106624" Jul 21 01:27:59.838450 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:27:59.838418 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bk86\" (UniqueName: \"kubernetes.io/projected/127c9c0d-53e3-42c7-8827-f3dbdc28356b-kube-api-access-6bk86\") pod \"test-curl-7496-140655676106624\" (UID: \"127c9c0d-53e3-42c7-8827-f3dbdc28356b\") " pod="opendatahub/test-curl-7496-140655676106624" Jul 21 01:27:59.847154 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:27:59.847119 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bk86\" (UniqueName: \"kubernetes.io/projected/127c9c0d-53e3-42c7-8827-f3dbdc28356b-kube-api-access-6bk86\") pod \"test-curl-7496-140655676106624\" (UID: \"127c9c0d-53e3-42c7-8827-f3dbdc28356b\") " pod="opendatahub/test-curl-7496-140655676106624" Jul 21 01:27:59.930856 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:27:59.930792 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7496-140655676106624" Jul 21 01:28:00.061508 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:28:00.061483 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7496-140655676106624"] Jul 21 01:28:00.063383 ip-10-0-129-137 kubenswrapper[2580]: W0721 01:28:00.063348 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod127c9c0d_53e3_42c7_8827_f3dbdc28356b.slice/crio-05810d6103d69cf7ce0687520368b238683e4acb03d84b1087dc127ea2f87649 WatchSource:0}: Error finding container 05810d6103d69cf7ce0687520368b238683e4acb03d84b1087dc127ea2f87649: Status 404 returned error can't find the container with id 05810d6103d69cf7ce0687520368b238683e4acb03d84b1087dc127ea2f87649 Jul 21 01:28:00.301710 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:28:00.301626 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7496-140655676106624" event={"ID":"127c9c0d-53e3-42c7-8827-f3dbdc28356b","Type":"ContainerStarted","Data":"05810d6103d69cf7ce0687520368b238683e4acb03d84b1087dc127ea2f87649"} Jul 21 01:28:00.302789 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:28:00.302769 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c450056eb82cfece8a7f3e0e75afdfd69efd2f67c4c191521391f40b5997a3d" Jul 21 01:28:00.302910 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:28:00.302803 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7496-140655676106624" Jul 21 01:28:00.318881 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:28:00.318850 2580 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7496-140655676106624" oldPodUID="525e0333-162b-4e2a-8800-84e61e204120" podUID="127c9c0d-53e3-42c7-8827-f3dbdc28356b" Jul 21 01:28:00.912182 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:28:00.912150 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="525e0333-162b-4e2a-8800-84e61e204120" path="/var/lib/kubelet/pods/525e0333-162b-4e2a-8800-84e61e204120/volumes" Jul 21 01:28:01.308824 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:28:01.308740 2580 generic.go:356] "Generic (PLEG): container finished" podID="127c9c0d-53e3-42c7-8827-f3dbdc28356b" containerID="2cb6a1978f185eb87dc46e74892323046eede303eb53e67b832733c54e2787c4" exitCode=0 Jul 21 01:28:01.308824 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:28:01.308814 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7496-140655676106624" event={"ID":"127c9c0d-53e3-42c7-8827-f3dbdc28356b","Type":"ContainerDied","Data":"2cb6a1978f185eb87dc46e74892323046eede303eb53e67b832733c54e2787c4"} Jul 21 01:28:02.457980 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:28:02.457958 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7496-140655676106624" Jul 21 01:28:02.477880 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:28:02.477848 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7496-140655676106624_127c9c0d-53e3-42c7-8827-f3dbdc28356b/test-curl-7496-140655676106624/0.log" Jul 21 01:28:02.504817 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:28:02.504787 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7496-140655676106624"] Jul 21 01:28:02.506562 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:28:02.506542 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7496-140655676106624"] Jul 21 01:28:02.564645 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:28:02.564573 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6bk86\" (UniqueName: \"kubernetes.io/projected/127c9c0d-53e3-42c7-8827-f3dbdc28356b-kube-api-access-6bk86\") pod \"127c9c0d-53e3-42c7-8827-f3dbdc28356b\" (UID: \"127c9c0d-53e3-42c7-8827-f3dbdc28356b\") " Jul 21 01:28:02.566567 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:28:02.566545 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/127c9c0d-53e3-42c7-8827-f3dbdc28356b-kube-api-access-6bk86" (OuterVolumeSpecName: "kube-api-access-6bk86") pod "127c9c0d-53e3-42c7-8827-f3dbdc28356b" (UID: "127c9c0d-53e3-42c7-8827-f3dbdc28356b"). InnerVolumeSpecName "kube-api-access-6bk86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 01:28:02.630324 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:28:02.627389 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7496-140655676108064"] Jul 21 01:28:02.630324 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:28:02.628285 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="127c9c0d-53e3-42c7-8827-f3dbdc28356b" containerName="test-curl-7496-140655676106624" Jul 21 01:28:02.630324 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:28:02.628308 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="127c9c0d-53e3-42c7-8827-f3dbdc28356b" containerName="test-curl-7496-140655676106624" Jul 21 01:28:02.630324 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:28:02.628554 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="127c9c0d-53e3-42c7-8827-f3dbdc28356b" containerName="test-curl-7496-140655676106624" Jul 21 01:28:02.632394 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:28:02.632368 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7496-140655676108064" Jul 21 01:28:02.634960 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:28:02.634938 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7496-140655676108064"] Jul 21 01:28:02.665650 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:28:02.665624 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bk86\" (UniqueName: \"kubernetes.io/projected/127c9c0d-53e3-42c7-8827-f3dbdc28356b-kube-api-access-6bk86\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 21 01:28:02.766181 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:28:02.766145 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pw2ls\" (UniqueName: \"kubernetes.io/projected/9eb18164-8044-43aa-9ff3-7a741e5a075e-kube-api-access-pw2ls\") pod \"test-curl-7496-140655676108064\" (UID: \"9eb18164-8044-43aa-9ff3-7a741e5a075e\") " pod="opendatahub/test-curl-7496-140655676108064" Jul 21 01:28:02.867443 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:28:02.867407 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw2ls\" (UniqueName: \"kubernetes.io/projected/9eb18164-8044-43aa-9ff3-7a741e5a075e-kube-api-access-pw2ls\") pod \"test-curl-7496-140655676108064\" (UID: \"9eb18164-8044-43aa-9ff3-7a741e5a075e\") " pod="opendatahub/test-curl-7496-140655676108064" Jul 21 01:28:02.875503 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:28:02.875480 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw2ls\" (UniqueName: \"kubernetes.io/projected/9eb18164-8044-43aa-9ff3-7a741e5a075e-kube-api-access-pw2ls\") pod \"test-curl-7496-140655676108064\" (UID: \"9eb18164-8044-43aa-9ff3-7a741e5a075e\") " pod="opendatahub/test-curl-7496-140655676108064" Jul 21 01:28:02.912766 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:28:02.912738 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="127c9c0d-53e3-42c7-8827-f3dbdc28356b" path="/var/lib/kubelet/pods/127c9c0d-53e3-42c7-8827-f3dbdc28356b/volumes" Jul 21 01:28:02.944932 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:28:02.944908 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7496-140655676108064" Jul 21 01:28:03.078848 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:28:03.078818 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7496-140655676108064"] Jul 21 01:28:03.080883 ip-10-0-129-137 kubenswrapper[2580]: W0721 01:28:03.080857 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9eb18164_8044_43aa_9ff3_7a741e5a075e.slice/crio-7af65c5095003b759e5bbfc7d711fc3ee0f8b08d1ba471d6e6de8796cfe61c18 WatchSource:0}: Error finding container 7af65c5095003b759e5bbfc7d711fc3ee0f8b08d1ba471d6e6de8796cfe61c18: Status 404 returned error can't find the container with id 7af65c5095003b759e5bbfc7d711fc3ee0f8b08d1ba471d6e6de8796cfe61c18 Jul 21 01:28:03.319800 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:28:03.319704 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7496-140655676108064" event={"ID":"9eb18164-8044-43aa-9ff3-7a741e5a075e","Type":"ContainerStarted","Data":"7af65c5095003b759e5bbfc7d711fc3ee0f8b08d1ba471d6e6de8796cfe61c18"} Jul 21 01:28:03.320935 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:28:03.320917 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7496-140655676106624" Jul 21 01:28:03.321011 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:28:03.320929 2580 scope.go:117] "RemoveContainer" containerID="2cb6a1978f185eb87dc46e74892323046eede303eb53e67b832733c54e2787c4" Jul 21 01:28:04.327216 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:28:04.327181 2580 generic.go:356] "Generic (PLEG): container finished" podID="9eb18164-8044-43aa-9ff3-7a741e5a075e" containerID="be9e9ca95304c551e901a38354e0e5f20f82c274e1d1bfe1a7c984b73c96cb66" exitCode=0 Jul 21 01:28:04.327654 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:28:04.327270 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7496-140655676108064" event={"ID":"9eb18164-8044-43aa-9ff3-7a741e5a075e","Type":"ContainerDied","Data":"be9e9ca95304c551e901a38354e0e5f20f82c274e1d1bfe1a7c984b73c96cb66"} Jul 21 01:28:05.466681 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:28:05.466658 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7496-140655676108064" Jul 21 01:28:05.487690 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:28:05.487667 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7496-140655676108064_9eb18164-8044-43aa-9ff3-7a741e5a075e/test-curl-7496-140655676108064/0.log" Jul 21 01:28:05.513159 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:28:05.513136 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7496-140655676108064"] Jul 21 01:28:05.514934 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:28:05.514915 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7496-140655676108064"] Jul 21 01:28:05.592358 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:28:05.592293 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pw2ls\" (UniqueName: \"kubernetes.io/projected/9eb18164-8044-43aa-9ff3-7a741e5a075e-kube-api-access-pw2ls\") pod \"9eb18164-8044-43aa-9ff3-7a741e5a075e\" (UID: \"9eb18164-8044-43aa-9ff3-7a741e5a075e\") " Jul 21 01:28:05.594118 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:28:05.594097 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9eb18164-8044-43aa-9ff3-7a741e5a075e-kube-api-access-pw2ls" (OuterVolumeSpecName: "kube-api-access-pw2ls") pod "9eb18164-8044-43aa-9ff3-7a741e5a075e" (UID: "9eb18164-8044-43aa-9ff3-7a741e5a075e"). InnerVolumeSpecName "kube-api-access-pw2ls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 01:28:05.693874 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:28:05.693836 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pw2ls\" (UniqueName: \"kubernetes.io/projected/9eb18164-8044-43aa-9ff3-7a741e5a075e-kube-api-access-pw2ls\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 21 01:28:06.340637 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:28:06.340606 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7496-140655676108064" Jul 21 01:28:06.340637 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:28:06.340625 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7af65c5095003b759e5bbfc7d711fc3ee0f8b08d1ba471d6e6de8796cfe61c18" Jul 21 01:28:06.353150 ip-10-0-129-137 kubenswrapper[2580]: E0721 01:28:06.353125 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7496-140655676108064\" is forbidden: User \"system:node:ip-10-0-129-137.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-137.ec2.internal' and this object" podUID="9eb18164-8044-43aa-9ff3-7a741e5a075e" pod="opendatahub/test-curl-7496-140655676108064" Jul 21 01:28:06.914376 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:28:06.912855 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9eb18164-8044-43aa-9ff3-7a741e5a075e" path="/var/lib/kubelet/pods/9eb18164-8044-43aa-9ff3-7a741e5a075e/volumes" Jul 21 01:28:31.615022 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:28:31.614986 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-03c5cb-br4wm"] Jul 21 01:28:31.615543 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:28:31.615426 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9eb18164-8044-43aa-9ff3-7a741e5a075e" containerName="test-curl-7496-140655676108064" Jul 21 01:28:31.615543 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:28:31.615438 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9eb18164-8044-43aa-9ff3-7a741e5a075e" containerName="test-curl-7496-140655676108064" Jul 21 01:28:31.615543 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:28:31.615508 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9eb18164-8044-43aa-9ff3-7a741e5a075e" containerName="test-curl-7496-140655676108064" Jul 21 01:28:31.620357 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:28:31.620337 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-03c5cb-br4wm" Jul 21 01:28:31.623157 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:28:31.623119 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8m7v8\"" Jul 21 01:28:31.625802 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:28:31.625775 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-03c5cb-br4wm"] Jul 21 01:28:31.731422 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:28:31.731386 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0eb6f227-b144-48a8-855b-1b400e2e19d5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-03c5cb-br4wm\" (UID: \"0eb6f227-b144-48a8-855b-1b400e2e19d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-03c5cb-br4wm" Jul 21 01:28:31.832502 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:28:31.832470 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0eb6f227-b144-48a8-855b-1b400e2e19d5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-03c5cb-br4wm\" (UID: \"0eb6f227-b144-48a8-855b-1b400e2e19d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-03c5cb-br4wm" Jul 21 01:28:31.833157 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:28:31.833136 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0eb6f227-b144-48a8-855b-1b400e2e19d5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-03c5cb-br4wm\" (UID: \"0eb6f227-b144-48a8-855b-1b400e2e19d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-03c5cb-br4wm" Jul 21 01:28:31.933399 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:28:31.933319 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-03c5cb-br4wm" Jul 21 01:28:32.074314 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:28:32.074275 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-03c5cb-br4wm"] Jul 21 01:28:32.076130 ip-10-0-129-137 kubenswrapper[2580]: W0721 01:28:32.076098 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eb6f227_b144_48a8_855b_1b400e2e19d5.slice/crio-e5ee06a0fc028a34b6fe4a0299544fe9f6a0e46cfbce83526f69615d4c24f337 WatchSource:0}: Error finding container e5ee06a0fc028a34b6fe4a0299544fe9f6a0e46cfbce83526f69615d4c24f337: Status 404 returned error can't find the container with id e5ee06a0fc028a34b6fe4a0299544fe9f6a0e46cfbce83526f69615d4c24f337 Jul 21 01:28:32.486377 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:28:32.486343 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-03c5cb-br4wm" event={"ID":"0eb6f227-b144-48a8-855b-1b400e2e19d5","Type":"ContainerStarted","Data":"e5ee06a0fc028a34b6fe4a0299544fe9f6a0e46cfbce83526f69615d4c24f337"} Jul 21 01:28:35.504057 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:28:35.504017 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-03c5cb-br4wm" event={"ID":"0eb6f227-b144-48a8-855b-1b400e2e19d5","Type":"ContainerStarted","Data":"721264a6c2cb084991491d2dcfb9f00c11fff3074a951908106d2147e446effc"} Jul 21 01:28:35.521426 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:28:35.521380 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-03c5cb-br4wm" podStartSLOduration=1.5797075729999999 podStartE2EDuration="4.521366037s" totalImagesPullingTime="2.941658464s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 01:28:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:28:35.518666047 +0000 UTC m=+2863.124775087" watchObservedRunningTime="2026-07-21 01:28:35.521366037 +0000 UTC m=+2863.127475076" Jul 21 01:28:36.510698 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:28:36.510666 2580 generic.go:356] "Generic (PLEG): container finished" podID="0eb6f227-b144-48a8-855b-1b400e2e19d5" containerID="721264a6c2cb084991491d2dcfb9f00c11fff3074a951908106d2147e446effc" exitCode=0 Jul 21 01:28:36.511069 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:28:36.510754 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-03c5cb-br4wm" event={"ID":"0eb6f227-b144-48a8-855b-1b400e2e19d5","Type":"ContainerDied","Data":"721264a6c2cb084991491d2dcfb9f00c11fff3074a951908106d2147e446effc"} Jul 21 01:28:37.652080 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:28:37.652055 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-03c5cb-br4wm" Jul 21 01:28:37.688038 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:28:37.688006 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0eb6f227-b144-48a8-855b-1b400e2e19d5-maas-api-service-ca\") pod \"0eb6f227-b144-48a8-855b-1b400e2e19d5\" (UID: \"0eb6f227-b144-48a8-855b-1b400e2e19d5\") " Jul 21 01:28:37.688333 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:28:37.688311 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0eb6f227-b144-48a8-855b-1b400e2e19d5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0eb6f227-b144-48a8-855b-1b400e2e19d5" (UID: "0eb6f227-b144-48a8-855b-1b400e2e19d5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 01:28:37.688446 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:28:37.688427 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0eb6f227-b144-48a8-855b-1b400e2e19d5-maas-api-service-ca\") on node \"ip-10-0-129-137.ec2.internal\" DevicePath \"\"" Jul 21 01:28:38.522433 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:28:38.522404 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-03c5cb-br4wm" Jul 21 01:28:38.522595 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:28:38.522430 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-03c5cb-br4wm" event={"ID":"0eb6f227-b144-48a8-855b-1b400e2e19d5","Type":"ContainerDied","Data":"e5ee06a0fc028a34b6fe4a0299544fe9f6a0e46cfbce83526f69615d4c24f337"} Jul 21 01:28:38.522595 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:28:38.522471 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e5ee06a0fc028a34b6fe4a0299544fe9f6a0e46cfbce83526f69615d4c24f337" Jul 21 01:28:46.685652 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:28:46.685567 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-03c5cb-br4wm"] Jul 21 01:28:46.687694 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:28:46.687673 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-03c5cb-br4wm"] Jul 21 01:28:46.912397 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:28:46.912366 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0eb6f227-b144-48a8-855b-1b400e2e19d5" path="/var/lib/kubelet/pods/0eb6f227-b144-48a8-855b-1b400e2e19d5/volumes" Jul 21 01:28:53.754626 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:28:53.754597 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7b9456ddfc-nvvzj_b48d40a2-bf25-42bb-95df-b088de4ac80a/authorino/0.log" Jul 21 01:28:59.271869 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:28:59.271832 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-4zltv_6993a3d6-6e24-48cf-b63c-35f1cbf9c771/manager/0.log" Jul 21 01:28:59.395532 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:28:59.395503 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-nhnq7_86f64a6f-7e40-4f9d-84e9-f1849e0340c9/manager/0.log" Jul 21 01:28:59.529497 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:28:59.529419 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7765874568-dn64s_017bf68e-1d4d-4145-b90b-ee6ad6d0100f/manager/2.log" Jul 21 01:28:59.648459 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:28:59.648428 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-vr6pj_d9a7a22c-1842-4eca-9b2d-4e3509bcf9f1/server/0.log" Jul 21 01:28:59.772415 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:28:59.772381 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-zmq6v_115ede03-1e4e-432b-a874-579f0fa66f73/manager/1.log" Jul 21 01:29:00.038337 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:00.038308 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5f889c5d8-szrmh_dec0567c-68f4-474f-862e-bf50cb9e0af5/copy-manifests/0.log" Jul 21 01:29:00.054256 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:00.054203 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5f889c5d8-szrmh_dec0567c-68f4-474f-862e-bf50cb9e0af5/manager/0.log" Jul 21 01:29:00.537021 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:00.536990 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6bc4785f87-2p6gx_761360e3-de73-49dd-9bf3-292a44066f1a/maas-api/0.log" Jul 21 01:29:00.895045 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:00.895017 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-vkmt7_983b7ef4-755f-467b-a0da-f5683974deb4/postgres/0.log" Jul 21 01:29:02.276516 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:02.276485 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7b9456ddfc-nvvzj_b48d40a2-bf25-42bb-95df-b088de4ac80a/authorino/0.log" Jul 21 01:29:02.416013 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:02.415981 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-gpzbr_0a38eb15-7b87-4f2b-9033-9c3e8715f642/manager/0.log" Jul 21 01:29:02.551140 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:02.551067 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-n5dmt_c33d7edb-ac53-4bd2-9a59-bc09fd48ae4c/manager/0.log" Jul 21 01:29:02.797078 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:02.797054 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-7dg7r_11ed89e1-5c04-404c-84dc-16a12138c760/registry-server/0.log" Jul 21 01:29:02.933542 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:02.933506 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-jmp7l_90ffe3aa-c193-4a69-a632-e2d1cb0013f7/manager/0.log" Jul 21 01:29:03.191256 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:03.191173 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-tgtqn_a53c5164-d2ff-43eb-bffa-3dd49d9f0310/manager/0.log" Jul 21 01:29:03.792562 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:03.792533 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5c7494f749-46ckj_8d95153d-b67d-4c35-81cb-7dc970ac6ba6/kube-auth-proxy/0.log" Jul 21 01:29:04.190751 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:04.190725 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-v45mh_ae4bd066-5fdf-4c2c-883c-bee5800b0069/payload-pre-processing/0.log" Jul 21 01:29:04.456396 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:04.456325 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-69fd648678-dm5nn_9970c160-3cf7-4d90-b892-1a337f23a065/router/0.log" Jul 21 01:29:04.813435 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:04.813366 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-7nhqf_959d2ecf-daef-4d66-b1a3-153e0f52d094/main/0.log" Jul 21 01:29:04.939284 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:04.939255 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-s4b8b_58f90218-920c-48a6-b0d8-3373396703c0/main/0.log" Jul 21 01:29:05.314725 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:05.314691 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-hw24q_db171be2-aedf-4c85-b387-c90902e082b9/main/0.log" Jul 21 01:29:14.773270 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:14.773245 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-dzcmw_fda1cd2e-3d5d-4dc1-94a1-1b8ebbb4a199/global-pull-secret-syncer/0.log" Jul 21 01:29:14.878182 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:14.878146 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-ntk2c_e7629fc1-b058-4431-b0d3-1b465dee3b48/konnectivity-agent/0.log" Jul 21 01:29:14.978584 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:14.978556 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-137.ec2.internal_f449d6f92faa6a51a14c54944141985e/haproxy/0.log" Jul 21 01:29:20.176957 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:20.176918 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7b9456ddfc-nvvzj_b48d40a2-bf25-42bb-95df-b088de4ac80a/authorino/0.log" Jul 21 01:29:20.206996 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:20.206969 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-gpzbr_0a38eb15-7b87-4f2b-9033-9c3e8715f642/manager/0.log" Jul 21 01:29:20.236006 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:20.235985 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-n5dmt_c33d7edb-ac53-4bd2-9a59-bc09fd48ae4c/manager/0.log" Jul 21 01:29:20.307330 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:20.307270 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-7dg7r_11ed89e1-5c04-404c-84dc-16a12138c760/registry-server/0.log" Jul 21 01:29:20.404340 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:20.404311 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-jmp7l_90ffe3aa-c193-4a69-a632-e2d1cb0013f7/manager/0.log" Jul 21 01:29:20.623280 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:20.623252 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-tgtqn_a53c5164-d2ff-43eb-bffa-3dd49d9f0310/manager/0.log" Jul 21 01:29:22.491480 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:22.491449 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-b8c9w_8379df88-86a0-4d28-95fc-a102fff737fd/cluster-monitoring-operator/0.log" Jul 21 01:29:22.807029 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:22.807001 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-z6c8m_de793b0f-6ee5-4ed5-b11c-9850496fd84a/node-exporter/0.log" Jul 21 01:29:22.830879 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:22.830858 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-z6c8m_de793b0f-6ee5-4ed5-b11c-9850496fd84a/kube-rbac-proxy/0.log" Jul 21 01:29:22.855596 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:22.855575 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-z6c8m_de793b0f-6ee5-4ed5-b11c-9850496fd84a/init-textfile/0.log" Jul 21 01:29:22.999639 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:22.999569 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f/prometheus/0.log" Jul 21 01:29:23.025425 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:23.025404 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f/config-reloader/0.log" Jul 21 01:29:23.048827 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:23.048801 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f/thanos-sidecar/0.log" Jul 21 01:29:23.075331 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:23.075311 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f/kube-rbac-proxy-web/0.log" Jul 21 01:29:23.103708 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:23.103689 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f/kube-rbac-proxy/0.log" Jul 21 01:29:23.131765 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:23.131748 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f/kube-rbac-proxy-thanos/0.log" Jul 21 01:29:23.159586 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:23.159566 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5f8dde4a-ce95-4b0d-80d5-c7c26fa3a41f/init-config-reloader/0.log" Jul 21 01:29:24.666038 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:24.666010 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-7bc7489cb5-sx8h8_40960433-ccd7-4aa8-ab03-2cde39f80875/networking-console-plugin/0.log" Jul 21 01:29:24.685697 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:24.685673 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-6qs98/perf-node-gather-daemonset-gfrnd"] Jul 21 01:29:24.686078 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:24.686066 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0eb6f227-b144-48a8-855b-1b400e2e19d5" containerName="revoke-keys" Jul 21 01:29:24.686124 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:24.686080 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0eb6f227-b144-48a8-855b-1b400e2e19d5" containerName="revoke-keys" Jul 21 01:29:24.686176 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:24.686166 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0eb6f227-b144-48a8-855b-1b400e2e19d5" containerName="revoke-keys" Jul 21 01:29:24.699768 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:24.699744 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6qs98/perf-node-gather-daemonset-gfrnd"] Jul 21 01:29:24.699917 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:24.699832 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6qs98/perf-node-gather-daemonset-gfrnd" Jul 21 01:29:24.702501 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:24.702480 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-6qs98\"/\"default-dockercfg-78gjl\"" Jul 21 01:29:24.703728 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:24.703703 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6qs98\"/\"kube-root-ca.crt\"" Jul 21 01:29:24.703872 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:24.703802 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6qs98\"/\"openshift-service-ca.crt\"" Jul 21 01:29:24.805255 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:24.805205 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1195bc5f-0584-4276-845d-964b061cd064-proc\") pod \"perf-node-gather-daemonset-gfrnd\" (UID: \"1195bc5f-0584-4276-845d-964b061cd064\") " pod="openshift-must-gather-6qs98/perf-node-gather-daemonset-gfrnd" Jul 21 01:29:24.805415 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:24.805302 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1195bc5f-0584-4276-845d-964b061cd064-sys\") pod \"perf-node-gather-daemonset-gfrnd\" (UID: \"1195bc5f-0584-4276-845d-964b061cd064\") " pod="openshift-must-gather-6qs98/perf-node-gather-daemonset-gfrnd" Jul 21 01:29:24.805415 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:24.805343 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1195bc5f-0584-4276-845d-964b061cd064-lib-modules\") pod \"perf-node-gather-daemonset-gfrnd\" (UID: \"1195bc5f-0584-4276-845d-964b061cd064\") " pod="openshift-must-gather-6qs98/perf-node-gather-daemonset-gfrnd" Jul 21 01:29:24.805415 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:24.805385 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmhjd\" (UniqueName: \"kubernetes.io/projected/1195bc5f-0584-4276-845d-964b061cd064-kube-api-access-jmhjd\") pod \"perf-node-gather-daemonset-gfrnd\" (UID: \"1195bc5f-0584-4276-845d-964b061cd064\") " pod="openshift-must-gather-6qs98/perf-node-gather-daemonset-gfrnd" Jul 21 01:29:24.805532 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:24.805419 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1195bc5f-0584-4276-845d-964b061cd064-podres\") pod \"perf-node-gather-daemonset-gfrnd\" (UID: \"1195bc5f-0584-4276-845d-964b061cd064\") " pod="openshift-must-gather-6qs98/perf-node-gather-daemonset-gfrnd" Jul 21 01:29:24.906040 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:24.906008 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmhjd\" (UniqueName: \"kubernetes.io/projected/1195bc5f-0584-4276-845d-964b061cd064-kube-api-access-jmhjd\") pod \"perf-node-gather-daemonset-gfrnd\" (UID: \"1195bc5f-0584-4276-845d-964b061cd064\") " pod="openshift-must-gather-6qs98/perf-node-gather-daemonset-gfrnd" Jul 21 01:29:24.906196 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:24.906051 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1195bc5f-0584-4276-845d-964b061cd064-podres\") pod \"perf-node-gather-daemonset-gfrnd\" (UID: \"1195bc5f-0584-4276-845d-964b061cd064\") " pod="openshift-must-gather-6qs98/perf-node-gather-daemonset-gfrnd" Jul 21 01:29:24.906196 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:24.906074 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1195bc5f-0584-4276-845d-964b061cd064-proc\") pod \"perf-node-gather-daemonset-gfrnd\" (UID: \"1195bc5f-0584-4276-845d-964b061cd064\") " pod="openshift-must-gather-6qs98/perf-node-gather-daemonset-gfrnd" Jul 21 01:29:24.906196 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:24.906105 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1195bc5f-0584-4276-845d-964b061cd064-sys\") pod \"perf-node-gather-daemonset-gfrnd\" (UID: \"1195bc5f-0584-4276-845d-964b061cd064\") " pod="openshift-must-gather-6qs98/perf-node-gather-daemonset-gfrnd" Jul 21 01:29:24.906196 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:24.906134 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1195bc5f-0584-4276-845d-964b061cd064-lib-modules\") pod \"perf-node-gather-daemonset-gfrnd\" (UID: \"1195bc5f-0584-4276-845d-964b061cd064\") " pod="openshift-must-gather-6qs98/perf-node-gather-daemonset-gfrnd" Jul 21 01:29:24.906405 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:24.906206 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1195bc5f-0584-4276-845d-964b061cd064-podres\") pod \"perf-node-gather-daemonset-gfrnd\" (UID: \"1195bc5f-0584-4276-845d-964b061cd064\") " pod="openshift-must-gather-6qs98/perf-node-gather-daemonset-gfrnd" Jul 21 01:29:24.906405 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:24.906255 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1195bc5f-0584-4276-845d-964b061cd064-proc\") pod \"perf-node-gather-daemonset-gfrnd\" (UID: \"1195bc5f-0584-4276-845d-964b061cd064\") " pod="openshift-must-gather-6qs98/perf-node-gather-daemonset-gfrnd" Jul 21 01:29:24.906405 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:24.906271 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1195bc5f-0584-4276-845d-964b061cd064-sys\") pod \"perf-node-gather-daemonset-gfrnd\" (UID: \"1195bc5f-0584-4276-845d-964b061cd064\") " pod="openshift-must-gather-6qs98/perf-node-gather-daemonset-gfrnd" Jul 21 01:29:24.906405 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:24.906293 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1195bc5f-0584-4276-845d-964b061cd064-lib-modules\") pod \"perf-node-gather-daemonset-gfrnd\" (UID: \"1195bc5f-0584-4276-845d-964b061cd064\") " pod="openshift-must-gather-6qs98/perf-node-gather-daemonset-gfrnd" Jul 21 01:29:24.914128 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:24.914107 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmhjd\" (UniqueName: \"kubernetes.io/projected/1195bc5f-0584-4276-845d-964b061cd064-kube-api-access-jmhjd\") pod \"perf-node-gather-daemonset-gfrnd\" (UID: \"1195bc5f-0584-4276-845d-964b061cd064\") " pod="openshift-must-gather-6qs98/perf-node-gather-daemonset-gfrnd" Jul 21 01:29:25.010928 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:25.010868 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6qs98/perf-node-gather-daemonset-gfrnd" Jul 21 01:29:25.144608 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:25.144586 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6qs98/perf-node-gather-daemonset-gfrnd"] Jul 21 01:29:25.147196 ip-10-0-129-137 kubenswrapper[2580]: W0721 01:29:25.147171 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1195bc5f_0584_4276_845d_964b061cd064.slice/crio-8d909e6ea8622bb3680b2758640065bb7e4f8cf47d651b81a6f52d376fca6659 WatchSource:0}: Error finding container 8d909e6ea8622bb3680b2758640065bb7e4f8cf47d651b81a6f52d376fca6659: Status 404 returned error can't find the container with id 8d909e6ea8622bb3680b2758640065bb7e4f8cf47d651b81a6f52d376fca6659 Jul 21 01:29:25.376848 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:25.376821 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-rb4ph_3ee1dbf0-b4ff-4871-9432-284eb25f97b1/console-operator/1.log" Jul 21 01:29:25.381869 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:25.381853 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-rb4ph_3ee1dbf0-b4ff-4871-9432-284eb25f97b1/console-operator/2.log" Jul 21 01:29:25.769845 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:25.769747 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6qs98/perf-node-gather-daemonset-gfrnd" event={"ID":"1195bc5f-0584-4276-845d-964b061cd064","Type":"ContainerStarted","Data":"1d6051edbb859963a4d96eaf37ad3dc9c94ccba2e93ae58a8fa28ecc50639b3a"} Jul 21 01:29:25.769845 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:25.769803 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6qs98/perf-node-gather-daemonset-gfrnd" event={"ID":"1195bc5f-0584-4276-845d-964b061cd064","Type":"ContainerStarted","Data":"8d909e6ea8622bb3680b2758640065bb7e4f8cf47d651b81a6f52d376fca6659"} Jul 21 01:29:25.769845 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:25.769831 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-6qs98/perf-node-gather-daemonset-gfrnd" Jul 21 01:29:25.787411 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:25.787359 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-6qs98/perf-node-gather-daemonset-gfrnd" podStartSLOduration=1.787344914 podStartE2EDuration="1.787344914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 01:29:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:29:25.786287794 +0000 UTC m=+2913.392396837" watchObservedRunningTime="2026-07-21 01:29:25.787344914 +0000 UTC m=+2913.393453988" Jul 21 01:29:25.911639 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:25.911613 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-5bc67dcd46-wnqtz_d4c2ffde-2686-4b53-a5f8-03df48ab6b6f/download-server/0.log" Jul 21 01:29:26.499104 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:26.499077 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78cf4c6f6-mss59_f43e318f-e1aa-4f27-971b-615e797691d9/volume-data-source-validator/0.log" Jul 21 01:29:27.591431 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:27.591405 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-nlszd_49cc45d2-5087-447a-9123-3885c7bc03a2/dns/0.log" Jul 21 01:29:27.616292 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:27.616262 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-nlszd_49cc45d2-5087-447a-9123-3885c7bc03a2/kube-rbac-proxy/0.log" Jul 21 01:29:27.699785 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:27.699762 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-858wv_df3aad65-414d-4dd0-9e23-c0c0e46e4f06/dns-node-resolver/0.log" Jul 21 01:29:28.291470 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:28.291441 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5dbc6848f8-w729h_6c1e84ef-e02d-4b63-8538-e88272dfdffd/registry/0.log" Jul 21 01:29:28.315471 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:28.315449 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2rxh4_6fbd5477-ce7d-444d-938b-2c54aa3c040c/node-ca/0.log" Jul 21 01:29:29.426089 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:29.426067 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5c7494f749-46ckj_8d95153d-b67d-4c35-81cb-7dc970ac6ba6/kube-auth-proxy/0.log" Jul 21 01:29:29.537661 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:29.537632 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-v45mh_ae4bd066-5fdf-4c2c-883c-bee5800b0069/payload-pre-processing/0.log" Jul 21 01:29:29.591635 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:29.591608 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-69fd648678-dm5nn_9970c160-3cf7-4d90-b892-1a337f23a065/router/0.log" Jul 21 01:29:30.277211 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:30.277183 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2vh4t_a625da2a-a501-4fa3-a8c0-21ac34950978/serve-healthcheck-canary/0.log" Jul 21 01:29:30.811882 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:30.811847 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-9rqvh_826d0b75-8b78-4bf5-9a54-d71162f8e0da/insights-operator/0.log" Jul 21 01:29:30.812530 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:30.812501 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-9rqvh_826d0b75-8b78-4bf5-9a54-d71162f8e0da/insights-operator/1.log" Jul 21 01:29:30.834400 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:30.834377 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5xx6h_2d8da6d4-7c9a-42d9-ac48-581a8640f880/kube-rbac-proxy/0.log" Jul 21 01:29:30.857356 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:30.857332 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5xx6h_2d8da6d4-7c9a-42d9-ac48-581a8640f880/exporter/0.log" Jul 21 01:29:30.880095 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:30.880076 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5xx6h_2d8da6d4-7c9a-42d9-ac48-581a8640f880/extractor/0.log" Jul 21 01:29:31.785190 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:31.785164 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-6qs98/perf-node-gather-daemonset-gfrnd" Jul 21 01:29:33.219829 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:33.219802 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-4zltv_6993a3d6-6e24-48cf-b63c-35f1cbf9c771/manager/0.log" Jul 21 01:29:33.400463 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:33.400426 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-nhnq7_86f64a6f-7e40-4f9d-84e9-f1849e0340c9/manager/0.log" Jul 21 01:29:33.421161 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:33.421139 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7765874568-dn64s_017bf68e-1d4d-4145-b90b-ee6ad6d0100f/manager/1.log" Jul 21 01:29:33.451928 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:33.451902 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7765874568-dn64s_017bf68e-1d4d-4145-b90b-ee6ad6d0100f/manager/2.log" Jul 21 01:29:33.473312 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:33.473255 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-vr6pj_d9a7a22c-1842-4eca-9b2d-4e3509bcf9f1/server/0.log" Jul 21 01:29:33.541633 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:33.541604 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-zmq6v_115ede03-1e4e-432b-a874-579f0fa66f73/manager/1.log" Jul 21 01:29:33.549278 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:33.549257 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-zmq6v_115ede03-1e4e-432b-a874-579f0fa66f73/manager/0.log" Jul 21 01:29:33.686568 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:33.686540 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5f889c5d8-szrmh_dec0567c-68f4-474f-862e-bf50cb9e0af5/manager/0.log" Jul 21 01:29:33.706982 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:33.706959 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5f889c5d8-szrmh_dec0567c-68f4-474f-862e-bf50cb9e0af5/copy-manifests/0.log" Jul 21 01:29:36.244546 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:36.244520 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5c7494f749-46ckj_8d95153d-b67d-4c35-81cb-7dc970ac6ba6/kube-auth-proxy/0.log" Jul 21 01:29:36.357557 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:36.357532 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-v45mh_ae4bd066-5fdf-4c2c-883c-bee5800b0069/payload-pre-processing/0.log" Jul 21 01:29:36.399302 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:36.399268 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-69fd648678-dm5nn_9970c160-3cf7-4d90-b892-1a337f23a065/router/0.log" Jul 21 01:29:37.727839 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:37.727796 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5c7494f749-46ckj_8d95153d-b67d-4c35-81cb-7dc970ac6ba6/kube-auth-proxy/0.log" Jul 21 01:29:37.821202 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:37.821176 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-v45mh_ae4bd066-5fdf-4c2c-883c-bee5800b0069/payload-pre-processing/0.log" Jul 21 01:29:37.862363 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:37.862338 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-69fd648678-dm5nn_9970c160-3cf7-4d90-b892-1a337f23a065/router/0.log" Jul 21 01:29:40.769849 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:40.769818 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-tdb28_f7fa2eb6-5645-47f1-8199-1dd64ea512dc/kube-storage-version-migrator-operator/1.log" Jul 21 01:29:40.770751 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:40.770730 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-tdb28_f7fa2eb6-5645-47f1-8199-1dd64ea512dc/kube-storage-version-migrator-operator/0.log" Jul 21 01:29:41.792543 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:41.792511 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6zg78_f3e26871-c996-46c1-adfb-c96c1869814b/kube-multus-additional-cni-plugins/0.log" Jul 21 01:29:41.815616 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:41.815583 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6zg78_f3e26871-c996-46c1-adfb-c96c1869814b/egress-router-binary-copy/0.log" Jul 21 01:29:41.842620 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:41.842598 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6zg78_f3e26871-c996-46c1-adfb-c96c1869814b/cni-plugins/0.log" Jul 21 01:29:41.866956 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:41.866931 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6zg78_f3e26871-c996-46c1-adfb-c96c1869814b/bond-cni-plugin/0.log" Jul 21 01:29:41.891434 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:41.891409 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6zg78_f3e26871-c996-46c1-adfb-c96c1869814b/routeoverride-cni/0.log" Jul 21 01:29:41.914781 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:41.914755 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6zg78_f3e26871-c996-46c1-adfb-c96c1869814b/whereabouts-cni-bincopy/0.log" Jul 21 01:29:41.939948 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:41.939925 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6zg78_f3e26871-c996-46c1-adfb-c96c1869814b/whereabouts-cni/0.log" Jul 21 01:29:42.287520 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:42.287479 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-krnqh_9aeff916-a90a-49ae-9ac5-0e7a46d7f272/kube-multus/0.log" Jul 21 01:29:42.335880 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:42.335839 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-4nq5b_6cfb338e-19eb-45ee-a3de-ea7311469109/network-metrics-daemon/0.log" Jul 21 01:29:42.355596 ip-10-0-129-137 kubenswrapper[2580]: I0721 01:29:42.355571 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-4nq5b_6cfb338e-19eb-45ee-a3de-ea7311469109/kube-rbac-proxy/0.log"