Jul 22 00:24:48.800508 ip-10-0-139-192 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 22 00:24:48.800518 ip-10-0-139-192 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 22 00:24:48.800526 ip-10-0-139-192 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 22 00:24:48.800749 ip-10-0-139-192 systemd[1]: Failed to start Kubernetes Kubelet. Jul 22 00:24:59.049843 ip-10-0-139-192 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 22 00:24:59.049863 ip-10-0-139-192 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 9692f54a228c4c8bbde48dfcef74c81c -- Jul 22 00:26:53.351461 ip-10-0-139-192 systemd[1]: Starting Kubernetes Kubelet... Jul 22 00:26:53.651369 ip-10-0-139-192 kubenswrapper[2574]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 22 00:26:53.651369 ip-10-0-139-192 kubenswrapper[2574]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 22 00:26:53.651369 ip-10-0-139-192 kubenswrapper[2574]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 22 00:26:53.651369 ip-10-0-139-192 kubenswrapper[2574]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 22 00:26:53.651369 ip-10-0-139-192 kubenswrapper[2574]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 22 00:26:53.652140 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.652056 2574 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 22 00:26:53.658591 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658566 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 00:26:53.658591 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658585 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 00:26:53.658591 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658589 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 00:26:53.658591 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658593 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 00:26:53.658591 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658596 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 00:26:53.658591 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658599 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 00:26:53.658804 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658603 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 00:26:53.658804 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658606 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 00:26:53.658804 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658609 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 00:26:53.658804 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658612 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 00:26:53.658804 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658614 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 00:26:53.658804 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658617 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 00:26:53.658804 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658620 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 00:26:53.658804 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658622 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 00:26:53.658804 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658625 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 00:26:53.658804 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658628 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 00:26:53.658804 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658630 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 00:26:53.658804 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658633 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 00:26:53.658804 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658636 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 00:26:53.658804 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658639 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 00:26:53.658804 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658641 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 00:26:53.658804 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658644 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 00:26:53.658804 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658646 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 00:26:53.658804 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658649 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 00:26:53.658804 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658651 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 00:26:53.659266 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658654 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 00:26:53.659266 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658657 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 00:26:53.659266 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658660 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 00:26:53.659266 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658663 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 00:26:53.659266 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658665 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 00:26:53.659266 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658668 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 00:26:53.659266 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658670 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 00:26:53.659266 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658673 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 00:26:53.659266 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658676 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 00:26:53.659266 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658678 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 00:26:53.659266 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658681 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 00:26:53.659266 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658683 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 00:26:53.659266 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658686 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 00:26:53.659266 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658688 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 00:26:53.659266 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658692 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 00:26:53.659266 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658695 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 00:26:53.659266 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658698 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 00:26:53.659266 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658700 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 00:26:53.659266 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658703 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 00:26:53.659266 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658706 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 00:26:53.659785 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658708 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 00:26:53.659785 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658711 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 00:26:53.659785 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658713 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 00:26:53.659785 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658717 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 00:26:53.659785 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658722 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 00:26:53.659785 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658727 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 00:26:53.659785 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658731 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 00:26:53.659785 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658734 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 22 00:26:53.659785 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658737 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 00:26:53.659785 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658740 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 00:26:53.659785 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658743 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 00:26:53.659785 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658746 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 00:26:53.659785 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658748 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 00:26:53.659785 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658751 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 00:26:53.659785 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658753 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 00:26:53.659785 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658756 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 00:26:53.659785 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658759 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 00:26:53.659785 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658762 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 00:26:53.659785 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658764 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 00:26:53.660264 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658767 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 00:26:53.660264 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658769 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 00:26:53.660264 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658772 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 00:26:53.660264 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658775 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 00:26:53.660264 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658777 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 00:26:53.660264 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658781 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 00:26:53.660264 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658785 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 00:26:53.660264 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658789 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 00:26:53.660264 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658792 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 00:26:53.660264 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658794 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 00:26:53.660264 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658797 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 00:26:53.660264 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658800 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 00:26:53.660264 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658802 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 00:26:53.660264 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658805 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 00:26:53.660264 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658808 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 00:26:53.660264 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658812 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 00:26:53.660264 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658814 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 00:26:53.660264 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658818 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 00:26:53.660264 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658820 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 00:26:53.660264 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658824 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 00:26:53.660749 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658826 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 00:26:53.660749 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658829 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 00:26:53.660749 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658832 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 00:26:53.660749 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658834 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 00:26:53.660749 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658837 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 00:26:53.660749 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658839 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 00:26:53.660749 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658842 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 00:26:53.660749 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658844 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 00:26:53.660749 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658847 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 00:26:53.660749 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658850 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 00:26:53.660749 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658852 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 00:26:53.660749 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658855 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 00:26:53.660749 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658857 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 00:26:53.660749 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658860 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 00:26:53.660749 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.658863 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 00:26:53.660749 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659583 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 00:26:53.660749 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659590 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 00:26:53.660749 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659593 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 00:26:53.660749 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659596 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 00:26:53.660749 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659599 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 00:26:53.661236 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659602 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 00:26:53.661236 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659604 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 00:26:53.661236 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659607 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 00:26:53.661236 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659610 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 00:26:53.661236 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659613 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 00:26:53.661236 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659615 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 00:26:53.661236 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659618 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 00:26:53.661236 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659621 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 00:26:53.661236 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659623 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 00:26:53.661236 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659626 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 00:26:53.661236 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659629 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 00:26:53.661236 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659631 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 00:26:53.661236 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659634 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 00:26:53.661236 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659637 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 00:26:53.661236 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659639 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 00:26:53.661236 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659642 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 00:26:53.661236 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659645 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 00:26:53.661236 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659647 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 00:26:53.661236 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659650 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 00:26:53.661236 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659652 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 00:26:53.661733 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659654 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 00:26:53.661733 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659657 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 00:26:53.661733 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659661 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 00:26:53.661733 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659665 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 00:26:53.661733 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659668 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 00:26:53.661733 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659671 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 00:26:53.661733 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659673 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 00:26:53.661733 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659675 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 00:26:53.661733 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659678 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 00:26:53.661733 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659680 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 00:26:53.661733 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659683 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 00:26:53.661733 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659688 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 00:26:53.661733 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659691 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 00:26:53.661733 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659694 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 00:26:53.661733 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659696 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 00:26:53.661733 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659699 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 00:26:53.661733 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659703 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 00:26:53.661733 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659706 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 00:26:53.661733 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659708 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 00:26:53.662202 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659710 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 00:26:53.662202 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659713 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 00:26:53.662202 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659716 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 00:26:53.662202 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659718 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 00:26:53.662202 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659721 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 00:26:53.662202 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659723 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 00:26:53.662202 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659728 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 00:26:53.662202 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659733 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 00:26:53.662202 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659736 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 00:26:53.662202 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659738 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 00:26:53.662202 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659741 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 00:26:53.662202 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659744 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 00:26:53.662202 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659747 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 00:26:53.662202 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659750 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 00:26:53.662202 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659753 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 00:26:53.662202 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659756 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 00:26:53.662202 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659758 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 00:26:53.662202 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659761 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 00:26:53.662202 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659764 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 00:26:53.662651 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659766 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 00:26:53.662651 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659769 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 00:26:53.662651 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659772 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 00:26:53.662651 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659774 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 00:26:53.662651 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659778 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 00:26:53.662651 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659780 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 00:26:53.662651 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659782 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 00:26:53.662651 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659785 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 00:26:53.662651 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659787 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 00:26:53.662651 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659791 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 00:26:53.662651 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659793 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 00:26:53.662651 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659796 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 00:26:53.662651 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659799 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 00:26:53.662651 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659801 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 00:26:53.662651 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659804 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 00:26:53.662651 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659807 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 00:26:53.662651 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659809 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 00:26:53.662651 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659812 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 00:26:53.662651 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659814 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 00:26:53.663109 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659818 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 00:26:53.663109 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659822 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 22 00:26:53.663109 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659825 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 00:26:53.663109 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659828 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 00:26:53.663109 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659831 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 00:26:53.663109 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659834 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 00:26:53.663109 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659837 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 00:26:53.663109 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659840 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 00:26:53.663109 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659843 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 00:26:53.663109 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659845 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 00:26:53.663109 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659848 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 00:26:53.663109 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659850 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 00:26:53.663109 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659853 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 00:26:53.663109 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659856 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 00:26:53.663109 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659858 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 00:26:53.663109 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659861 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 00:26:53.663109 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.659864 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 00:26:53.663109 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660264 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 00:26:53.663109 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660270 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 00:26:53.663109 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660273 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 00:26:53.663623 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660277 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 00:26:53.663623 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660279 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 00:26:53.663623 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660282 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 00:26:53.663623 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660284 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 00:26:53.663623 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660287 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 00:26:53.663623 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660290 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 00:26:53.663623 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660293 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 00:26:53.663623 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660295 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 00:26:53.663623 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660298 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 00:26:53.663623 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660301 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 00:26:53.663623 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660303 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 00:26:53.663623 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660308 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 00:26:53.663623 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660311 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 00:26:53.663623 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660315 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 00:26:53.663623 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660318 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 00:26:53.663623 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660321 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 22 00:26:53.663623 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660323 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 00:26:53.663623 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660326 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 00:26:53.663623 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660328 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 00:26:53.663623 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660331 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 00:26:53.664106 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660333 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 00:26:53.664106 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660336 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 00:26:53.664106 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660338 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 00:26:53.664106 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660341 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 00:26:53.664106 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660343 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 00:26:53.664106 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660346 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 00:26:53.664106 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660349 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 00:26:53.664106 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660351 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 00:26:53.664106 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660354 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 00:26:53.664106 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660357 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 00:26:53.664106 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660361 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 00:26:53.664106 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660363 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 00:26:53.664106 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660365 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 00:26:53.664106 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660377 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 00:26:53.664106 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660379 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 00:26:53.664106 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660382 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 00:26:53.664106 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660385 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 00:26:53.664106 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660388 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 00:26:53.664106 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660391 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 00:26:53.664584 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660393 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 00:26:53.664584 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660397 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 00:26:53.664584 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660400 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 00:26:53.664584 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660402 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 00:26:53.664584 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660405 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 00:26:53.664584 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660407 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 00:26:53.664584 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660410 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 00:26:53.664584 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660413 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 00:26:53.664584 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660416 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 00:26:53.664584 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660418 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 00:26:53.664584 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660421 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 00:26:53.664584 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660423 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 00:26:53.664584 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660426 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 00:26:53.664584 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660428 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 00:26:53.664584 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660430 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 00:26:53.664584 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660433 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 00:26:53.664584 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660435 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 00:26:53.664584 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660439 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 00:26:53.664584 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660441 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 00:26:53.664584 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660444 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 00:26:53.665100 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660447 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 00:26:53.665100 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660450 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 00:26:53.665100 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660452 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 00:26:53.665100 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660455 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 00:26:53.665100 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660457 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 00:26:53.665100 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660460 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 00:26:53.665100 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660462 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 00:26:53.665100 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660464 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 00:26:53.665100 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660467 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 00:26:53.665100 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660469 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 00:26:53.665100 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660473 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 00:26:53.665100 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660476 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 00:26:53.665100 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660478 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 00:26:53.665100 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660481 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 00:26:53.665100 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660484 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 00:26:53.665100 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660486 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 00:26:53.665100 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660489 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 00:26:53.665100 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660491 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 00:26:53.665100 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660494 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 00:26:53.665565 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660496 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 00:26:53.665565 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660499 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 00:26:53.665565 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660502 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 00:26:53.665565 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660504 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 00:26:53.665565 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660507 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 00:26:53.665565 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660509 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 00:26:53.665565 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660512 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 00:26:53.665565 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660514 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 00:26:53.665565 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660517 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 00:26:53.665565 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660519 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 00:26:53.665565 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660522 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 00:26:53.665565 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660524 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 00:26:53.665565 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660530 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 00:26:53.665565 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660533 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 00:26:53.665565 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660535 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 00:26:53.665565 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660538 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 00:26:53.665565 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660540 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 00:26:53.665565 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.660542 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 00:26:53.665565 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660612 2574 flags.go:64] FLAG: --address="0.0.0.0" Jul 22 00:26:53.665565 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660619 2574 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 22 00:26:53.665565 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660627 2574 flags.go:64] FLAG: --anonymous-auth="true" Jul 22 00:26:53.666069 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660634 2574 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 22 00:26:53.666069 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660640 2574 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 22 00:26:53.666069 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660644 2574 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 22 00:26:53.666069 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660648 2574 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 22 00:26:53.666069 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660653 2574 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 22 00:26:53.666069 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660656 2574 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 22 00:26:53.666069 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660659 2574 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 22 00:26:53.666069 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660663 2574 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 22 00:26:53.666069 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660666 2574 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 22 00:26:53.666069 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660669 2574 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 22 00:26:53.666069 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660672 2574 flags.go:64] FLAG: --cgroup-root="" Jul 22 00:26:53.666069 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660675 2574 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 22 00:26:53.666069 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660678 2574 flags.go:64] FLAG: --client-ca-file="" Jul 22 00:26:53.666069 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660681 2574 flags.go:64] FLAG: --cloud-provider="external" Jul 22 00:26:53.666069 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660684 2574 flags.go:64] FLAG: --cluster-dns="[]" Jul 22 00:26:53.666069 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660688 2574 flags.go:64] FLAG: --cluster-domain="" Jul 22 00:26:53.666069 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660691 2574 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 22 00:26:53.666069 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660694 2574 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 22 00:26:53.666069 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660697 2574 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 22 00:26:53.666069 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660700 2574 flags.go:64] FLAG: --container-log-max-files="5" Jul 22 00:26:53.666069 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660704 2574 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 22 00:26:53.666069 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660707 2574 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 22 00:26:53.666069 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660710 2574 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 22 00:26:53.666651 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660714 2574 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 22 00:26:53.666651 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660717 2574 flags.go:64] FLAG: --contention-profiling="false" Jul 22 00:26:53.666651 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660720 2574 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 22 00:26:53.666651 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660723 2574 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 22 00:26:53.666651 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660726 2574 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 22 00:26:53.666651 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660729 2574 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 22 00:26:53.666651 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660733 2574 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 22 00:26:53.666651 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660736 2574 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 22 00:26:53.666651 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660739 2574 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 22 00:26:53.666651 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660742 2574 flags.go:64] FLAG: --enable-load-reader="false" Jul 22 00:26:53.666651 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660745 2574 flags.go:64] FLAG: --enable-server="true" Jul 22 00:26:53.666651 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660748 2574 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 22 00:26:53.666651 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660752 2574 flags.go:64] FLAG: --event-burst="100" Jul 22 00:26:53.666651 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660756 2574 flags.go:64] FLAG: --event-qps="50" Jul 22 00:26:53.666651 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660759 2574 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 22 00:26:53.666651 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660762 2574 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 22 00:26:53.666651 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660765 2574 flags.go:64] FLAG: --eviction-hard="" Jul 22 00:26:53.666651 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660769 2574 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 22 00:26:53.666651 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660772 2574 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 22 00:26:53.666651 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660774 2574 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 22 00:26:53.666651 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660778 2574 flags.go:64] FLAG: --eviction-soft="" Jul 22 00:26:53.666651 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660780 2574 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 22 00:26:53.666651 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660783 2574 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 22 00:26:53.666651 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660787 2574 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 22 00:26:53.666651 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660790 2574 flags.go:64] FLAG: --experimental-mounter-path="" Jul 22 00:26:53.667283 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660793 2574 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 22 00:26:53.667283 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660795 2574 flags.go:64] FLAG: --fail-swap-on="true" Jul 22 00:26:53.667283 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660799 2574 flags.go:64] FLAG: --feature-gates="" Jul 22 00:26:53.667283 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660802 2574 flags.go:64] FLAG: --file-check-frequency="20s" Jul 22 00:26:53.667283 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660805 2574 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 22 00:26:53.667283 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660808 2574 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 22 00:26:53.667283 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660811 2574 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 22 00:26:53.667283 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660816 2574 flags.go:64] FLAG: --healthz-port="10248" Jul 22 00:26:53.667283 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660819 2574 flags.go:64] FLAG: --help="false" Jul 22 00:26:53.667283 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660821 2574 flags.go:64] FLAG: --hostname-override="ip-10-0-139-192.ec2.internal" Jul 22 00:26:53.667283 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660824 2574 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 22 00:26:53.667283 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660827 2574 flags.go:64] FLAG: --http-check-frequency="20s" Jul 22 00:26:53.667283 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660830 2574 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 22 00:26:53.667283 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660833 2574 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 22 00:26:53.667283 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660837 2574 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 22 00:26:53.667283 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660840 2574 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 22 00:26:53.667283 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660842 2574 flags.go:64] FLAG: --image-service-endpoint="" Jul 22 00:26:53.667283 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660845 2574 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 22 00:26:53.667283 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660848 2574 flags.go:64] FLAG: --kube-api-burst="100" Jul 22 00:26:53.667283 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660850 2574 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 22 00:26:53.667283 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660853 2574 flags.go:64] FLAG: --kube-api-qps="50" Jul 22 00:26:53.667283 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660858 2574 flags.go:64] FLAG: --kube-reserved="" Jul 22 00:26:53.667283 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660861 2574 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 22 00:26:53.667283 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660863 2574 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 22 00:26:53.667896 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660866 2574 flags.go:64] FLAG: --kubelet-cgroups="" Jul 22 00:26:53.667896 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660869 2574 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 22 00:26:53.667896 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660872 2574 flags.go:64] FLAG: --lock-file="" Jul 22 00:26:53.667896 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660875 2574 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 22 00:26:53.667896 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660877 2574 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 22 00:26:53.667896 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660880 2574 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 22 00:26:53.667896 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660885 2574 flags.go:64] FLAG: --log-json-split-stream="false" Jul 22 00:26:53.667896 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660890 2574 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 22 00:26:53.667896 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660893 2574 flags.go:64] FLAG: --log-text-split-stream="false" Jul 22 00:26:53.667896 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660895 2574 flags.go:64] FLAG: --logging-format="text" Jul 22 00:26:53.667896 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660898 2574 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 22 00:26:53.667896 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660901 2574 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 22 00:26:53.667896 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660904 2574 flags.go:64] FLAG: --manifest-url="" Jul 22 00:26:53.667896 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660907 2574 flags.go:64] FLAG: --manifest-url-header="" Jul 22 00:26:53.667896 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660911 2574 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 22 00:26:53.667896 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660915 2574 flags.go:64] FLAG: --max-open-files="1000000" Jul 22 00:26:53.667896 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660920 2574 flags.go:64] FLAG: --max-pods="110" Jul 22 00:26:53.667896 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660923 2574 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 22 00:26:53.667896 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660926 2574 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 22 00:26:53.667896 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660928 2574 flags.go:64] FLAG: --memory-manager-policy="None" Jul 22 00:26:53.667896 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660931 2574 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 22 00:26:53.667896 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660934 2574 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 22 00:26:53.667896 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660937 2574 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 22 00:26:53.667896 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660940 2574 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 22 00:26:53.667896 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660948 2574 flags.go:64] FLAG: --node-status-max-images="50" Jul 22 00:26:53.668570 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660951 2574 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 22 00:26:53.668570 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660954 2574 flags.go:64] FLAG: --oom-score-adj="-999" Jul 22 00:26:53.668570 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660956 2574 flags.go:64] FLAG: --pod-cidr="" Jul 22 00:26:53.668570 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660959 2574 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 22 00:26:53.668570 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660966 2574 flags.go:64] FLAG: --pod-manifest-path="" Jul 22 00:26:53.668570 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660969 2574 flags.go:64] FLAG: --pod-max-pids="-1" Jul 22 00:26:53.668570 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660972 2574 flags.go:64] FLAG: --pods-per-core="0" Jul 22 00:26:53.668570 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660975 2574 flags.go:64] FLAG: --port="10250" Jul 22 00:26:53.668570 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660978 2574 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 22 00:26:53.668570 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660981 2574 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-02e5646b4b3ceb54e" Jul 22 00:26:53.668570 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660984 2574 flags.go:64] FLAG: --qos-reserved="" Jul 22 00:26:53.668570 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660987 2574 flags.go:64] FLAG: --read-only-port="10255" Jul 22 00:26:53.668570 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660990 2574 flags.go:64] FLAG: --register-node="true" Jul 22 00:26:53.668570 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660993 2574 flags.go:64] FLAG: --register-with-taints="" Jul 22 00:26:53.668570 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.660996 2574 flags.go:64] FLAG: --registry-burst="10" Jul 22 00:26:53.668570 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.661000 2574 flags.go:64] FLAG: --registry-qps="5" Jul 22 00:26:53.668570 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.661002 2574 flags.go:64] FLAG: --reserved-cpus="" Jul 22 00:26:53.668570 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.661005 2574 flags.go:64] FLAG: --reserved-memory="" Jul 22 00:26:53.668570 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.661009 2574 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 22 00:26:53.668570 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.661012 2574 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 22 00:26:53.668570 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.661015 2574 flags.go:64] FLAG: --rotate-certificates="false" Jul 22 00:26:53.668570 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.661017 2574 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 22 00:26:53.668570 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.661020 2574 flags.go:64] FLAG: --runonce="false" Jul 22 00:26:53.668570 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.661024 2574 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 22 00:26:53.668570 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.661027 2574 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 22 00:26:53.669188 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.661030 2574 flags.go:64] FLAG: --seccomp-default="false" Jul 22 00:26:53.669188 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.661033 2574 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 22 00:26:53.669188 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.661036 2574 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 22 00:26:53.669188 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.661038 2574 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 22 00:26:53.669188 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.661041 2574 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 22 00:26:53.669188 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.661044 2574 flags.go:64] FLAG: --storage-driver-password="root" Jul 22 00:26:53.669188 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.661047 2574 flags.go:64] FLAG: --storage-driver-secure="false" Jul 22 00:26:53.669188 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.661050 2574 flags.go:64] FLAG: --storage-driver-table="stats" Jul 22 00:26:53.669188 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.661053 2574 flags.go:64] FLAG: --storage-driver-user="root" Jul 22 00:26:53.669188 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.661056 2574 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 22 00:26:53.669188 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.661059 2574 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 22 00:26:53.669188 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.661063 2574 flags.go:64] FLAG: --system-cgroups="" Jul 22 00:26:53.669188 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.661066 2574 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 22 00:26:53.669188 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.661071 2574 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 22 00:26:53.669188 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.661074 2574 flags.go:64] FLAG: --tls-cert-file="" Jul 22 00:26:53.669188 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.661077 2574 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 22 00:26:53.669188 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.661081 2574 flags.go:64] FLAG: --tls-min-version="" Jul 22 00:26:53.669188 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.661084 2574 flags.go:64] FLAG: --tls-private-key-file="" Jul 22 00:26:53.669188 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.661086 2574 flags.go:64] FLAG: --topology-manager-policy="none" Jul 22 00:26:53.669188 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.661089 2574 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 22 00:26:53.669188 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.661092 2574 flags.go:64] FLAG: --topology-manager-scope="container" Jul 22 00:26:53.669188 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.661095 2574 flags.go:64] FLAG: --v="2" Jul 22 00:26:53.669188 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.661099 2574 flags.go:64] FLAG: --version="false" Jul 22 00:26:53.669188 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.661104 2574 flags.go:64] FLAG: --vmodule="" Jul 22 00:26:53.669188 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.661107 2574 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 22 00:26:53.669804 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.661111 2574 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 22 00:26:53.669804 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661243 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 00:26:53.669804 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661249 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 00:26:53.669804 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661253 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 00:26:53.669804 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661255 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 00:26:53.669804 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661260 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 00:26:53.669804 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661263 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 00:26:53.669804 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661265 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 00:26:53.669804 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661268 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 00:26:53.669804 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661271 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 00:26:53.669804 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661273 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 00:26:53.669804 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661276 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 00:26:53.669804 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661278 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 00:26:53.669804 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661281 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 00:26:53.669804 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661283 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 00:26:53.669804 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661286 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 00:26:53.669804 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661289 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 00:26:53.669804 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661291 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 00:26:53.669804 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661295 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 00:26:53.669804 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661298 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 00:26:53.670362 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661301 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 00:26:53.670362 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661304 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 00:26:53.670362 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661306 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 00:26:53.670362 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661309 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 00:26:53.670362 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661311 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 00:26:53.670362 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661314 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 00:26:53.670362 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661316 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 00:26:53.670362 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661319 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 00:26:53.670362 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661322 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 00:26:53.670362 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661324 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 00:26:53.670362 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661328 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 00:26:53.670362 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661333 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 00:26:53.670362 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661337 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 00:26:53.670362 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661340 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 00:26:53.670362 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661343 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 00:26:53.670362 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661346 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 00:26:53.670362 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661349 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 00:26:53.670362 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661353 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 00:26:53.670362 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661355 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 00:26:53.670832 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661358 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 00:26:53.670832 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661361 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 00:26:53.670832 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661363 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 00:26:53.670832 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661366 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 00:26:53.670832 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661369 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 00:26:53.670832 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661371 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 00:26:53.670832 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661374 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 00:26:53.670832 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661377 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 00:26:53.670832 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661379 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 00:26:53.670832 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661382 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 00:26:53.670832 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661386 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 00:26:53.670832 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661390 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 00:26:53.670832 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661393 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 00:26:53.670832 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661396 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 00:26:53.670832 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661399 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 00:26:53.670832 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661402 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 00:26:53.670832 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661404 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 00:26:53.670832 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661407 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 00:26:53.670832 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661409 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 00:26:53.671315 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661412 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 00:26:53.671315 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661414 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 00:26:53.671315 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661418 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 00:26:53.671315 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661420 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 00:26:53.671315 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661423 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 00:26:53.671315 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661425 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 22 00:26:53.671315 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661432 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 00:26:53.671315 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661435 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 00:26:53.671315 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661437 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 00:26:53.671315 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661440 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 00:26:53.671315 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661442 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 00:26:53.671315 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661446 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 00:26:53.671315 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661448 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 00:26:53.671315 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661451 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 00:26:53.671315 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661453 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 00:26:53.671315 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661456 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 00:26:53.671315 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661459 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 00:26:53.671315 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661461 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 00:26:53.671315 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661464 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 00:26:53.671315 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661466 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 00:26:53.671811 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661469 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 00:26:53.671811 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661472 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 00:26:53.671811 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661474 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 00:26:53.671811 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661477 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 00:26:53.671811 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661481 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 00:26:53.671811 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661484 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 00:26:53.671811 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661487 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 00:26:53.671811 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661489 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 00:26:53.671811 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661492 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 00:26:53.671811 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661494 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 00:26:53.671811 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661497 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 00:26:53.671811 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661499 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 00:26:53.671811 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661502 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 00:26:53.671811 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661504 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 00:26:53.671811 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661507 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 00:26:53.671811 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661510 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 00:26:53.671811 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661512 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 00:26:53.671811 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661515 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 00:26:53.671811 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661517 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 00:26:53.671811 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661520 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 00:26:53.672315 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661523 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 00:26:53.672315 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.661525 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 00:26:53.672315 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.662042 2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 22 00:26:53.672315 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.669796 2574 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 22 00:26:53.672315 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.669814 2574 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 22 00:26:53.672315 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.670426 2574 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 22 00:26:53.672315 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.670434 2574 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 22 00:26:53.672315 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670493 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 00:26:53.672315 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670498 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 00:26:53.672315 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670502 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 00:26:53.672315 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670505 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 00:26:53.672315 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670509 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 00:26:53.672315 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670513 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 00:26:53.672315 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670516 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 00:26:53.672315 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670519 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 00:26:53.672705 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670522 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 00:26:53.672705 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670525 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 00:26:53.672705 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670528 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 00:26:53.672705 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670531 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 00:26:53.672705 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670533 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 00:26:53.672705 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670536 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 00:26:53.672705 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670539 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 00:26:53.672705 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670541 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 00:26:53.672705 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670544 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 00:26:53.672705 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670547 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 00:26:53.672705 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670549 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 00:26:53.672705 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670552 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 00:26:53.672705 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670554 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 00:26:53.672705 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670557 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 00:26:53.672705 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670559 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 00:26:53.672705 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670562 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 00:26:53.672705 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670565 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 00:26:53.672705 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670567 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 00:26:53.672705 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670570 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 00:26:53.673163 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670574 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 00:26:53.673163 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670577 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 00:26:53.673163 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670582 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 00:26:53.673163 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670585 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 00:26:53.673163 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670587 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 00:26:53.673163 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670590 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 00:26:53.673163 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670592 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 00:26:53.673163 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670595 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 00:26:53.673163 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670598 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 00:26:53.673163 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670600 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 00:26:53.673163 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670603 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 00:26:53.673163 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670605 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 00:26:53.673163 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670608 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 00:26:53.673163 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670610 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 00:26:53.673163 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670613 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 00:26:53.673163 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670615 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 00:26:53.673163 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670618 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 00:26:53.673163 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670620 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 00:26:53.673163 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670622 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 00:26:53.673163 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670625 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 00:26:53.673717 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670628 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 00:26:53.673717 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670630 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 00:26:53.673717 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670633 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 00:26:53.673717 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670635 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 00:26:53.673717 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670638 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 00:26:53.673717 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670640 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 00:26:53.673717 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670644 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 00:26:53.673717 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670648 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 00:26:53.673717 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670651 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 00:26:53.673717 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670654 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 00:26:53.673717 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670657 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 00:26:53.673717 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670660 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 00:26:53.673717 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670662 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 00:26:53.673717 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670665 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 00:26:53.673717 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670669 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 00:26:53.673717 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670672 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 00:26:53.673717 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670675 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 00:26:53.673717 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670677 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 00:26:53.673717 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670680 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 00:26:53.673717 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670682 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 00:26:53.674296 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670685 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 00:26:53.674296 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670687 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 00:26:53.674296 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670689 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 00:26:53.674296 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670692 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 00:26:53.674296 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670696 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 00:26:53.674296 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670699 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 00:26:53.674296 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670702 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 00:26:53.674296 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670705 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 00:26:53.674296 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670708 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 00:26:53.674296 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670711 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 00:26:53.674296 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670713 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 00:26:53.674296 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670716 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 00:26:53.674296 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670719 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 22 00:26:53.674296 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670722 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 00:26:53.674296 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670724 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 00:26:53.674296 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670727 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 00:26:53.674296 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670730 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 00:26:53.674296 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670733 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 00:26:53.674296 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670735 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 00:26:53.674296 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670738 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 00:26:53.674831 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670740 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 00:26:53.674831 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670743 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 00:26:53.674831 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670745 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 00:26:53.674831 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670748 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 00:26:53.674831 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670751 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 00:26:53.674831 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670754 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 00:26:53.674831 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670757 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 00:26:53.674831 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670760 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 00:26:53.674831 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670767 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 00:26:53.674831 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670769 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 00:26:53.674831 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670772 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 00:26:53.674831 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670775 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 00:26:53.674831 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.670780 2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 22 00:26:53.674831 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670898 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 00:26:53.674831 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670903 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 00:26:53.675214 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670906 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 00:26:53.675214 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670908 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 22 00:26:53.675214 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670912 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 00:26:53.675214 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670915 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 00:26:53.675214 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670917 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 00:26:53.675214 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670920 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 00:26:53.675214 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670922 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 00:26:53.675214 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670925 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 00:26:53.675214 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670928 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 00:26:53.675214 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670931 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 00:26:53.675214 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670934 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 00:26:53.675214 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670936 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 00:26:53.675214 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670940 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 00:26:53.675214 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670944 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 00:26:53.675214 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670946 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 00:26:53.675214 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670950 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 00:26:53.675214 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670952 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 00:26:53.675214 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670955 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 00:26:53.675214 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670957 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 00:26:53.675672 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670959 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 00:26:53.675672 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670962 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 00:26:53.675672 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670971 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 00:26:53.675672 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670974 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 00:26:53.675672 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670977 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 00:26:53.675672 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670980 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 00:26:53.675672 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670983 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 00:26:53.675672 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670989 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 00:26:53.675672 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670991 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 00:26:53.675672 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670994 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 00:26:53.675672 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670996 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 00:26:53.675672 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.670999 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 00:26:53.675672 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.671001 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 00:26:53.675672 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.671004 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 00:26:53.675672 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.671006 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 00:26:53.675672 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.671009 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 00:26:53.675672 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.671011 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 00:26:53.675672 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.671013 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 00:26:53.675672 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.671016 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 00:26:53.675672 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.671018 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 00:26:53.676148 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.671021 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 00:26:53.676148 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.671023 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 00:26:53.676148 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.671026 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 00:26:53.676148 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.671028 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 00:26:53.676148 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.671031 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 00:26:53.676148 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.671033 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 00:26:53.676148 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.671035 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 00:26:53.676148 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.671038 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 00:26:53.676148 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.671040 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 00:26:53.676148 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.671043 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 00:26:53.676148 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.671045 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 00:26:53.676148 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.671047 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 00:26:53.676148 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.671050 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 00:26:53.676148 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.671052 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 00:26:53.676148 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.671055 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 00:26:53.676148 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.671057 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 00:26:53.676148 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.671060 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 00:26:53.676148 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.671063 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 00:26:53.676148 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.671065 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 00:26:53.676148 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.671068 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 00:26:53.676694 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.671076 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 00:26:53.676694 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.671079 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 00:26:53.676694 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.671082 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 00:26:53.676694 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.671084 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 00:26:53.676694 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.671087 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 00:26:53.676694 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.671089 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 00:26:53.676694 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.671092 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 00:26:53.676694 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.671094 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 00:26:53.676694 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.671096 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 00:26:53.676694 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.671099 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 00:26:53.676694 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.671101 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 00:26:53.676694 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.671104 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 00:26:53.676694 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.671106 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 00:26:53.676694 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.671109 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 00:26:53.676694 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.671111 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 00:26:53.676694 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.671114 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 00:26:53.676694 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.671116 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 00:26:53.676694 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.671118 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 00:26:53.676694 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.671121 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 00:26:53.677141 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.671123 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 00:26:53.677141 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.671126 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 00:26:53.677141 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.671128 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 00:26:53.677141 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.671131 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 00:26:53.677141 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.671133 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 00:26:53.677141 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.671135 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 00:26:53.677141 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.671138 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 00:26:53.677141 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.671140 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 00:26:53.677141 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.671143 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 00:26:53.677141 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.671146 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 00:26:53.677141 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.671149 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 00:26:53.677141 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.671151 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 00:26:53.677141 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.671153 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 00:26:53.677141 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.671157 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 00:26:53.677141 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.671177 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 00:26:53.677141 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.671180 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 00:26:53.677141 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.671183 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 00:26:53.677141 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.671187 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 00:26:53.677141 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:53.671190 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 00:26:53.677647 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.671196 2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 22 00:26:53.677647 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.671316 2574 server.go:962] "Client rotation is on, will bootstrap in background" Jul 22 00:26:53.677647 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.673377 2574 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 22 00:26:53.677647 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.674477 2574 server.go:1019] "Starting client certificate rotation" Jul 22 00:26:53.677647 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.674582 2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 22 00:26:53.677647 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.674622 2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 22 00:26:53.677647 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.676557 2574 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 22 00:26:53.678464 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.677743 2574 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 22 00:26:53.688639 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.688621 2574 log.go:25] "Validated CRI v1 runtime API" Jul 22 00:26:53.690646 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.690632 2574 log.go:25] "Validated CRI v1 image API" Jul 22 00:26:53.691457 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.691433 2574 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 22 00:26:53.693665 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.693639 2574 fs.go:135] Filesystem UUIDs: map[5617559e-2571-4279-9f68-496d35a0050a:/dev/nvme0n1p4 562b5ba9-fc55-4648-a1a9-48299e5ed5fc:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Jul 22 00:26:53.693717 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.693667 2574 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 22 00:26:53.699733 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.699624 2574 manager.go:216] Machine: {Timestamp:2026-07-22 00:26:53.698475346 +0000 UTC m=+0.253666006 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3103356 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2aa4d0e4decc3cfcbd8e6234ad02da SystemUUID:ec2aa4d0-e4de-cc3c-fcbd-8e6234ad02da BootID:9692f54a-228c-4c8b-bde4-8dfcef74c81c Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:86:f8:cf:51:d9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:86:f8:cf:51:d9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:c2:ff:9e:17:8e:57 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 22 00:26:53.699733 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.699726 2574 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 22 00:26:53.699855 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.699844 2574 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 22 00:26:53.702599 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.702569 2574 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 22 00:26:53.702735 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.702602 2574 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-192.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 22 00:26:53.703326 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.703316 2574 topology_manager.go:138] "Creating topology manager with none policy" Jul 22 00:26:53.703358 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.703328 2574 container_manager_linux.go:309] "Creating device plugin manager" Jul 22 00:26:53.703358 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.703335 2574 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 22 00:26:53.703845 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.703835 2574 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 22 00:26:53.705084 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.705073 2574 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 22 00:26:53.708047 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.708032 2574 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 22 00:26:53.708903 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.708892 2574 state_mem.go:36] "Initialized new in-memory state store" Jul 22 00:26:53.709026 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.709016 2574 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 22 00:26:53.709057 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.709044 2574 kubelet.go:486] "Attempting to sync node with API server" Jul 22 00:26:53.709094 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.709060 2574 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 22 00:26:53.709094 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.709072 2574 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 22 00:26:53.709094 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.709083 2574 kubelet.go:398] "Adding apiserver pod source" Jul 22 00:26:53.709094 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.709092 2574 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 22 00:26:53.709501 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.709489 2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 22 00:26:53.709566 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.709512 2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 22 00:26:53.711427 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.711411 2574 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 22 00:26:53.713254 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.713238 2574 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 22 00:26:53.713340 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.713276 2574 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 22 00:26:53.714314 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.714301 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 22 00:26:53.714385 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.714322 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 22 00:26:53.714385 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.714332 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 22 00:26:53.714385 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.714340 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 22 00:26:53.714522 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.714427 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 22 00:26:53.714522 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.714440 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 22 00:26:53.714522 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.714453 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 22 00:26:53.714522 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.714462 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 22 00:26:53.714522 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.714472 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 22 00:26:53.714522 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.714481 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 22 00:26:53.714522 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.714493 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 22 00:26:53.714522 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.714502 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 22 00:26:53.715315 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.715303 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 22 00:26:53.715366 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.715318 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 22 00:26:53.718447 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.718428 2574 server.go:1268] "Started kubelet" Jul 22 00:26:53.718574 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.718546 2574 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 22 00:26:53.718822 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.718731 2574 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 22 00:26:53.718955 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.718925 2574 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 22 00:26:53.720773 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:26:53.720571 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 22 00:26:53.720773 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.720656 2574 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-192.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 22 00:26:53.720773 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.720763 2574 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 22 00:26:53.720985 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:26:53.720878 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-192.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 22 00:26:53.721635 ip-10-0-139-192 systemd[1]: Started Kubernetes Kubelet. Jul 22 00:26:53.722830 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.722130 2574 server.go:310] "Adding debug handlers to kubelet server" Jul 22 00:26:53.723972 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:26:53.723956 2574 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 22 00:26:53.727500 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:26:53.726784 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-192.ec2.internal.18c475044f6cc8ce default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-192.ec2.internal,UID:ip-10-0-139-192.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-192.ec2.internal,},FirstTimestamp:2026-07-22 00:26:53.718407374 +0000 UTC m=+0.273598032,LastTimestamp:2026-07-22 00:26:53.718407374 +0000 UTC m=+0.273598032,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-192.ec2.internal,}" Jul 22 00:26:53.728067 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.728052 2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-k4hf6" Jul 22 00:26:53.729053 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.729031 2574 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 22 00:26:53.729053 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.729044 2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 22 00:26:53.729218 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:26:53.729100 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-192.ec2.internal\" not found" Jul 22 00:26:53.729545 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.729529 2574 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 22 00:26:53.729545 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.729527 2574 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 22 00:26:53.729665 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.729555 2574 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 22 00:26:53.729723 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.729709 2574 reconstruct.go:97] "Volume reconstruction finished" Jul 22 00:26:53.729723 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.729718 2574 reconciler.go:29] "Reconciler: start to sync state" Jul 22 00:26:53.730007 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.729989 2574 factory.go:153] Registering CRI-O factory Jul 22 00:26:53.730088 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.730030 2574 factory.go:223] Registration of the crio container factory successfully Jul 22 00:26:53.730088 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.730079 2574 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 22 00:26:53.730254 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.730098 2574 factory.go:55] Registering systemd factory Jul 22 00:26:53.730254 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.730106 2574 factory.go:223] Registration of the systemd container factory successfully Jul 22 00:26:53.730254 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.730126 2574 factory.go:103] Registering Raw factory Jul 22 00:26:53.730254 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.730139 2574 manager.go:1196] Started watching for new ooms in manager Jul 22 00:26:53.730501 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.730486 2574 manager.go:349] Starting recovery of all containers Jul 22 00:26:53.739628 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:26:53.739599 2574 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-192.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 22 00:26:53.739703 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:26:53.739619 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 22 00:26:53.741872 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.741850 2574 manager.go:354] Recovery completed Jul 22 00:26:53.744929 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.744911 2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-k4hf6" Jul 22 00:26:53.746812 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.746799 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 00:26:53.748553 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.748539 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-192.ec2.internal" event="NodeHasSufficientMemory" Jul 22 00:26:53.748616 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.748569 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-192.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 00:26:53.748616 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.748585 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-192.ec2.internal" event="NodeHasSufficientPID" Jul 22 00:26:53.749097 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.749082 2574 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 22 00:26:53.749097 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.749096 2574 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 22 00:26:53.749193 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.749111 2574 state_mem.go:36] "Initialized new in-memory state store" Jul 22 00:26:53.749193 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.749112 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 00:26:53.749982 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.749967 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-192.ec2.internal" event="NodeHasSufficientMemory" Jul 22 00:26:53.750043 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.749991 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-192.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 00:26:53.750043 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.750000 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-192.ec2.internal" event="NodeHasSufficientPID" Jul 22 00:26:53.751147 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.751135 2574 policy_none.go:49] "None policy: Start" Jul 22 00:26:53.751222 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.751154 2574 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 22 00:26:53.751222 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.751183 2574 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 22 00:26:53.753180 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.753156 2574 policy_none.go:47] "Start" Jul 22 00:26:53.791912 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.791724 2574 manager.go:337] "Starting Device Plugin manager" Jul 22 00:26:53.792016 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:26:53.791937 2574 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 22 00:26:53.792016 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.791949 2574 server.go:87] "Starting device plugin registration server" Jul 22 00:26:53.792194 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.792182 2574 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 22 00:26:53.792251 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.792199 2574 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 22 00:26:53.792317 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.792297 2574 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 22 00:26:53.792611 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.792404 2574 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 22 00:26:53.792611 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.792417 2574 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 22 00:26:53.792885 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:26:53.792866 2574 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 22 00:26:53.792946 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:26:53.792921 2574 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-192.ec2.internal\" not found" Jul 22 00:26:53.882500 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.882459 2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 22 00:26:53.884767 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.883757 2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 22 00:26:53.884767 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.883792 2574 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 22 00:26:53.884767 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.883818 2574 kubelet.go:2448] "Starting kubelet main sync loop" Jul 22 00:26:53.884767 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:26:53.883851 2574 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 22 00:26:53.886940 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.886923 2574 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 00:26:53.892971 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.892958 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 00:26:53.893801 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.893787 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-192.ec2.internal" event="NodeHasSufficientMemory" Jul 22 00:26:53.893859 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.893819 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-192.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 00:26:53.893859 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.893831 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-192.ec2.internal" event="NodeHasSufficientPID" Jul 22 00:26:53.893859 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.893856 2574 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-192.ec2.internal" Jul 22 00:26:53.900401 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:53.900388 2574 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-192.ec2.internal" Jul 22 00:26:53.900447 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:26:53.900414 2574 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-192.ec2.internal\": node \"ip-10-0-139-192.ec2.internal\" not found" Jul 22 00:26:53.913150 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:26:53.913101 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-192.ec2.internal\" not found" Jul 22 00:26:55.182690 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:55.182654 2574 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-192.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-192.ec2.internal"] Jul 22 00:26:55.182856 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:55.182732 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 00:26:55.183822 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:55.183804 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-192.ec2.internal" event="NodeHasSufficientMemory" Jul 22 00:26:55.183927 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:55.183834 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-192.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 00:26:55.183927 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:55.183844 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-192.ec2.internal" event="NodeHasSufficientPID" Jul 22 00:26:55.186230 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:55.186218 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 00:26:55.186346 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:55.186330 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-192.ec2.internal" Jul 22 00:26:55.186385 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:55.186377 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 00:26:55.186975 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:55.186957 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-192.ec2.internal" event="NodeHasSufficientMemory" Jul 22 00:26:55.187082 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:55.186957 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-192.ec2.internal" event="NodeHasSufficientMemory" Jul 22 00:26:55.187082 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:55.187020 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-192.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 00:26:55.187082 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:55.187039 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-192.ec2.internal" event="NodeHasSufficientPID" Jul 22 00:26:55.187082 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:55.186987 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-192.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 00:26:55.187082 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:55.187070 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-192.ec2.internal" event="NodeHasSufficientPID" Jul 22 00:26:55.189344 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:55.189328 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-192.ec2.internal" Jul 22 00:26:55.189428 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:55.189352 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 00:26:55.190121 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:55.190106 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-192.ec2.internal" event="NodeHasSufficientMemory" Jul 22 00:26:55.190200 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:55.190132 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-192.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 00:26:55.190200 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:55.190143 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-192.ec2.internal" event="NodeHasSufficientPID" Jul 22 00:26:55.207211 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:26:55.207190 2574 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-192.ec2.internal\" not found" node="ip-10-0-139-192.ec2.internal" Jul 22 00:26:55.210738 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:26:55.210719 2574 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-192.ec2.internal\" not found" node="ip-10-0-139-192.ec2.internal" Jul 22 00:26:55.211713 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:26:55.211698 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-192.ec2.internal\" not found" Jul 22 00:26:55.229783 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:55.229764 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2951ff349dfaeb74342ef57c90cc46ad-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-192.ec2.internal\" (UID: \"2951ff349dfaeb74342ef57c90cc46ad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-192.ec2.internal" Jul 22 00:26:55.229878 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:55.229792 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2951ff349dfaeb74342ef57c90cc46ad-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-192.ec2.internal\" (UID: \"2951ff349dfaeb74342ef57c90cc46ad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-192.ec2.internal" Jul 22 00:26:55.229878 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:55.229817 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5c5847e7e75764b70e99faa3e06c32f2-config\") pod \"kube-apiserver-proxy-ip-10-0-139-192.ec2.internal\" (UID: \"5c5847e7e75764b70e99faa3e06c32f2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-192.ec2.internal" Jul 22 00:26:55.312841 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:26:55.312795 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-192.ec2.internal\" not found" Jul 22 00:26:55.330144 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:55.330115 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5c5847e7e75764b70e99faa3e06c32f2-config\") pod \"kube-apiserver-proxy-ip-10-0-139-192.ec2.internal\" (UID: \"5c5847e7e75764b70e99faa3e06c32f2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-192.ec2.internal" Jul 22 00:26:55.330225 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:55.330147 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2951ff349dfaeb74342ef57c90cc46ad-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-192.ec2.internal\" (UID: \"2951ff349dfaeb74342ef57c90cc46ad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-192.ec2.internal" Jul 22 00:26:55.330225 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:55.330165 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2951ff349dfaeb74342ef57c90cc46ad-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-192.ec2.internal\" (UID: \"2951ff349dfaeb74342ef57c90cc46ad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-192.ec2.internal" Jul 22 00:26:55.330225 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:55.330206 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2951ff349dfaeb74342ef57c90cc46ad-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-192.ec2.internal\" (UID: \"2951ff349dfaeb74342ef57c90cc46ad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-192.ec2.internal" Jul 22 00:26:55.330369 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:55.330238 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2951ff349dfaeb74342ef57c90cc46ad-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-192.ec2.internal\" (UID: \"2951ff349dfaeb74342ef57c90cc46ad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-192.ec2.internal" Jul 22 00:26:55.330369 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:55.330238 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5c5847e7e75764b70e99faa3e06c32f2-config\") pod \"kube-apiserver-proxy-ip-10-0-139-192.ec2.internal\" (UID: \"5c5847e7e75764b70e99faa3e06c32f2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-192.ec2.internal" Jul 22 00:26:55.413659 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:26:55.413557 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-192.ec2.internal\" not found" Jul 22 00:26:55.510087 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:55.510045 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-192.ec2.internal" Jul 22 00:26:55.513254 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:55.513227 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-192.ec2.internal" Jul 22 00:26:55.513787 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:26:55.513771 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-192.ec2.internal\" not found" Jul 22 00:26:55.614874 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:26:55.614824 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-192.ec2.internal\" not found" Jul 22 00:26:55.715438 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:26:55.715367 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-192.ec2.internal\" not found" Jul 22 00:26:55.816003 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:26:55.815954 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-192.ec2.internal\" not found" Jul 22 00:26:55.872441 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:55.872408 2574 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 22 00:26:55.872870 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:55.872566 2574 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 22 00:26:55.916890 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:26:55.916853 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-192.ec2.internal\" not found" Jul 22 00:26:55.949123 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:55.949082 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-21 00:21:54 +0000 UTC" deadline="2028-03-28 06:01:13.428906141 +0000 UTC" Jul 22 00:26:55.949123 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:55.949113 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14765h34m17.479796556s" Jul 22 00:26:56.017502 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:26:56.017421 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-192.ec2.internal\" not found" Jul 22 00:26:56.117882 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:26:56.117850 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-192.ec2.internal\" not found" Jul 22 00:26:56.142213 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.142192 2574 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 00:26:56.218593 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:26:56.218554 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-192.ec2.internal\" not found" Jul 22 00:26:56.319219 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:26:56.319129 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-192.ec2.internal\" not found" Jul 22 00:26:56.338083 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.338058 2574 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 00:26:56.399922 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.399884 2574 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 00:26:56.427975 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.427938 2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-192.ec2.internal" Jul 22 00:26:56.437356 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.437329 2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 22 00:26:56.438185 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.438166 2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-192.ec2.internal" Jul 22 00:26:56.447802 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.447783 2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 22 00:26:56.533706 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:56.533665 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2951ff349dfaeb74342ef57c90cc46ad.slice/crio-3c8ce25c56ac3afd54c31f5c6138becf8e10773cb52e0012eb32af3024c68689 WatchSource:0}: Error finding container 3c8ce25c56ac3afd54c31f5c6138becf8e10773cb52e0012eb32af3024c68689: Status 404 returned error can't find the container with id 3c8ce25c56ac3afd54c31f5c6138becf8e10773cb52e0012eb32af3024c68689 Jul 22 00:26:56.534131 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:56.534112 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c5847e7e75764b70e99faa3e06c32f2.slice/crio-946bcaa8022149efc256e6323c0482d1b7dee2eef1332d03cab0cd294770a89c WatchSource:0}: Error finding container 946bcaa8022149efc256e6323c0482d1b7dee2eef1332d03cab0cd294770a89c: Status 404 returned error can't find the container with id 946bcaa8022149efc256e6323c0482d1b7dee2eef1332d03cab0cd294770a89c Jul 22 00:26:56.538130 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.538113 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 00:26:56.909523 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.909485 2574 apiserver.go:52] "Watching apiserver" Jul 22 00:26:56.915928 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.915907 2574 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 22 00:26:56.918147 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.918126 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjxn9","openshift-cluster-node-tuning-operator/tuned-hsv57","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-192.ec2.internal","openshift-multus/multus-mhnl2","openshift-multus/network-metrics-daemon-5xz72","openshift-dns/node-resolver-zqvbx","openshift-image-registry/node-ca-7wkvg","openshift-multus/multus-additional-cni-plugins-v2ksf","openshift-network-diagnostics/network-check-target-6k9fm","openshift-network-operator/iptables-alerter-ctbfz","openshift-ovn-kubernetes/ovnkube-node-hr5fn","kube-system/global-pull-secret-syncer-sl6b2","kube-system/konnectivity-agent-zk4rp","kube-system/kube-apiserver-proxy-ip-10-0-139-192.ec2.internal"] Jul 22 00:26:56.921816 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.921796 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7wkvg" Jul 22 00:26:56.923844 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.923825 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zk4rp" Jul 22 00:26:56.924560 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.924541 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-sprbv\"" Jul 22 00:26:56.924674 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.924612 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 22 00:26:56.924674 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.924654 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 22 00:26:56.924773 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.924701 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 22 00:26:56.926020 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.925999 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-v2ksf" Jul 22 00:26:56.926100 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.926072 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 22 00:26:56.926275 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.926149 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-9wdn6\"" Jul 22 00:26:56.926525 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.926508 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 22 00:26:56.927918 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.927905 2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 22 00:26:56.928324 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.928298 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-mhnl2" Jul 22 00:26:56.928409 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.928334 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 22 00:26:56.928409 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.928359 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 22 00:26:56.928640 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.928624 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-thkc2\"" Jul 22 00:26:56.928716 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.928660 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 22 00:26:56.928716 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.928659 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 22 00:26:56.928716 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.928706 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 22 00:26:56.930461 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.930444 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5xz72" Jul 22 00:26:56.930556 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:26:56.930527 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5xz72" podUID="f36a57f8-6e88-43d3-b572-8adf3b0ddb7c" Jul 22 00:26:56.930668 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.930650 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 22 00:26:56.930732 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.930664 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-dvpfm\"" Jul 22 00:26:56.932700 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.932683 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zqvbx" Jul 22 00:26:56.934368 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.934354 2574 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 00:26:56.934948 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.934930 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-spqlp\"" Jul 22 00:26:56.935043 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.934966 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 22 00:26:56.935043 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.934933 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 22 00:26:56.935043 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.935031 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6k9fm" Jul 22 00:26:56.935183 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:26:56.935101 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6k9fm" podUID="bbfe455c-9568-48ba-8007-92b4142b845f" Jul 22 00:26:56.937296 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.937277 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjxn9" Jul 22 00:26:56.938458 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.938399 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/077d5538-1eab-48a5-97f9-3a623a4b63c9-cnibin\") pod \"multus-additional-cni-plugins-v2ksf\" (UID: \"077d5538-1eab-48a5-97f9-3a623a4b63c9\") " pod="openshift-multus/multus-additional-cni-plugins-v2ksf" Jul 22 00:26:56.938458 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.938446 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9b9207f1-4d6c-447c-9b24-c8cba3a2f21e-host-var-lib-cni-multus\") pod \"multus-mhnl2\" (UID: \"9b9207f1-4d6c-447c-9b24-c8cba3a2f21e\") " pod="openshift-multus/multus-mhnl2" Jul 22 00:26:56.938623 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.938487 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njshs\" (UniqueName: \"kubernetes.io/projected/9b9207f1-4d6c-447c-9b24-c8cba3a2f21e-kube-api-access-njshs\") pod \"multus-mhnl2\" (UID: \"9b9207f1-4d6c-447c-9b24-c8cba3a2f21e\") " pod="openshift-multus/multus-mhnl2" Jul 22 00:26:56.938623 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.938522 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsrv7\" (UniqueName: \"kubernetes.io/projected/44c9d8f9-4c85-4dfa-8572-2feb0f653720-kube-api-access-fsrv7\") pod \"node-ca-7wkvg\" (UID: \"44c9d8f9-4c85-4dfa-8572-2feb0f653720\") " pod="openshift-image-registry/node-ca-7wkvg" Jul 22 00:26:56.938623 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.938552 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/44c9d8f9-4c85-4dfa-8572-2feb0f653720-host\") pod \"node-ca-7wkvg\" (UID: \"44c9d8f9-4c85-4dfa-8572-2feb0f653720\") " pod="openshift-image-registry/node-ca-7wkvg" Jul 22 00:26:56.938623 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.938579 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d2fac953-0d0d-48cd-be69-240d4d16dd49-agent-certs\") pod \"konnectivity-agent-zk4rp\" (UID: \"d2fac953-0d0d-48cd-be69-240d4d16dd49\") " pod="kube-system/konnectivity-agent-zk4rp" Jul 22 00:26:56.938832 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.938627 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/077d5538-1eab-48a5-97f9-3a623a4b63c9-os-release\") pod \"multus-additional-cni-plugins-v2ksf\" (UID: \"077d5538-1eab-48a5-97f9-3a623a4b63c9\") " pod="openshift-multus/multus-additional-cni-plugins-v2ksf" Jul 22 00:26:56.938832 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.938658 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9b9207f1-4d6c-447c-9b24-c8cba3a2f21e-cnibin\") pod \"multus-mhnl2\" (UID: \"9b9207f1-4d6c-447c-9b24-c8cba3a2f21e\") " pod="openshift-multus/multus-mhnl2" Jul 22 00:26:56.938832 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.938690 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/077d5538-1eab-48a5-97f9-3a623a4b63c9-cni-binary-copy\") pod \"multus-additional-cni-plugins-v2ksf\" (UID: \"077d5538-1eab-48a5-97f9-3a623a4b63c9\") " pod="openshift-multus/multus-additional-cni-plugins-v2ksf" Jul 22 00:26:56.938832 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.938724 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/077d5538-1eab-48a5-97f9-3a623a4b63c9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v2ksf\" (UID: \"077d5538-1eab-48a5-97f9-3a623a4b63c9\") " pod="openshift-multus/multus-additional-cni-plugins-v2ksf" Jul 22 00:26:56.938832 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.938758 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9b9207f1-4d6c-447c-9b24-c8cba3a2f21e-cni-binary-copy\") pod \"multus-mhnl2\" (UID: \"9b9207f1-4d6c-447c-9b24-c8cba3a2f21e\") " pod="openshift-multus/multus-mhnl2" Jul 22 00:26:56.938832 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.938796 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9b9207f1-4d6c-447c-9b24-c8cba3a2f21e-multus-socket-dir-parent\") pod \"multus-mhnl2\" (UID: \"9b9207f1-4d6c-447c-9b24-c8cba3a2f21e\") " pod="openshift-multus/multus-mhnl2" Jul 22 00:26:56.939116 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.938841 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9b9207f1-4d6c-447c-9b24-c8cba3a2f21e-host-run-netns\") pod \"multus-mhnl2\" (UID: \"9b9207f1-4d6c-447c-9b24-c8cba3a2f21e\") " pod="openshift-multus/multus-mhnl2" Jul 22 00:26:56.939116 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.938874 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9b9207f1-4d6c-447c-9b24-c8cba3a2f21e-host-run-multus-certs\") pod \"multus-mhnl2\" (UID: \"9b9207f1-4d6c-447c-9b24-c8cba3a2f21e\") " pod="openshift-multus/multus-mhnl2" Jul 22 00:26:56.939116 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.938907 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d2fac953-0d0d-48cd-be69-240d4d16dd49-konnectivity-ca\") pod \"konnectivity-agent-zk4rp\" (UID: \"d2fac953-0d0d-48cd-be69-240d4d16dd49\") " pod="kube-system/konnectivity-agent-zk4rp" Jul 22 00:26:56.939116 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.938932 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/077d5538-1eab-48a5-97f9-3a623a4b63c9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v2ksf\" (UID: \"077d5538-1eab-48a5-97f9-3a623a4b63c9\") " pod="openshift-multus/multus-additional-cni-plugins-v2ksf" Jul 22 00:26:56.939116 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.939008 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9b9207f1-4d6c-447c-9b24-c8cba3a2f21e-multus-cni-dir\") pod \"multus-mhnl2\" (UID: \"9b9207f1-4d6c-447c-9b24-c8cba3a2f21e\") " pod="openshift-multus/multus-mhnl2" Jul 22 00:26:56.939116 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.939042 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9b9207f1-4d6c-447c-9b24-c8cba3a2f21e-hostroot\") pod \"multus-mhnl2\" (UID: \"9b9207f1-4d6c-447c-9b24-c8cba3a2f21e\") " pod="openshift-multus/multus-mhnl2" Jul 22 00:26:56.939116 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.939074 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/077d5538-1eab-48a5-97f9-3a623a4b63c9-system-cni-dir\") pod \"multus-additional-cni-plugins-v2ksf\" (UID: \"077d5538-1eab-48a5-97f9-3a623a4b63c9\") " pod="openshift-multus/multus-additional-cni-plugins-v2ksf" Jul 22 00:26:56.939116 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.939102 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8mzm\" (UniqueName: \"kubernetes.io/projected/077d5538-1eab-48a5-97f9-3a623a4b63c9-kube-api-access-d8mzm\") pod \"multus-additional-cni-plugins-v2ksf\" (UID: \"077d5538-1eab-48a5-97f9-3a623a4b63c9\") " pod="openshift-multus/multus-additional-cni-plugins-v2ksf" Jul 22 00:26:56.939453 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.939131 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9b9207f1-4d6c-447c-9b24-c8cba3a2f21e-os-release\") pod \"multus-mhnl2\" (UID: \"9b9207f1-4d6c-447c-9b24-c8cba3a2f21e\") " pod="openshift-multus/multus-mhnl2" Jul 22 00:26:56.939453 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.939187 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9b9207f1-4d6c-447c-9b24-c8cba3a2f21e-host-run-k8s-cni-cncf-io\") pod \"multus-mhnl2\" (UID: \"9b9207f1-4d6c-447c-9b24-c8cba3a2f21e\") " pod="openshift-multus/multus-mhnl2" Jul 22 00:26:56.939453 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.939219 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9b9207f1-4d6c-447c-9b24-c8cba3a2f21e-multus-conf-dir\") pod \"multus-mhnl2\" (UID: \"9b9207f1-4d6c-447c-9b24-c8cba3a2f21e\") " pod="openshift-multus/multus-mhnl2" Jul 22 00:26:56.939453 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.939250 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9b9207f1-4d6c-447c-9b24-c8cba3a2f21e-etc-kubernetes\") pod \"multus-mhnl2\" (UID: \"9b9207f1-4d6c-447c-9b24-c8cba3a2f21e\") " pod="openshift-multus/multus-mhnl2" Jul 22 00:26:56.939453 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.939279 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9b9207f1-4d6c-447c-9b24-c8cba3a2f21e-system-cni-dir\") pod \"multus-mhnl2\" (UID: \"9b9207f1-4d6c-447c-9b24-c8cba3a2f21e\") " pod="openshift-multus/multus-mhnl2" Jul 22 00:26:56.939453 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.939303 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9b9207f1-4d6c-447c-9b24-c8cba3a2f21e-host-var-lib-kubelet\") pod \"multus-mhnl2\" (UID: \"9b9207f1-4d6c-447c-9b24-c8cba3a2f21e\") " pod="openshift-multus/multus-mhnl2" Jul 22 00:26:56.939453 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.939333 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9b9207f1-4d6c-447c-9b24-c8cba3a2f21e-multus-daemon-config\") pod \"multus-mhnl2\" (UID: \"9b9207f1-4d6c-447c-9b24-c8cba3a2f21e\") " pod="openshift-multus/multus-mhnl2" Jul 22 00:26:56.939453 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.939370 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/44c9d8f9-4c85-4dfa-8572-2feb0f653720-serviceca\") pod \"node-ca-7wkvg\" (UID: \"44c9d8f9-4c85-4dfa-8572-2feb0f653720\") " pod="openshift-image-registry/node-ca-7wkvg" Jul 22 00:26:56.939453 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.939406 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/077d5538-1eab-48a5-97f9-3a623a4b63c9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v2ksf\" (UID: \"077d5538-1eab-48a5-97f9-3a623a4b63c9\") " pod="openshift-multus/multus-additional-cni-plugins-v2ksf" Jul 22 00:26:56.939453 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.939436 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9b9207f1-4d6c-447c-9b24-c8cba3a2f21e-host-var-lib-cni-bin\") pod \"multus-mhnl2\" (UID: \"9b9207f1-4d6c-447c-9b24-c8cba3a2f21e\") " pod="openshift-multus/multus-mhnl2" Jul 22 00:26:56.940350 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.940330 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hsv57" Jul 22 00:26:56.941491 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.941316 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 22 00:26:56.941491 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.941353 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 22 00:26:56.941656 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.941545 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-9nf5t\"" Jul 22 00:26:56.941791 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.941762 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 22 00:26:56.942403 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.942380 2574 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 22 00:26:56.942641 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.942624 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 22 00:26:56.943013 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.942997 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-z64sk\"" Jul 22 00:26:56.943110 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.943093 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ctbfz" Jul 22 00:26:56.943200 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.943181 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 22 00:26:56.945310 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.945293 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 22 00:26:56.945536 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.945517 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 22 00:26:56.945670 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.945523 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 22 00:26:56.945670 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.945584 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-hczsn\"" Jul 22 00:26:56.945795 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.945711 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hr5fn" Jul 22 00:26:56.947708 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.947692 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 22 00:26:56.948229 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.947969 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 22 00:26:56.948229 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.948038 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 22 00:26:56.948229 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.948047 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 22 00:26:56.948229 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.947969 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 22 00:26:56.948414 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.948252 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sl6b2" Jul 22 00:26:56.948414 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:26:56.948309 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sl6b2" podUID="08b6ba5c-7b7f-4cbd-a6d9-584bc9a829fd" Jul 22 00:26:56.948414 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.948386 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-wxkc6\"" Jul 22 00:26:56.948632 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.948618 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 22 00:26:56.964504 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.964483 2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-m7vqf" Jul 22 00:26:56.973701 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:56.973681 2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-m7vqf" Jul 22 00:26:57.029076 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.029051 2574 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 22 00:26:57.039679 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.039642 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/077d5538-1eab-48a5-97f9-3a623a4b63c9-cni-binary-copy\") pod \"multus-additional-cni-plugins-v2ksf\" (UID: \"077d5538-1eab-48a5-97f9-3a623a4b63c9\") " pod="openshift-multus/multus-additional-cni-plugins-v2ksf" Jul 22 00:26:57.039679 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.039674 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/077d5538-1eab-48a5-97f9-3a623a4b63c9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v2ksf\" (UID: \"077d5538-1eab-48a5-97f9-3a623a4b63c9\") " pod="openshift-multus/multus-additional-cni-plugins-v2ksf" Jul 22 00:26:57.039900 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.039789 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9b9207f1-4d6c-447c-9b24-c8cba3a2f21e-cni-binary-copy\") pod \"multus-mhnl2\" (UID: \"9b9207f1-4d6c-447c-9b24-c8cba3a2f21e\") " pod="openshift-multus/multus-mhnl2" Jul 22 00:26:57.039900 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.039818 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9b9207f1-4d6c-447c-9b24-c8cba3a2f21e-host-run-netns\") pod \"multus-mhnl2\" (UID: \"9b9207f1-4d6c-447c-9b24-c8cba3a2f21e\") " pod="openshift-multus/multus-mhnl2" Jul 22 00:26:57.039900 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.039837 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/352175c8-0733-4744-be74-676a559bb4cd-device-dir\") pod \"aws-ebs-csi-driver-node-sjxn9\" (UID: \"352175c8-0733-4744-be74-676a559bb4cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjxn9" Jul 22 00:26:57.040052 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.039914 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9b9207f1-4d6c-447c-9b24-c8cba3a2f21e-host-run-netns\") pod \"multus-mhnl2\" (UID: \"9b9207f1-4d6c-447c-9b24-c8cba3a2f21e\") " pod="openshift-multus/multus-mhnl2" Jul 22 00:26:57.040052 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.039958 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/17c40464-61b1-49c5-8648-d71165b62469-etc-modprobe-d\") pod \"tuned-hsv57\" (UID: \"17c40464-61b1-49c5-8648-d71165b62469\") " pod="openshift-cluster-node-tuning-operator/tuned-hsv57" Jul 22 00:26:57.040052 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.039990 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/17c40464-61b1-49c5-8648-d71165b62469-etc-sysctl-conf\") pod \"tuned-hsv57\" (UID: \"17c40464-61b1-49c5-8648-d71165b62469\") " pod="openshift-cluster-node-tuning-operator/tuned-hsv57" Jul 22 00:26:57.040052 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.040019 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d2fac953-0d0d-48cd-be69-240d4d16dd49-konnectivity-ca\") pod \"konnectivity-agent-zk4rp\" (UID: \"d2fac953-0d0d-48cd-be69-240d4d16dd49\") " pod="kube-system/konnectivity-agent-zk4rp" Jul 22 00:26:57.040052 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.040047 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/077d5538-1eab-48a5-97f9-3a623a4b63c9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v2ksf\" (UID: \"077d5538-1eab-48a5-97f9-3a623a4b63c9\") " pod="openshift-multus/multus-additional-cni-plugins-v2ksf" Jul 22 00:26:57.040295 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.040071 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9b9207f1-4d6c-447c-9b24-c8cba3a2f21e-multus-cni-dir\") pod \"multus-mhnl2\" (UID: \"9b9207f1-4d6c-447c-9b24-c8cba3a2f21e\") " pod="openshift-multus/multus-mhnl2" Jul 22 00:26:57.040295 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.040094 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/17c40464-61b1-49c5-8648-d71165b62469-etc-sysconfig\") pod \"tuned-hsv57\" (UID: \"17c40464-61b1-49c5-8648-d71165b62469\") " pod="openshift-cluster-node-tuning-operator/tuned-hsv57" Jul 22 00:26:57.040295 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.040117 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/17c40464-61b1-49c5-8648-d71165b62469-etc-sysctl-d\") pod \"tuned-hsv57\" (UID: \"17c40464-61b1-49c5-8648-d71165b62469\") " pod="openshift-cluster-node-tuning-operator/tuned-hsv57" Jul 22 00:26:57.040295 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.040138 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/17c40464-61b1-49c5-8648-d71165b62469-tmp\") pod \"tuned-hsv57\" (UID: \"17c40464-61b1-49c5-8648-d71165b62469\") " pod="openshift-cluster-node-tuning-operator/tuned-hsv57" Jul 22 00:26:57.040295 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.040163 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/077d5538-1eab-48a5-97f9-3a623a4b63c9-system-cni-dir\") pod \"multus-additional-cni-plugins-v2ksf\" (UID: \"077d5538-1eab-48a5-97f9-3a623a4b63c9\") " pod="openshift-multus/multus-additional-cni-plugins-v2ksf" Jul 22 00:26:57.040295 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.040188 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8mzm\" (UniqueName: \"kubernetes.io/projected/077d5538-1eab-48a5-97f9-3a623a4b63c9-kube-api-access-d8mzm\") pod \"multus-additional-cni-plugins-v2ksf\" (UID: \"077d5538-1eab-48a5-97f9-3a623a4b63c9\") " pod="openshift-multus/multus-additional-cni-plugins-v2ksf" Jul 22 00:26:57.040295 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.040213 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9b9207f1-4d6c-447c-9b24-c8cba3a2f21e-etc-kubernetes\") pod \"multus-mhnl2\" (UID: \"9b9207f1-4d6c-447c-9b24-c8cba3a2f21e\") " pod="openshift-multus/multus-mhnl2" Jul 22 00:26:57.040295 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.040225 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/077d5538-1eab-48a5-97f9-3a623a4b63c9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v2ksf\" (UID: \"077d5538-1eab-48a5-97f9-3a623a4b63c9\") " pod="openshift-multus/multus-additional-cni-plugins-v2ksf" Jul 22 00:26:57.040295 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.040236 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkrgp\" (UniqueName: \"kubernetes.io/projected/bbfe455c-9568-48ba-8007-92b4142b845f-kube-api-access-pkrgp\") pod \"network-check-target-6k9fm\" (UID: \"bbfe455c-9568-48ba-8007-92b4142b845f\") " pod="openshift-network-diagnostics/network-check-target-6k9fm" Jul 22 00:26:57.040295 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.040233 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/077d5538-1eab-48a5-97f9-3a623a4b63c9-system-cni-dir\") pod \"multus-additional-cni-plugins-v2ksf\" (UID: \"077d5538-1eab-48a5-97f9-3a623a4b63c9\") " pod="openshift-multus/multus-additional-cni-plugins-v2ksf" Jul 22 00:26:57.040685 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.040299 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9b9207f1-4d6c-447c-9b24-c8cba3a2f21e-multus-cni-dir\") pod \"multus-mhnl2\" (UID: \"9b9207f1-4d6c-447c-9b24-c8cba3a2f21e\") " pod="openshift-multus/multus-mhnl2" Jul 22 00:26:57.040685 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.040317 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9b9207f1-4d6c-447c-9b24-c8cba3a2f21e-etc-kubernetes\") pod \"multus-mhnl2\" (UID: \"9b9207f1-4d6c-447c-9b24-c8cba3a2f21e\") " pod="openshift-multus/multus-mhnl2" Jul 22 00:26:57.040685 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.040346 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/077d5538-1eab-48a5-97f9-3a623a4b63c9-cni-binary-copy\") pod \"multus-additional-cni-plugins-v2ksf\" (UID: \"077d5538-1eab-48a5-97f9-3a623a4b63c9\") " pod="openshift-multus/multus-additional-cni-plugins-v2ksf" Jul 22 00:26:57.040685 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.040410 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9b9207f1-4d6c-447c-9b24-c8cba3a2f21e-cni-binary-copy\") pod \"multus-mhnl2\" (UID: \"9b9207f1-4d6c-447c-9b24-c8cba3a2f21e\") " pod="openshift-multus/multus-mhnl2" Jul 22 00:26:57.040685 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.040567 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/17c40464-61b1-49c5-8648-d71165b62469-etc-kubernetes\") pod \"tuned-hsv57\" (UID: \"17c40464-61b1-49c5-8648-d71165b62469\") " pod="openshift-cluster-node-tuning-operator/tuned-hsv57" Jul 22 00:26:57.040685 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.040626 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d2fac953-0d0d-48cd-be69-240d4d16dd49-konnectivity-ca\") pod \"konnectivity-agent-zk4rp\" (UID: \"d2fac953-0d0d-48cd-be69-240d4d16dd49\") " pod="kube-system/konnectivity-agent-zk4rp" Jul 22 00:26:57.040685 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.040642 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/17c40464-61b1-49c5-8648-d71165b62469-lib-modules\") pod \"tuned-hsv57\" (UID: \"17c40464-61b1-49c5-8648-d71165b62469\") " pod="openshift-cluster-node-tuning-operator/tuned-hsv57" Jul 22 00:26:57.040685 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.040667 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/947d311a-ec4b-42e7-a0fb-ee5edc0caf8d-systemd-units\") pod \"ovnkube-node-hr5fn\" (UID: \"947d311a-ec4b-42e7-a0fb-ee5edc0caf8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5fn" Jul 22 00:26:57.040685 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.040683 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/947d311a-ec4b-42e7-a0fb-ee5edc0caf8d-host-slash\") pod \"ovnkube-node-hr5fn\" (UID: \"947d311a-ec4b-42e7-a0fb-ee5edc0caf8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5fn" Jul 22 00:26:57.041026 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.040697 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/352175c8-0733-4744-be74-676a559bb4cd-socket-dir\") pod \"aws-ebs-csi-driver-node-sjxn9\" (UID: \"352175c8-0733-4744-be74-676a559bb4cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjxn9" Jul 22 00:26:57.041026 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.040713 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/352175c8-0733-4744-be74-676a559bb4cd-etc-selinux\") pod \"aws-ebs-csi-driver-node-sjxn9\" (UID: \"352175c8-0733-4744-be74-676a559bb4cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjxn9" Jul 22 00:26:57.041026 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.040751 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bh4l\" (UniqueName: \"kubernetes.io/projected/5e5e6032-d779-43fe-8dac-33b4123b50bf-kube-api-access-5bh4l\") pod \"iptables-alerter-ctbfz\" (UID: \"5e5e6032-d779-43fe-8dac-33b4123b50bf\") " pod="openshift-network-operator/iptables-alerter-ctbfz" Jul 22 00:26:57.041026 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.040767 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/947d311a-ec4b-42e7-a0fb-ee5edc0caf8d-run-openvswitch\") pod \"ovnkube-node-hr5fn\" (UID: \"947d311a-ec4b-42e7-a0fb-ee5edc0caf8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5fn" Jul 22 00:26:57.041026 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.040781 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/947d311a-ec4b-42e7-a0fb-ee5edc0caf8d-node-log\") pod \"ovnkube-node-hr5fn\" (UID: \"947d311a-ec4b-42e7-a0fb-ee5edc0caf8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5fn" Jul 22 00:26:57.041026 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.040800 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/947d311a-ec4b-42e7-a0fb-ee5edc0caf8d-ovn-node-metrics-cert\") pod \"ovnkube-node-hr5fn\" (UID: \"947d311a-ec4b-42e7-a0fb-ee5edc0caf8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5fn" Jul 22 00:26:57.041026 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.040802 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/077d5538-1eab-48a5-97f9-3a623a4b63c9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v2ksf\" (UID: \"077d5538-1eab-48a5-97f9-3a623a4b63c9\") " pod="openshift-multus/multus-additional-cni-plugins-v2ksf" Jul 22 00:26:57.041026 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.040825 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/44c9d8f9-4c85-4dfa-8572-2feb0f653720-serviceca\") pod \"node-ca-7wkvg\" (UID: \"44c9d8f9-4c85-4dfa-8572-2feb0f653720\") " pod="openshift-image-registry/node-ca-7wkvg" Jul 22 00:26:57.041026 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.040842 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/077d5538-1eab-48a5-97f9-3a623a4b63c9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v2ksf\" (UID: \"077d5538-1eab-48a5-97f9-3a623a4b63c9\") " pod="openshift-multus/multus-additional-cni-plugins-v2ksf" Jul 22 00:26:57.041026 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.040855 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f36a57f8-6e88-43d3-b572-8adf3b0ddb7c-metrics-certs\") pod \"network-metrics-daemon-5xz72\" (UID: \"f36a57f8-6e88-43d3-b572-8adf3b0ddb7c\") " pod="openshift-multus/network-metrics-daemon-5xz72" Jul 22 00:26:57.041026 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.040872 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkkxw\" (UniqueName: \"kubernetes.io/projected/352175c8-0733-4744-be74-676a559bb4cd-kube-api-access-qkkxw\") pod \"aws-ebs-csi-driver-node-sjxn9\" (UID: \"352175c8-0733-4744-be74-676a559bb4cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjxn9" Jul 22 00:26:57.041026 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.040894 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/947d311a-ec4b-42e7-a0fb-ee5edc0caf8d-etc-openvswitch\") pod \"ovnkube-node-hr5fn\" (UID: \"947d311a-ec4b-42e7-a0fb-ee5edc0caf8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5fn" Jul 22 00:26:57.041026 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.040916 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/947d311a-ec4b-42e7-a0fb-ee5edc0caf8d-host-cni-netd\") pod \"ovnkube-node-hr5fn\" (UID: \"947d311a-ec4b-42e7-a0fb-ee5edc0caf8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5fn" Jul 22 00:26:57.041026 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.040942 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njshs\" (UniqueName: \"kubernetes.io/projected/9b9207f1-4d6c-447c-9b24-c8cba3a2f21e-kube-api-access-njshs\") pod \"multus-mhnl2\" (UID: \"9b9207f1-4d6c-447c-9b24-c8cba3a2f21e\") " pod="openshift-multus/multus-mhnl2" Jul 22 00:26:57.041026 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.040979 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/947d311a-ec4b-42e7-a0fb-ee5edc0caf8d-host-kubelet\") pod \"ovnkube-node-hr5fn\" (UID: \"947d311a-ec4b-42e7-a0fb-ee5edc0caf8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5fn" Jul 22 00:26:57.041632 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.041038 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/947d311a-ec4b-42e7-a0fb-ee5edc0caf8d-log-socket\") pod \"ovnkube-node-hr5fn\" (UID: \"947d311a-ec4b-42e7-a0fb-ee5edc0caf8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5fn" Jul 22 00:26:57.041632 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.041062 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/947d311a-ec4b-42e7-a0fb-ee5edc0caf8d-env-overrides\") pod \"ovnkube-node-hr5fn\" (UID: \"947d311a-ec4b-42e7-a0fb-ee5edc0caf8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5fn" Jul 22 00:26:57.041632 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.041088 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/44c9d8f9-4c85-4dfa-8572-2feb0f653720-host\") pod \"node-ca-7wkvg\" (UID: \"44c9d8f9-4c85-4dfa-8572-2feb0f653720\") " pod="openshift-image-registry/node-ca-7wkvg" Jul 22 00:26:57.041632 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.041161 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d2fac953-0d0d-48cd-be69-240d4d16dd49-agent-certs\") pod \"konnectivity-agent-zk4rp\" (UID: \"d2fac953-0d0d-48cd-be69-240d4d16dd49\") " pod="kube-system/konnectivity-agent-zk4rp" Jul 22 00:26:57.041632 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.041169 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/44c9d8f9-4c85-4dfa-8572-2feb0f653720-host\") pod \"node-ca-7wkvg\" (UID: \"44c9d8f9-4c85-4dfa-8572-2feb0f653720\") " pod="openshift-image-registry/node-ca-7wkvg" Jul 22 00:26:57.041632 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.041187 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfgd2\" (UniqueName: \"kubernetes.io/projected/17c40464-61b1-49c5-8648-d71165b62469-kube-api-access-bfgd2\") pod \"tuned-hsv57\" (UID: \"17c40464-61b1-49c5-8648-d71165b62469\") " pod="openshift-cluster-node-tuning-operator/tuned-hsv57" Jul 22 00:26:57.041632 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.041211 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/08b6ba5c-7b7f-4cbd-a6d9-584bc9a829fd-dbus\") pod \"global-pull-secret-syncer-sl6b2\" (UID: \"08b6ba5c-7b7f-4cbd-a6d9-584bc9a829fd\") " pod="kube-system/global-pull-secret-syncer-sl6b2" Jul 22 00:26:57.041632 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.041242 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9b9207f1-4d6c-447c-9b24-c8cba3a2f21e-multus-socket-dir-parent\") pod \"multus-mhnl2\" (UID: \"9b9207f1-4d6c-447c-9b24-c8cba3a2f21e\") " pod="openshift-multus/multus-mhnl2" Jul 22 00:26:57.041632 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.041270 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9b9207f1-4d6c-447c-9b24-c8cba3a2f21e-host-run-multus-certs\") pod \"multus-mhnl2\" (UID: \"9b9207f1-4d6c-447c-9b24-c8cba3a2f21e\") " pod="openshift-multus/multus-mhnl2" Jul 22 00:26:57.041632 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.041305 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/17c40464-61b1-49c5-8648-d71165b62469-etc-tuned\") pod \"tuned-hsv57\" (UID: \"17c40464-61b1-49c5-8648-d71165b62469\") " pod="openshift-cluster-node-tuning-operator/tuned-hsv57" Jul 22 00:26:57.041632 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.041304 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/44c9d8f9-4c85-4dfa-8572-2feb0f653720-serviceca\") pod \"node-ca-7wkvg\" (UID: \"44c9d8f9-4c85-4dfa-8572-2feb0f653720\") " pod="openshift-image-registry/node-ca-7wkvg" Jul 22 00:26:57.041632 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.041362 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9b9207f1-4d6c-447c-9b24-c8cba3a2f21e-multus-socket-dir-parent\") pod \"multus-mhnl2\" (UID: \"9b9207f1-4d6c-447c-9b24-c8cba3a2f21e\") " pod="openshift-multus/multus-mhnl2" Jul 22 00:26:57.041632 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.041363 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9b9207f1-4d6c-447c-9b24-c8cba3a2f21e-host-run-multus-certs\") pod \"multus-mhnl2\" (UID: \"9b9207f1-4d6c-447c-9b24-c8cba3a2f21e\") " pod="openshift-multus/multus-mhnl2" Jul 22 00:26:57.041632 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.041379 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/947d311a-ec4b-42e7-a0fb-ee5edc0caf8d-run-ovn\") pod \"ovnkube-node-hr5fn\" (UID: \"947d311a-ec4b-42e7-a0fb-ee5edc0caf8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5fn" Jul 22 00:26:57.041632 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.041399 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/947d311a-ec4b-42e7-a0fb-ee5edc0caf8d-ovnkube-script-lib\") pod \"ovnkube-node-hr5fn\" (UID: \"947d311a-ec4b-42e7-a0fb-ee5edc0caf8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5fn" Jul 22 00:26:57.041632 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.041360 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/077d5538-1eab-48a5-97f9-3a623a4b63c9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v2ksf\" (UID: \"077d5538-1eab-48a5-97f9-3a623a4b63c9\") " pod="openshift-multus/multus-additional-cni-plugins-v2ksf" Jul 22 00:26:57.041632 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.041425 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9b9207f1-4d6c-447c-9b24-c8cba3a2f21e-hostroot\") pod \"multus-mhnl2\" (UID: \"9b9207f1-4d6c-447c-9b24-c8cba3a2f21e\") " pod="openshift-multus/multus-mhnl2" Jul 22 00:26:57.042196 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.041451 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/35037e44-306a-4ec1-9937-d3f4b3b7891a-hosts-file\") pod \"node-resolver-zqvbx\" (UID: \"35037e44-306a-4ec1-9937-d3f4b3b7891a\") " pod="openshift-dns/node-resolver-zqvbx" Jul 22 00:26:57.042196 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.041475 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9b9207f1-4d6c-447c-9b24-c8cba3a2f21e-hostroot\") pod \"multus-mhnl2\" (UID: \"9b9207f1-4d6c-447c-9b24-c8cba3a2f21e\") " pod="openshift-multus/multus-mhnl2" Jul 22 00:26:57.042196 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.041500 2574 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 22 00:26:57.042196 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.041519 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/947d311a-ec4b-42e7-a0fb-ee5edc0caf8d-host-cni-bin\") pod \"ovnkube-node-hr5fn\" (UID: \"947d311a-ec4b-42e7-a0fb-ee5edc0caf8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5fn" Jul 22 00:26:57.042196 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.041548 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9b9207f1-4d6c-447c-9b24-c8cba3a2f21e-os-release\") pod \"multus-mhnl2\" (UID: \"9b9207f1-4d6c-447c-9b24-c8cba3a2f21e\") " pod="openshift-multus/multus-mhnl2" Jul 22 00:26:57.042196 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.041575 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9b9207f1-4d6c-447c-9b24-c8cba3a2f21e-host-run-k8s-cni-cncf-io\") pod \"multus-mhnl2\" (UID: \"9b9207f1-4d6c-447c-9b24-c8cba3a2f21e\") " pod="openshift-multus/multus-mhnl2" Jul 22 00:26:57.042196 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.041594 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9b9207f1-4d6c-447c-9b24-c8cba3a2f21e-multus-conf-dir\") pod \"multus-mhnl2\" (UID: \"9b9207f1-4d6c-447c-9b24-c8cba3a2f21e\") " pod="openshift-multus/multus-mhnl2" Jul 22 00:26:57.042196 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.041633 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/352175c8-0733-4744-be74-676a559bb4cd-sys-fs\") pod \"aws-ebs-csi-driver-node-sjxn9\" (UID: \"352175c8-0733-4744-be74-676a559bb4cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjxn9" Jul 22 00:26:57.042196 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.041631 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9b9207f1-4d6c-447c-9b24-c8cba3a2f21e-host-run-k8s-cni-cncf-io\") pod \"multus-mhnl2\" (UID: \"9b9207f1-4d6c-447c-9b24-c8cba3a2f21e\") " pod="openshift-multus/multus-mhnl2" Jul 22 00:26:57.042196 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.041663 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/17c40464-61b1-49c5-8648-d71165b62469-sys\") pod \"tuned-hsv57\" (UID: \"17c40464-61b1-49c5-8648-d71165b62469\") " pod="openshift-cluster-node-tuning-operator/tuned-hsv57" Jul 22 00:26:57.042196 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.041667 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9b9207f1-4d6c-447c-9b24-c8cba3a2f21e-multus-conf-dir\") pod \"multus-mhnl2\" (UID: \"9b9207f1-4d6c-447c-9b24-c8cba3a2f21e\") " pod="openshift-multus/multus-mhnl2" Jul 22 00:26:57.042196 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.041684 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5e5e6032-d779-43fe-8dac-33b4123b50bf-iptables-alerter-script\") pod \"iptables-alerter-ctbfz\" (UID: \"5e5e6032-d779-43fe-8dac-33b4123b50bf\") " pod="openshift-network-operator/iptables-alerter-ctbfz" Jul 22 00:26:57.042196 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.041717 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/947d311a-ec4b-42e7-a0fb-ee5edc0caf8d-host-run-ovn-kubernetes\") pod \"ovnkube-node-hr5fn\" (UID: \"947d311a-ec4b-42e7-a0fb-ee5edc0caf8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5fn" Jul 22 00:26:57.042196 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.041728 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9b9207f1-4d6c-447c-9b24-c8cba3a2f21e-os-release\") pod \"multus-mhnl2\" (UID: \"9b9207f1-4d6c-447c-9b24-c8cba3a2f21e\") " pod="openshift-multus/multus-mhnl2" Jul 22 00:26:57.042196 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.041751 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/08b6ba5c-7b7f-4cbd-a6d9-584bc9a829fd-original-pull-secret\") pod \"global-pull-secret-syncer-sl6b2\" (UID: \"08b6ba5c-7b7f-4cbd-a6d9-584bc9a829fd\") " pod="kube-system/global-pull-secret-syncer-sl6b2" Jul 22 00:26:57.042196 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.041778 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/17c40464-61b1-49c5-8648-d71165b62469-var-lib-kubelet\") pod \"tuned-hsv57\" (UID: \"17c40464-61b1-49c5-8648-d71165b62469\") " pod="openshift-cluster-node-tuning-operator/tuned-hsv57" Jul 22 00:26:57.042196 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.041798 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9b9207f1-4d6c-447c-9b24-c8cba3a2f21e-system-cni-dir\") pod \"multus-mhnl2\" (UID: \"9b9207f1-4d6c-447c-9b24-c8cba3a2f21e\") " pod="openshift-multus/multus-mhnl2" Jul 22 00:26:57.042196 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.041812 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9b9207f1-4d6c-447c-9b24-c8cba3a2f21e-host-var-lib-kubelet\") pod \"multus-mhnl2\" (UID: \"9b9207f1-4d6c-447c-9b24-c8cba3a2f21e\") " pod="openshift-multus/multus-mhnl2" Jul 22 00:26:57.043007 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.041826 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9b9207f1-4d6c-447c-9b24-c8cba3a2f21e-multus-daemon-config\") pod \"multus-mhnl2\" (UID: \"9b9207f1-4d6c-447c-9b24-c8cba3a2f21e\") " pod="openshift-multus/multus-mhnl2" Jul 22 00:26:57.043007 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.041848 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/352175c8-0733-4744-be74-676a559bb4cd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sjxn9\" (UID: \"352175c8-0733-4744-be74-676a559bb4cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjxn9" Jul 22 00:26:57.043007 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.041872 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9b9207f1-4d6c-447c-9b24-c8cba3a2f21e-system-cni-dir\") pod \"multus-mhnl2\" (UID: \"9b9207f1-4d6c-447c-9b24-c8cba3a2f21e\") " pod="openshift-multus/multus-mhnl2" Jul 22 00:26:57.043007 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.041872 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/947d311a-ec4b-42e7-a0fb-ee5edc0caf8d-host-run-netns\") pod \"ovnkube-node-hr5fn\" (UID: \"947d311a-ec4b-42e7-a0fb-ee5edc0caf8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5fn" Jul 22 00:26:57.043007 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.041882 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9b9207f1-4d6c-447c-9b24-c8cba3a2f21e-host-var-lib-kubelet\") pod \"multus-mhnl2\" (UID: \"9b9207f1-4d6c-447c-9b24-c8cba3a2f21e\") " pod="openshift-multus/multus-mhnl2" Jul 22 00:26:57.043007 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.041905 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/947d311a-ec4b-42e7-a0fb-ee5edc0caf8d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hr5fn\" (UID: \"947d311a-ec4b-42e7-a0fb-ee5edc0caf8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5fn" Jul 22 00:26:57.043007 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.041929 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9b9207f1-4d6c-447c-9b24-c8cba3a2f21e-host-var-lib-cni-bin\") pod \"multus-mhnl2\" (UID: \"9b9207f1-4d6c-447c-9b24-c8cba3a2f21e\") " pod="openshift-multus/multus-mhnl2" Jul 22 00:26:57.043007 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.041945 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lggx\" (UniqueName: \"kubernetes.io/projected/f36a57f8-6e88-43d3-b572-8adf3b0ddb7c-kube-api-access-2lggx\") pod \"network-metrics-daemon-5xz72\" (UID: \"f36a57f8-6e88-43d3-b572-8adf3b0ddb7c\") " pod="openshift-multus/network-metrics-daemon-5xz72" Jul 22 00:26:57.043007 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.041988 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9b9207f1-4d6c-447c-9b24-c8cba3a2f21e-host-var-lib-cni-bin\") pod \"multus-mhnl2\" (UID: \"9b9207f1-4d6c-447c-9b24-c8cba3a2f21e\") " pod="openshift-multus/multus-mhnl2" Jul 22 00:26:57.043007 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.041981 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb697\" (UniqueName: \"kubernetes.io/projected/35037e44-306a-4ec1-9937-d3f4b3b7891a-kube-api-access-vb697\") pod \"node-resolver-zqvbx\" (UID: \"35037e44-306a-4ec1-9937-d3f4b3b7891a\") " pod="openshift-dns/node-resolver-zqvbx" Jul 22 00:26:57.043007 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.042027 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5e5e6032-d779-43fe-8dac-33b4123b50bf-host-slash\") pod \"iptables-alerter-ctbfz\" (UID: \"5e5e6032-d779-43fe-8dac-33b4123b50bf\") " pod="openshift-network-operator/iptables-alerter-ctbfz" Jul 22 00:26:57.043007 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.042052 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/077d5538-1eab-48a5-97f9-3a623a4b63c9-cnibin\") pod \"multus-additional-cni-plugins-v2ksf\" (UID: \"077d5538-1eab-48a5-97f9-3a623a4b63c9\") " pod="openshift-multus/multus-additional-cni-plugins-v2ksf" Jul 22 00:26:57.043007 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.042083 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9b9207f1-4d6c-447c-9b24-c8cba3a2f21e-host-var-lib-cni-multus\") pod \"multus-mhnl2\" (UID: \"9b9207f1-4d6c-447c-9b24-c8cba3a2f21e\") " pod="openshift-multus/multus-mhnl2" Jul 22 00:26:57.043007 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.042088 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/077d5538-1eab-48a5-97f9-3a623a4b63c9-cnibin\") pod \"multus-additional-cni-plugins-v2ksf\" (UID: \"077d5538-1eab-48a5-97f9-3a623a4b63c9\") " pod="openshift-multus/multus-additional-cni-plugins-v2ksf" Jul 22 00:26:57.043007 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.042103 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/352175c8-0733-4744-be74-676a559bb4cd-registration-dir\") pod \"aws-ebs-csi-driver-node-sjxn9\" (UID: \"352175c8-0733-4744-be74-676a559bb4cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjxn9" Jul 22 00:26:57.043007 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.042116 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9b9207f1-4d6c-447c-9b24-c8cba3a2f21e-host-var-lib-cni-multus\") pod \"multus-mhnl2\" (UID: \"9b9207f1-4d6c-447c-9b24-c8cba3a2f21e\") " pod="openshift-multus/multus-mhnl2" Jul 22 00:26:57.043007 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.042126 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/17c40464-61b1-49c5-8648-d71165b62469-host\") pod \"tuned-hsv57\" (UID: \"17c40464-61b1-49c5-8648-d71165b62469\") " pod="openshift-cluster-node-tuning-operator/tuned-hsv57" Jul 22 00:26:57.043483 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.042146 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/947d311a-ec4b-42e7-a0fb-ee5edc0caf8d-run-systemd\") pod \"ovnkube-node-hr5fn\" (UID: \"947d311a-ec4b-42e7-a0fb-ee5edc0caf8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5fn" Jul 22 00:26:57.043483 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.042164 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/947d311a-ec4b-42e7-a0fb-ee5edc0caf8d-ovnkube-config\") pod \"ovnkube-node-hr5fn\" (UID: \"947d311a-ec4b-42e7-a0fb-ee5edc0caf8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5fn" Jul 22 00:26:57.043483 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.042183 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqrd4\" (UniqueName: \"kubernetes.io/projected/947d311a-ec4b-42e7-a0fb-ee5edc0caf8d-kube-api-access-fqrd4\") pod \"ovnkube-node-hr5fn\" (UID: \"947d311a-ec4b-42e7-a0fb-ee5edc0caf8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5fn" Jul 22 00:26:57.043483 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.042206 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsrv7\" (UniqueName: \"kubernetes.io/projected/44c9d8f9-4c85-4dfa-8572-2feb0f653720-kube-api-access-fsrv7\") pod \"node-ca-7wkvg\" (UID: \"44c9d8f9-4c85-4dfa-8572-2feb0f653720\") " pod="openshift-image-registry/node-ca-7wkvg" Jul 22 00:26:57.043483 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.042230 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35037e44-306a-4ec1-9937-d3f4b3b7891a-tmp-dir\") pod \"node-resolver-zqvbx\" (UID: \"35037e44-306a-4ec1-9937-d3f4b3b7891a\") " pod="openshift-dns/node-resolver-zqvbx" Jul 22 00:26:57.043483 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.042247 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/947d311a-ec4b-42e7-a0fb-ee5edc0caf8d-var-lib-openvswitch\") pod \"ovnkube-node-hr5fn\" (UID: \"947d311a-ec4b-42e7-a0fb-ee5edc0caf8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5fn" Jul 22 00:26:57.043483 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.042260 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9b9207f1-4d6c-447c-9b24-c8cba3a2f21e-multus-daemon-config\") pod \"multus-mhnl2\" (UID: \"9b9207f1-4d6c-447c-9b24-c8cba3a2f21e\") " pod="openshift-multus/multus-mhnl2" Jul 22 00:26:57.043483 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.042262 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/08b6ba5c-7b7f-4cbd-a6d9-584bc9a829fd-kubelet-config\") pod \"global-pull-secret-syncer-sl6b2\" (UID: \"08b6ba5c-7b7f-4cbd-a6d9-584bc9a829fd\") " pod="kube-system/global-pull-secret-syncer-sl6b2" Jul 22 00:26:57.043483 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.042301 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/077d5538-1eab-48a5-97f9-3a623a4b63c9-os-release\") pod \"multus-additional-cni-plugins-v2ksf\" (UID: \"077d5538-1eab-48a5-97f9-3a623a4b63c9\") " pod="openshift-multus/multus-additional-cni-plugins-v2ksf" Jul 22 00:26:57.043483 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.042317 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9b9207f1-4d6c-447c-9b24-c8cba3a2f21e-cnibin\") pod \"multus-mhnl2\" (UID: \"9b9207f1-4d6c-447c-9b24-c8cba3a2f21e\") " pod="openshift-multus/multus-mhnl2" Jul 22 00:26:57.043483 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.042331 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/17c40464-61b1-49c5-8648-d71165b62469-etc-systemd\") pod \"tuned-hsv57\" (UID: \"17c40464-61b1-49c5-8648-d71165b62469\") " pod="openshift-cluster-node-tuning-operator/tuned-hsv57" Jul 22 00:26:57.043483 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.042351 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/17c40464-61b1-49c5-8648-d71165b62469-run\") pod \"tuned-hsv57\" (UID: \"17c40464-61b1-49c5-8648-d71165b62469\") " pod="openshift-cluster-node-tuning-operator/tuned-hsv57" Jul 22 00:26:57.043483 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.042358 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/077d5538-1eab-48a5-97f9-3a623a4b63c9-os-release\") pod \"multus-additional-cni-plugins-v2ksf\" (UID: \"077d5538-1eab-48a5-97f9-3a623a4b63c9\") " pod="openshift-multus/multus-additional-cni-plugins-v2ksf" Jul 22 00:26:57.043483 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.042383 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9b9207f1-4d6c-447c-9b24-c8cba3a2f21e-cnibin\") pod \"multus-mhnl2\" (UID: \"9b9207f1-4d6c-447c-9b24-c8cba3a2f21e\") " pod="openshift-multus/multus-mhnl2" Jul 22 00:26:57.044976 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.044960 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d2fac953-0d0d-48cd-be69-240d4d16dd49-agent-certs\") pod \"konnectivity-agent-zk4rp\" (UID: \"d2fac953-0d0d-48cd-be69-240d4d16dd49\") " pod="kube-system/konnectivity-agent-zk4rp" Jul 22 00:26:57.049174 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.049149 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8mzm\" (UniqueName: \"kubernetes.io/projected/077d5538-1eab-48a5-97f9-3a623a4b63c9-kube-api-access-d8mzm\") pod \"multus-additional-cni-plugins-v2ksf\" (UID: \"077d5538-1eab-48a5-97f9-3a623a4b63c9\") " pod="openshift-multus/multus-additional-cni-plugins-v2ksf" Jul 22 00:26:57.049282 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.049152 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njshs\" (UniqueName: \"kubernetes.io/projected/9b9207f1-4d6c-447c-9b24-c8cba3a2f21e-kube-api-access-njshs\") pod \"multus-mhnl2\" (UID: \"9b9207f1-4d6c-447c-9b24-c8cba3a2f21e\") " pod="openshift-multus/multus-mhnl2" Jul 22 00:26:57.049408 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.049390 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsrv7\" (UniqueName: \"kubernetes.io/projected/44c9d8f9-4c85-4dfa-8572-2feb0f653720-kube-api-access-fsrv7\") pod \"node-ca-7wkvg\" (UID: \"44c9d8f9-4c85-4dfa-8572-2feb0f653720\") " pod="openshift-image-registry/node-ca-7wkvg" Jul 22 00:26:57.088019 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.087965 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-192.ec2.internal" event={"ID":"2951ff349dfaeb74342ef57c90cc46ad","Type":"ContainerStarted","Data":"3c8ce25c56ac3afd54c31f5c6138becf8e10773cb52e0012eb32af3024c68689"} Jul 22 00:26:57.089223 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.089194 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-192.ec2.internal" event={"ID":"5c5847e7e75764b70e99faa3e06c32f2","Type":"ContainerStarted","Data":"946bcaa8022149efc256e6323c0482d1b7dee2eef1332d03cab0cd294770a89c"} Jul 22 00:26:57.142971 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.142937 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f36a57f8-6e88-43d3-b572-8adf3b0ddb7c-metrics-certs\") pod \"network-metrics-daemon-5xz72\" (UID: \"f36a57f8-6e88-43d3-b572-8adf3b0ddb7c\") " pod="openshift-multus/network-metrics-daemon-5xz72" Jul 22 00:26:57.143198 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.142986 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkkxw\" (UniqueName: \"kubernetes.io/projected/352175c8-0733-4744-be74-676a559bb4cd-kube-api-access-qkkxw\") pod \"aws-ebs-csi-driver-node-sjxn9\" (UID: \"352175c8-0733-4744-be74-676a559bb4cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjxn9" Jul 22 00:26:57.143198 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.143013 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/947d311a-ec4b-42e7-a0fb-ee5edc0caf8d-etc-openvswitch\") pod \"ovnkube-node-hr5fn\" (UID: \"947d311a-ec4b-42e7-a0fb-ee5edc0caf8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5fn" Jul 22 00:26:57.143198 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.143040 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/947d311a-ec4b-42e7-a0fb-ee5edc0caf8d-host-cni-netd\") pod \"ovnkube-node-hr5fn\" (UID: \"947d311a-ec4b-42e7-a0fb-ee5edc0caf8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5fn" Jul 22 00:26:57.143198 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.143066 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/947d311a-ec4b-42e7-a0fb-ee5edc0caf8d-host-kubelet\") pod \"ovnkube-node-hr5fn\" (UID: \"947d311a-ec4b-42e7-a0fb-ee5edc0caf8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5fn" Jul 22 00:26:57.143198 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.143107 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/947d311a-ec4b-42e7-a0fb-ee5edc0caf8d-log-socket\") pod \"ovnkube-node-hr5fn\" (UID: \"947d311a-ec4b-42e7-a0fb-ee5edc0caf8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5fn" Jul 22 00:26:57.143198 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:26:57.143118 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 00:26:57.143198 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.143143 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/947d311a-ec4b-42e7-a0fb-ee5edc0caf8d-etc-openvswitch\") pod \"ovnkube-node-hr5fn\" (UID: \"947d311a-ec4b-42e7-a0fb-ee5edc0caf8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5fn" Jul 22 00:26:57.143198 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.143130 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/947d311a-ec4b-42e7-a0fb-ee5edc0caf8d-env-overrides\") pod \"ovnkube-node-hr5fn\" (UID: \"947d311a-ec4b-42e7-a0fb-ee5edc0caf8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5fn" Jul 22 00:26:57.143688 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.143219 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/947d311a-ec4b-42e7-a0fb-ee5edc0caf8d-host-kubelet\") pod \"ovnkube-node-hr5fn\" (UID: \"947d311a-ec4b-42e7-a0fb-ee5edc0caf8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5fn" Jul 22 00:26:57.143688 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:26:57.143241 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f36a57f8-6e88-43d3-b572-8adf3b0ddb7c-metrics-certs podName:f36a57f8-6e88-43d3-b572-8adf3b0ddb7c nodeName:}" failed. No retries permitted until 2026-07-22 00:26:57.643170876 +0000 UTC m=+2.999928749 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f36a57f8-6e88-43d3-b572-8adf3b0ddb7c-metrics-certs") pod "network-metrics-daemon-5xz72" (UID: "f36a57f8-6e88-43d3-b572-8adf3b0ddb7c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 00:26:57.143688 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.143268 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/947d311a-ec4b-42e7-a0fb-ee5edc0caf8d-host-cni-netd\") pod \"ovnkube-node-hr5fn\" (UID: \"947d311a-ec4b-42e7-a0fb-ee5edc0caf8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5fn" Jul 22 00:26:57.143688 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.143315 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/947d311a-ec4b-42e7-a0fb-ee5edc0caf8d-log-socket\") pod \"ovnkube-node-hr5fn\" (UID: \"947d311a-ec4b-42e7-a0fb-ee5edc0caf8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5fn" Jul 22 00:26:57.143688 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.143327 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfgd2\" (UniqueName: \"kubernetes.io/projected/17c40464-61b1-49c5-8648-d71165b62469-kube-api-access-bfgd2\") pod \"tuned-hsv57\" (UID: \"17c40464-61b1-49c5-8648-d71165b62469\") " pod="openshift-cluster-node-tuning-operator/tuned-hsv57" Jul 22 00:26:57.143688 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.143353 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/08b6ba5c-7b7f-4cbd-a6d9-584bc9a829fd-dbus\") pod \"global-pull-secret-syncer-sl6b2\" (UID: \"08b6ba5c-7b7f-4cbd-a6d9-584bc9a829fd\") " pod="kube-system/global-pull-secret-syncer-sl6b2" Jul 22 00:26:57.143688 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.143376 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/17c40464-61b1-49c5-8648-d71165b62469-etc-tuned\") pod \"tuned-hsv57\" (UID: \"17c40464-61b1-49c5-8648-d71165b62469\") " pod="openshift-cluster-node-tuning-operator/tuned-hsv57" Jul 22 00:26:57.143688 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.143391 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/947d311a-ec4b-42e7-a0fb-ee5edc0caf8d-run-ovn\") pod \"ovnkube-node-hr5fn\" (UID: \"947d311a-ec4b-42e7-a0fb-ee5edc0caf8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5fn" Jul 22 00:26:57.143688 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.143405 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/947d311a-ec4b-42e7-a0fb-ee5edc0caf8d-ovnkube-script-lib\") pod \"ovnkube-node-hr5fn\" (UID: \"947d311a-ec4b-42e7-a0fb-ee5edc0caf8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5fn" Jul 22 00:26:57.143688 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.143422 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/35037e44-306a-4ec1-9937-d3f4b3b7891a-hosts-file\") pod \"node-resolver-zqvbx\" (UID: \"35037e44-306a-4ec1-9937-d3f4b3b7891a\") " pod="openshift-dns/node-resolver-zqvbx" Jul 22 00:26:57.143688 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.143443 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/947d311a-ec4b-42e7-a0fb-ee5edc0caf8d-host-cni-bin\") pod \"ovnkube-node-hr5fn\" (UID: \"947d311a-ec4b-42e7-a0fb-ee5edc0caf8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5fn" Jul 22 00:26:57.143688 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.143463 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/352175c8-0733-4744-be74-676a559bb4cd-sys-fs\") pod \"aws-ebs-csi-driver-node-sjxn9\" (UID: \"352175c8-0733-4744-be74-676a559bb4cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjxn9" Jul 22 00:26:57.143688 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.143480 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/17c40464-61b1-49c5-8648-d71165b62469-sys\") pod \"tuned-hsv57\" (UID: \"17c40464-61b1-49c5-8648-d71165b62469\") " pod="openshift-cluster-node-tuning-operator/tuned-hsv57" Jul 22 00:26:57.143688 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.143496 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5e5e6032-d779-43fe-8dac-33b4123b50bf-iptables-alerter-script\") pod \"iptables-alerter-ctbfz\" (UID: \"5e5e6032-d779-43fe-8dac-33b4123b50bf\") " pod="openshift-network-operator/iptables-alerter-ctbfz" Jul 22 00:26:57.143688 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.143518 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/947d311a-ec4b-42e7-a0fb-ee5edc0caf8d-host-run-ovn-kubernetes\") pod \"ovnkube-node-hr5fn\" (UID: \"947d311a-ec4b-42e7-a0fb-ee5edc0caf8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5fn" Jul 22 00:26:57.143688 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.143532 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/35037e44-306a-4ec1-9937-d3f4b3b7891a-hosts-file\") pod \"node-resolver-zqvbx\" (UID: \"35037e44-306a-4ec1-9937-d3f4b3b7891a\") " pod="openshift-dns/node-resolver-zqvbx" Jul 22 00:26:57.143688 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.143566 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/947d311a-ec4b-42e7-a0fb-ee5edc0caf8d-host-cni-bin\") pod \"ovnkube-node-hr5fn\" (UID: \"947d311a-ec4b-42e7-a0fb-ee5edc0caf8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5fn" Jul 22 00:26:57.144482 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.143625 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/947d311a-ec4b-42e7-a0fb-ee5edc0caf8d-env-overrides\") pod \"ovnkube-node-hr5fn\" (UID: \"947d311a-ec4b-42e7-a0fb-ee5edc0caf8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5fn" Jul 22 00:26:57.144482 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:26:57.143634 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 00:26:57.144482 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.143659 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/947d311a-ec4b-42e7-a0fb-ee5edc0caf8d-host-run-ovn-kubernetes\") pod \"ovnkube-node-hr5fn\" (UID: \"947d311a-ec4b-42e7-a0fb-ee5edc0caf8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5fn" Jul 22 00:26:57.144482 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.143537 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/08b6ba5c-7b7f-4cbd-a6d9-584bc9a829fd-original-pull-secret\") pod \"global-pull-secret-syncer-sl6b2\" (UID: \"08b6ba5c-7b7f-4cbd-a6d9-584bc9a829fd\") " pod="kube-system/global-pull-secret-syncer-sl6b2" Jul 22 00:26:57.144482 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.143631 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/17c40464-61b1-49c5-8648-d71165b62469-sys\") pod \"tuned-hsv57\" (UID: \"17c40464-61b1-49c5-8648-d71165b62469\") " pod="openshift-cluster-node-tuning-operator/tuned-hsv57" Jul 22 00:26:57.144482 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:26:57.143683 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08b6ba5c-7b7f-4cbd-a6d9-584bc9a829fd-original-pull-secret podName:08b6ba5c-7b7f-4cbd-a6d9-584bc9a829fd nodeName:}" failed. No retries permitted until 2026-07-22 00:26:57.64366762 +0000 UTC m=+3.000425489 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/08b6ba5c-7b7f-4cbd-a6d9-584bc9a829fd-original-pull-secret") pod "global-pull-secret-syncer-sl6b2" (UID: "08b6ba5c-7b7f-4cbd-a6d9-584bc9a829fd") : object "kube-system"/"original-pull-secret" not registered Jul 22 00:26:57.144482 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.143721 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/17c40464-61b1-49c5-8648-d71165b62469-var-lib-kubelet\") pod \"tuned-hsv57\" (UID: \"17c40464-61b1-49c5-8648-d71165b62469\") " pod="openshift-cluster-node-tuning-operator/tuned-hsv57" Jul 22 00:26:57.144482 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.143734 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/08b6ba5c-7b7f-4cbd-a6d9-584bc9a829fd-dbus\") pod \"global-pull-secret-syncer-sl6b2\" (UID: \"08b6ba5c-7b7f-4cbd-a6d9-584bc9a829fd\") " pod="kube-system/global-pull-secret-syncer-sl6b2" Jul 22 00:26:57.144482 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.143740 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/352175c8-0733-4744-be74-676a559bb4cd-sys-fs\") pod \"aws-ebs-csi-driver-node-sjxn9\" (UID: \"352175c8-0733-4744-be74-676a559bb4cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjxn9" Jul 22 00:26:57.144482 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.143755 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/352175c8-0733-4744-be74-676a559bb4cd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sjxn9\" (UID: \"352175c8-0733-4744-be74-676a559bb4cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjxn9" Jul 22 00:26:57.144482 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.143778 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/947d311a-ec4b-42e7-a0fb-ee5edc0caf8d-run-ovn\") pod \"ovnkube-node-hr5fn\" (UID: \"947d311a-ec4b-42e7-a0fb-ee5edc0caf8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5fn" Jul 22 00:26:57.144482 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.143779 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/947d311a-ec4b-42e7-a0fb-ee5edc0caf8d-host-run-netns\") pod \"ovnkube-node-hr5fn\" (UID: \"947d311a-ec4b-42e7-a0fb-ee5edc0caf8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5fn" Jul 22 00:26:57.144482 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.143816 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/947d311a-ec4b-42e7-a0fb-ee5edc0caf8d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hr5fn\" (UID: \"947d311a-ec4b-42e7-a0fb-ee5edc0caf8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5fn" Jul 22 00:26:57.144482 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.143827 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/352175c8-0733-4744-be74-676a559bb4cd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sjxn9\" (UID: \"352175c8-0733-4744-be74-676a559bb4cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjxn9" Jul 22 00:26:57.144482 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.143848 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lggx\" (UniqueName: \"kubernetes.io/projected/f36a57f8-6e88-43d3-b572-8adf3b0ddb7c-kube-api-access-2lggx\") pod \"network-metrics-daemon-5xz72\" (UID: \"f36a57f8-6e88-43d3-b572-8adf3b0ddb7c\") " pod="openshift-multus/network-metrics-daemon-5xz72" Jul 22 00:26:57.144482 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.143853 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/947d311a-ec4b-42e7-a0fb-ee5edc0caf8d-host-run-netns\") pod \"ovnkube-node-hr5fn\" (UID: \"947d311a-ec4b-42e7-a0fb-ee5edc0caf8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5fn" Jul 22 00:26:57.144482 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.143790 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/17c40464-61b1-49c5-8648-d71165b62469-var-lib-kubelet\") pod \"tuned-hsv57\" (UID: \"17c40464-61b1-49c5-8648-d71165b62469\") " pod="openshift-cluster-node-tuning-operator/tuned-hsv57" Jul 22 00:26:57.145404 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.143867 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/947d311a-ec4b-42e7-a0fb-ee5edc0caf8d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hr5fn\" (UID: \"947d311a-ec4b-42e7-a0fb-ee5edc0caf8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5fn" Jul 22 00:26:57.145404 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.143883 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb697\" (UniqueName: \"kubernetes.io/projected/35037e44-306a-4ec1-9937-d3f4b3b7891a-kube-api-access-vb697\") pod \"node-resolver-zqvbx\" (UID: \"35037e44-306a-4ec1-9937-d3f4b3b7891a\") " pod="openshift-dns/node-resolver-zqvbx" Jul 22 00:26:57.145404 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.143910 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5e5e6032-d779-43fe-8dac-33b4123b50bf-host-slash\") pod \"iptables-alerter-ctbfz\" (UID: \"5e5e6032-d779-43fe-8dac-33b4123b50bf\") " pod="openshift-network-operator/iptables-alerter-ctbfz" Jul 22 00:26:57.145404 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.143947 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/352175c8-0733-4744-be74-676a559bb4cd-registration-dir\") pod \"aws-ebs-csi-driver-node-sjxn9\" (UID: \"352175c8-0733-4744-be74-676a559bb4cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjxn9" Jul 22 00:26:57.145404 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.143987 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/17c40464-61b1-49c5-8648-d71165b62469-host\") pod \"tuned-hsv57\" (UID: \"17c40464-61b1-49c5-8648-d71165b62469\") " pod="openshift-cluster-node-tuning-operator/tuned-hsv57" Jul 22 00:26:57.145404 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.144005 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/352175c8-0733-4744-be74-676a559bb4cd-registration-dir\") pod \"aws-ebs-csi-driver-node-sjxn9\" (UID: \"352175c8-0733-4744-be74-676a559bb4cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjxn9" Jul 22 00:26:57.145404 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.144014 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/947d311a-ec4b-42e7-a0fb-ee5edc0caf8d-run-systemd\") pod \"ovnkube-node-hr5fn\" (UID: \"947d311a-ec4b-42e7-a0fb-ee5edc0caf8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5fn" Jul 22 00:26:57.145404 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.144038 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/947d311a-ec4b-42e7-a0fb-ee5edc0caf8d-ovnkube-config\") pod \"ovnkube-node-hr5fn\" (UID: \"947d311a-ec4b-42e7-a0fb-ee5edc0caf8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5fn" Jul 22 00:26:57.145404 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.144078 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/17c40464-61b1-49c5-8648-d71165b62469-host\") pod \"tuned-hsv57\" (UID: \"17c40464-61b1-49c5-8648-d71165b62469\") " pod="openshift-cluster-node-tuning-operator/tuned-hsv57" Jul 22 00:26:57.145404 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.144084 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqrd4\" (UniqueName: \"kubernetes.io/projected/947d311a-ec4b-42e7-a0fb-ee5edc0caf8d-kube-api-access-fqrd4\") pod \"ovnkube-node-hr5fn\" (UID: \"947d311a-ec4b-42e7-a0fb-ee5edc0caf8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5fn" Jul 22 00:26:57.145404 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.144111 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35037e44-306a-4ec1-9937-d3f4b3b7891a-tmp-dir\") pod \"node-resolver-zqvbx\" (UID: \"35037e44-306a-4ec1-9937-d3f4b3b7891a\") " pod="openshift-dns/node-resolver-zqvbx" Jul 22 00:26:57.145404 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.144116 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5e5e6032-d779-43fe-8dac-33b4123b50bf-host-slash\") pod \"iptables-alerter-ctbfz\" (UID: \"5e5e6032-d779-43fe-8dac-33b4123b50bf\") " pod="openshift-network-operator/iptables-alerter-ctbfz" Jul 22 00:26:57.145404 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.144111 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/947d311a-ec4b-42e7-a0fb-ee5edc0caf8d-ovnkube-script-lib\") pod \"ovnkube-node-hr5fn\" (UID: \"947d311a-ec4b-42e7-a0fb-ee5edc0caf8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5fn" Jul 22 00:26:57.145404 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.144149 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/947d311a-ec4b-42e7-a0fb-ee5edc0caf8d-var-lib-openvswitch\") pod \"ovnkube-node-hr5fn\" (UID: \"947d311a-ec4b-42e7-a0fb-ee5edc0caf8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5fn" Jul 22 00:26:57.145404 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.144150 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5e5e6032-d779-43fe-8dac-33b4123b50bf-iptables-alerter-script\") pod \"iptables-alerter-ctbfz\" (UID: \"5e5e6032-d779-43fe-8dac-33b4123b50bf\") " pod="openshift-network-operator/iptables-alerter-ctbfz" Jul 22 00:26:57.145404 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.144179 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/08b6ba5c-7b7f-4cbd-a6d9-584bc9a829fd-kubelet-config\") pod \"global-pull-secret-syncer-sl6b2\" (UID: \"08b6ba5c-7b7f-4cbd-a6d9-584bc9a829fd\") " pod="kube-system/global-pull-secret-syncer-sl6b2" Jul 22 00:26:57.145404 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.144195 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/947d311a-ec4b-42e7-a0fb-ee5edc0caf8d-run-systemd\") pod \"ovnkube-node-hr5fn\" (UID: \"947d311a-ec4b-42e7-a0fb-ee5edc0caf8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5fn" Jul 22 00:26:57.146190 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.144238 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/08b6ba5c-7b7f-4cbd-a6d9-584bc9a829fd-kubelet-config\") pod \"global-pull-secret-syncer-sl6b2\" (UID: \"08b6ba5c-7b7f-4cbd-a6d9-584bc9a829fd\") " pod="kube-system/global-pull-secret-syncer-sl6b2" Jul 22 00:26:57.146190 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.144259 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/947d311a-ec4b-42e7-a0fb-ee5edc0caf8d-var-lib-openvswitch\") pod \"ovnkube-node-hr5fn\" (UID: \"947d311a-ec4b-42e7-a0fb-ee5edc0caf8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5fn" Jul 22 00:26:57.146190 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.144325 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/17c40464-61b1-49c5-8648-d71165b62469-etc-systemd\") pod \"tuned-hsv57\" (UID: \"17c40464-61b1-49c5-8648-d71165b62469\") " pod="openshift-cluster-node-tuning-operator/tuned-hsv57" Jul 22 00:26:57.146190 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.144359 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/17c40464-61b1-49c5-8648-d71165b62469-run\") pod \"tuned-hsv57\" (UID: \"17c40464-61b1-49c5-8648-d71165b62469\") " pod="openshift-cluster-node-tuning-operator/tuned-hsv57" Jul 22 00:26:57.146190 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.144386 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/352175c8-0733-4744-be74-676a559bb4cd-device-dir\") pod \"aws-ebs-csi-driver-node-sjxn9\" (UID: \"352175c8-0733-4744-be74-676a559bb4cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjxn9" Jul 22 00:26:57.146190 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.144410 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/17c40464-61b1-49c5-8648-d71165b62469-etc-modprobe-d\") pod \"tuned-hsv57\" (UID: \"17c40464-61b1-49c5-8648-d71165b62469\") " pod="openshift-cluster-node-tuning-operator/tuned-hsv57" Jul 22 00:26:57.146190 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.144416 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/17c40464-61b1-49c5-8648-d71165b62469-etc-systemd\") pod \"tuned-hsv57\" (UID: \"17c40464-61b1-49c5-8648-d71165b62469\") " pod="openshift-cluster-node-tuning-operator/tuned-hsv57" Jul 22 00:26:57.146190 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.144428 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/17c40464-61b1-49c5-8648-d71165b62469-run\") pod \"tuned-hsv57\" (UID: \"17c40464-61b1-49c5-8648-d71165b62469\") " pod="openshift-cluster-node-tuning-operator/tuned-hsv57" Jul 22 00:26:57.146190 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.144446 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/17c40464-61b1-49c5-8648-d71165b62469-etc-sysctl-conf\") pod \"tuned-hsv57\" (UID: \"17c40464-61b1-49c5-8648-d71165b62469\") " pod="openshift-cluster-node-tuning-operator/tuned-hsv57" Jul 22 00:26:57.146190 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.144464 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/352175c8-0733-4744-be74-676a559bb4cd-device-dir\") pod \"aws-ebs-csi-driver-node-sjxn9\" (UID: \"352175c8-0733-4744-be74-676a559bb4cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjxn9" Jul 22 00:26:57.146190 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.144498 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/17c40464-61b1-49c5-8648-d71165b62469-etc-sysconfig\") pod \"tuned-hsv57\" (UID: \"17c40464-61b1-49c5-8648-d71165b62469\") " pod="openshift-cluster-node-tuning-operator/tuned-hsv57" Jul 22 00:26:57.146190 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.144538 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/17c40464-61b1-49c5-8648-d71165b62469-etc-sysctl-d\") pod \"tuned-hsv57\" (UID: \"17c40464-61b1-49c5-8648-d71165b62469\") " pod="openshift-cluster-node-tuning-operator/tuned-hsv57" Jul 22 00:26:57.146190 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.144560 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/17c40464-61b1-49c5-8648-d71165b62469-etc-modprobe-d\") pod \"tuned-hsv57\" (UID: \"17c40464-61b1-49c5-8648-d71165b62469\") " pod="openshift-cluster-node-tuning-operator/tuned-hsv57" Jul 22 00:26:57.146190 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.144556 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/17c40464-61b1-49c5-8648-d71165b62469-tmp\") pod \"tuned-hsv57\" (UID: \"17c40464-61b1-49c5-8648-d71165b62469\") " pod="openshift-cluster-node-tuning-operator/tuned-hsv57" Jul 22 00:26:57.146190 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.144617 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/17c40464-61b1-49c5-8648-d71165b62469-etc-sysctl-conf\") pod \"tuned-hsv57\" (UID: \"17c40464-61b1-49c5-8648-d71165b62469\") " pod="openshift-cluster-node-tuning-operator/tuned-hsv57" Jul 22 00:26:57.146190 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.144638 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/17c40464-61b1-49c5-8648-d71165b62469-etc-sysconfig\") pod \"tuned-hsv57\" (UID: \"17c40464-61b1-49c5-8648-d71165b62469\") " pod="openshift-cluster-node-tuning-operator/tuned-hsv57" Jul 22 00:26:57.146190 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.144640 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/947d311a-ec4b-42e7-a0fb-ee5edc0caf8d-ovnkube-config\") pod \"ovnkube-node-hr5fn\" (UID: \"947d311a-ec4b-42e7-a0fb-ee5edc0caf8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5fn" Jul 22 00:26:57.146190 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.144429 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35037e44-306a-4ec1-9937-d3f4b3b7891a-tmp-dir\") pod \"node-resolver-zqvbx\" (UID: \"35037e44-306a-4ec1-9937-d3f4b3b7891a\") " pod="openshift-dns/node-resolver-zqvbx" Jul 22 00:26:57.147076 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.144637 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkrgp\" (UniqueName: \"kubernetes.io/projected/bbfe455c-9568-48ba-8007-92b4142b845f-kube-api-access-pkrgp\") pod \"network-check-target-6k9fm\" (UID: \"bbfe455c-9568-48ba-8007-92b4142b845f\") " pod="openshift-network-diagnostics/network-check-target-6k9fm" Jul 22 00:26:57.147076 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.144692 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/17c40464-61b1-49c5-8648-d71165b62469-etc-kubernetes\") pod \"tuned-hsv57\" (UID: \"17c40464-61b1-49c5-8648-d71165b62469\") " pod="openshift-cluster-node-tuning-operator/tuned-hsv57" Jul 22 00:26:57.147076 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.144717 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/17c40464-61b1-49c5-8648-d71165b62469-lib-modules\") pod \"tuned-hsv57\" (UID: \"17c40464-61b1-49c5-8648-d71165b62469\") " pod="openshift-cluster-node-tuning-operator/tuned-hsv57" Jul 22 00:26:57.147076 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.144741 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/947d311a-ec4b-42e7-a0fb-ee5edc0caf8d-systemd-units\") pod \"ovnkube-node-hr5fn\" (UID: \"947d311a-ec4b-42e7-a0fb-ee5edc0caf8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5fn" Jul 22 00:26:57.147076 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.144766 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/947d311a-ec4b-42e7-a0fb-ee5edc0caf8d-host-slash\") pod \"ovnkube-node-hr5fn\" (UID: \"947d311a-ec4b-42e7-a0fb-ee5edc0caf8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5fn" Jul 22 00:26:57.147076 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.144770 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/17c40464-61b1-49c5-8648-d71165b62469-etc-sysctl-d\") pod \"tuned-hsv57\" (UID: \"17c40464-61b1-49c5-8648-d71165b62469\") " pod="openshift-cluster-node-tuning-operator/tuned-hsv57" Jul 22 00:26:57.147076 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.144807 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/947d311a-ec4b-42e7-a0fb-ee5edc0caf8d-host-slash\") pod \"ovnkube-node-hr5fn\" (UID: \"947d311a-ec4b-42e7-a0fb-ee5edc0caf8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5fn" Jul 22 00:26:57.147076 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.144815 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/17c40464-61b1-49c5-8648-d71165b62469-etc-kubernetes\") pod \"tuned-hsv57\" (UID: \"17c40464-61b1-49c5-8648-d71165b62469\") " pod="openshift-cluster-node-tuning-operator/tuned-hsv57" Jul 22 00:26:57.147076 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.144822 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/352175c8-0733-4744-be74-676a559bb4cd-socket-dir\") pod \"aws-ebs-csi-driver-node-sjxn9\" (UID: \"352175c8-0733-4744-be74-676a559bb4cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjxn9" Jul 22 00:26:57.147076 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.144848 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/352175c8-0733-4744-be74-676a559bb4cd-etc-selinux\") pod \"aws-ebs-csi-driver-node-sjxn9\" (UID: \"352175c8-0733-4744-be74-676a559bb4cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjxn9" Jul 22 00:26:57.147076 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.144864 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/947d311a-ec4b-42e7-a0fb-ee5edc0caf8d-systemd-units\") pod \"ovnkube-node-hr5fn\" (UID: \"947d311a-ec4b-42e7-a0fb-ee5edc0caf8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5fn" Jul 22 00:26:57.147076 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.144874 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bh4l\" (UniqueName: \"kubernetes.io/projected/5e5e6032-d779-43fe-8dac-33b4123b50bf-kube-api-access-5bh4l\") pod \"iptables-alerter-ctbfz\" (UID: \"5e5e6032-d779-43fe-8dac-33b4123b50bf\") " pod="openshift-network-operator/iptables-alerter-ctbfz" Jul 22 00:26:57.147076 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.144889 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/947d311a-ec4b-42e7-a0fb-ee5edc0caf8d-run-openvswitch\") pod \"ovnkube-node-hr5fn\" (UID: \"947d311a-ec4b-42e7-a0fb-ee5edc0caf8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5fn" Jul 22 00:26:57.147076 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.144914 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/947d311a-ec4b-42e7-a0fb-ee5edc0caf8d-node-log\") pod \"ovnkube-node-hr5fn\" (UID: \"947d311a-ec4b-42e7-a0fb-ee5edc0caf8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5fn" Jul 22 00:26:57.147076 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.144925 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/17c40464-61b1-49c5-8648-d71165b62469-lib-modules\") pod \"tuned-hsv57\" (UID: \"17c40464-61b1-49c5-8648-d71165b62469\") " pod="openshift-cluster-node-tuning-operator/tuned-hsv57" Jul 22 00:26:57.147076 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.144956 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/947d311a-ec4b-42e7-a0fb-ee5edc0caf8d-ovn-node-metrics-cert\") pod \"ovnkube-node-hr5fn\" (UID: \"947d311a-ec4b-42e7-a0fb-ee5edc0caf8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5fn" Jul 22 00:26:57.147076 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.144987 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/352175c8-0733-4744-be74-676a559bb4cd-etc-selinux\") pod \"aws-ebs-csi-driver-node-sjxn9\" (UID: \"352175c8-0733-4744-be74-676a559bb4cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjxn9" Jul 22 00:26:57.147858 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.145023 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/352175c8-0733-4744-be74-676a559bb4cd-socket-dir\") pod \"aws-ebs-csi-driver-node-sjxn9\" (UID: \"352175c8-0733-4744-be74-676a559bb4cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjxn9" Jul 22 00:26:57.147858 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.145027 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/947d311a-ec4b-42e7-a0fb-ee5edc0caf8d-run-openvswitch\") pod \"ovnkube-node-hr5fn\" (UID: \"947d311a-ec4b-42e7-a0fb-ee5edc0caf8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5fn" Jul 22 00:26:57.147858 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.145068 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/947d311a-ec4b-42e7-a0fb-ee5edc0caf8d-node-log\") pod \"ovnkube-node-hr5fn\" (UID: \"947d311a-ec4b-42e7-a0fb-ee5edc0caf8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5fn" Jul 22 00:26:57.147858 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.146128 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/17c40464-61b1-49c5-8648-d71165b62469-etc-tuned\") pod \"tuned-hsv57\" (UID: \"17c40464-61b1-49c5-8648-d71165b62469\") " pod="openshift-cluster-node-tuning-operator/tuned-hsv57" Jul 22 00:26:57.147858 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.147044 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/17c40464-61b1-49c5-8648-d71165b62469-tmp\") pod \"tuned-hsv57\" (UID: \"17c40464-61b1-49c5-8648-d71165b62469\") " pod="openshift-cluster-node-tuning-operator/tuned-hsv57" Jul 22 00:26:57.147858 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.147296 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/947d311a-ec4b-42e7-a0fb-ee5edc0caf8d-ovn-node-metrics-cert\") pod \"ovnkube-node-hr5fn\" (UID: \"947d311a-ec4b-42e7-a0fb-ee5edc0caf8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5fn" Jul 22 00:26:57.150885 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:26:57.150856 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 00:26:57.150885 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:26:57.150880 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 00:26:57.151046 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:26:57.150893 2574 projected.go:196] Error preparing data for projected volume kube-api-access-pkrgp for pod openshift-network-diagnostics/network-check-target-6k9fm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 00:26:57.151046 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:26:57.150997 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bbfe455c-9568-48ba-8007-92b4142b845f-kube-api-access-pkrgp podName:bbfe455c-9568-48ba-8007-92b4142b845f nodeName:}" failed. No retries permitted until 2026-07-22 00:26:57.650981688 +0000 UTC m=+3.007739560 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-pkrgp" (UniqueName: "kubernetes.io/projected/bbfe455c-9568-48ba-8007-92b4142b845f-kube-api-access-pkrgp") pod "network-check-target-6k9fm" (UID: "bbfe455c-9568-48ba-8007-92b4142b845f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 00:26:57.152954 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.152392 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfgd2\" (UniqueName: \"kubernetes.io/projected/17c40464-61b1-49c5-8648-d71165b62469-kube-api-access-bfgd2\") pod \"tuned-hsv57\" (UID: \"17c40464-61b1-49c5-8648-d71165b62469\") " pod="openshift-cluster-node-tuning-operator/tuned-hsv57" Jul 22 00:26:57.152954 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.152911 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkkxw\" (UniqueName: \"kubernetes.io/projected/352175c8-0733-4744-be74-676a559bb4cd-kube-api-access-qkkxw\") pod \"aws-ebs-csi-driver-node-sjxn9\" (UID: \"352175c8-0733-4744-be74-676a559bb4cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjxn9" Jul 22 00:26:57.153364 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.153338 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bh4l\" (UniqueName: \"kubernetes.io/projected/5e5e6032-d779-43fe-8dac-33b4123b50bf-kube-api-access-5bh4l\") pod \"iptables-alerter-ctbfz\" (UID: \"5e5e6032-d779-43fe-8dac-33b4123b50bf\") " pod="openshift-network-operator/iptables-alerter-ctbfz" Jul 22 00:26:57.154178 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.154144 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lggx\" (UniqueName: \"kubernetes.io/projected/f36a57f8-6e88-43d3-b572-8adf3b0ddb7c-kube-api-access-2lggx\") pod \"network-metrics-daemon-5xz72\" (UID: \"f36a57f8-6e88-43d3-b572-8adf3b0ddb7c\") " pod="openshift-multus/network-metrics-daemon-5xz72" Jul 22 00:26:57.154893 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.154872 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqrd4\" (UniqueName: \"kubernetes.io/projected/947d311a-ec4b-42e7-a0fb-ee5edc0caf8d-kube-api-access-fqrd4\") pod \"ovnkube-node-hr5fn\" (UID: \"947d311a-ec4b-42e7-a0fb-ee5edc0caf8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5fn" Jul 22 00:26:57.155296 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.155276 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb697\" (UniqueName: \"kubernetes.io/projected/35037e44-306a-4ec1-9937-d3f4b3b7891a-kube-api-access-vb697\") pod \"node-resolver-zqvbx\" (UID: \"35037e44-306a-4ec1-9937-d3f4b3b7891a\") " pod="openshift-dns/node-resolver-zqvbx" Jul 22 00:26:57.232647 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.232326 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7wkvg" Jul 22 00:26:57.239009 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.238901 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zk4rp" Jul 22 00:26:57.246558 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.246537 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-v2ksf" Jul 22 00:26:57.247485 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:57.247459 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44c9d8f9_4c85_4dfa_8572_2feb0f653720.slice/crio-ce544cfaf8e884b75e10d889b661f3c769ee6fade8548a6edb0424964c08e020 WatchSource:0}: Error finding container ce544cfaf8e884b75e10d889b661f3c769ee6fade8548a6edb0424964c08e020: Status 404 returned error can't find the container with id ce544cfaf8e884b75e10d889b661f3c769ee6fade8548a6edb0424964c08e020 Jul 22 00:26:57.250312 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.250295 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-mhnl2" Jul 22 00:26:57.256041 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.255854 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zqvbx" Jul 22 00:26:57.260721 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:57.260229 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod077d5538_1eab_48a5_97f9_3a623a4b63c9.slice/crio-2ff6a794af8d35046523bfd71a60e805892a231c0c39c9f93faf70cd517cc411 WatchSource:0}: Error finding container 2ff6a794af8d35046523bfd71a60e805892a231c0c39c9f93faf70cd517cc411: Status 404 returned error can't find the container with id 2ff6a794af8d35046523bfd71a60e805892a231c0c39c9f93faf70cd517cc411 Jul 22 00:26:57.265495 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.261844 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjxn9" Jul 22 00:26:57.267701 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.267406 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hsv57" Jul 22 00:26:57.268000 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:57.267974 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b9207f1_4d6c_447c_9b24_c8cba3a2f21e.slice/crio-a7c9cfa9fef51c7b56b411130ea56d434f6d945807be3da5b929863f3c3485e3 WatchSource:0}: Error finding container a7c9cfa9fef51c7b56b411130ea56d434f6d945807be3da5b929863f3c3485e3: Status 404 returned error can't find the container with id a7c9cfa9fef51c7b56b411130ea56d434f6d945807be3da5b929863f3c3485e3 Jul 22 00:26:57.272562 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:57.272527 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35037e44_306a_4ec1_9937_d3f4b3b7891a.slice/crio-b31ff289866020f8e868c3bc1697fbf63c2ac3beec2400f4feddef1f37e334e7 WatchSource:0}: Error finding container b31ff289866020f8e868c3bc1697fbf63c2ac3beec2400f4feddef1f37e334e7: Status 404 returned error can't find the container with id b31ff289866020f8e868c3bc1697fbf63c2ac3beec2400f4feddef1f37e334e7 Jul 22 00:26:57.273631 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.273575 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ctbfz" Jul 22 00:26:57.274925 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:57.274896 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod352175c8_0733_4744_be74_676a559bb4cd.slice/crio-407634bf88bee409eed4a68fd12a1bb7e1e63c8598653fa196dc2be5fec58866 WatchSource:0}: Error finding container 407634bf88bee409eed4a68fd12a1bb7e1e63c8598653fa196dc2be5fec58866: Status 404 returned error can't find the container with id 407634bf88bee409eed4a68fd12a1bb7e1e63c8598653fa196dc2be5fec58866 Jul 22 00:26:57.277898 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.277878 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hr5fn" Jul 22 00:26:57.283128 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:57.283102 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17c40464_61b1_49c5_8648_d71165b62469.slice/crio-428da7c0355fba52b4135ed85932321c86c0d5a7ccbbf9bee9f73e2d79c23cf1 WatchSource:0}: Error finding container 428da7c0355fba52b4135ed85932321c86c0d5a7ccbbf9bee9f73e2d79c23cf1: Status 404 returned error can't find the container with id 428da7c0355fba52b4135ed85932321c86c0d5a7ccbbf9bee9f73e2d79c23cf1 Jul 22 00:26:57.291229 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:57.291198 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e5e6032_d779_43fe_8dac_33b4123b50bf.slice/crio-4dd60aaac7a9dc353d80aa38319cfdaa15e12da7ae96d2c2a4fa58e5efcb3f63 WatchSource:0}: Error finding container 4dd60aaac7a9dc353d80aa38319cfdaa15e12da7ae96d2c2a4fa58e5efcb3f63: Status 404 returned error can't find the container with id 4dd60aaac7a9dc353d80aa38319cfdaa15e12da7ae96d2c2a4fa58e5efcb3f63 Jul 22 00:26:57.295899 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:26:57.295872 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod947d311a_ec4b_42e7_a0fb_ee5edc0caf8d.slice/crio-018f6920748d131290fdd1ba3419a67d451a62b4aba09b201e9f4bcd5777071f WatchSource:0}: Error finding container 018f6920748d131290fdd1ba3419a67d451a62b4aba09b201e9f4bcd5777071f: Status 404 returned error can't find the container with id 018f6920748d131290fdd1ba3419a67d451a62b4aba09b201e9f4bcd5777071f Jul 22 00:26:57.648942 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.648858 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f36a57f8-6e88-43d3-b572-8adf3b0ddb7c-metrics-certs\") pod \"network-metrics-daemon-5xz72\" (UID: \"f36a57f8-6e88-43d3-b572-8adf3b0ddb7c\") " pod="openshift-multus/network-metrics-daemon-5xz72" Jul 22 00:26:57.649100 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:26:57.649008 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 00:26:57.649100 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.649025 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/08b6ba5c-7b7f-4cbd-a6d9-584bc9a829fd-original-pull-secret\") pod \"global-pull-secret-syncer-sl6b2\" (UID: \"08b6ba5c-7b7f-4cbd-a6d9-584bc9a829fd\") " pod="kube-system/global-pull-secret-syncer-sl6b2" Jul 22 00:26:57.649100 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:26:57.649079 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f36a57f8-6e88-43d3-b572-8adf3b0ddb7c-metrics-certs podName:f36a57f8-6e88-43d3-b572-8adf3b0ddb7c nodeName:}" failed. No retries permitted until 2026-07-22 00:26:58.64905853 +0000 UTC m=+4.005816406 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f36a57f8-6e88-43d3-b572-8adf3b0ddb7c-metrics-certs") pod "network-metrics-daemon-5xz72" (UID: "f36a57f8-6e88-43d3-b572-8adf3b0ddb7c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 00:26:57.649252 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:26:57.649136 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 00:26:57.649252 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:26:57.649178 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08b6ba5c-7b7f-4cbd-a6d9-584bc9a829fd-original-pull-secret podName:08b6ba5c-7b7f-4cbd-a6d9-584bc9a829fd nodeName:}" failed. No retries permitted until 2026-07-22 00:26:58.649167472 +0000 UTC m=+4.005925331 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/08b6ba5c-7b7f-4cbd-a6d9-584bc9a829fd-original-pull-secret") pod "global-pull-secret-syncer-sl6b2" (UID: "08b6ba5c-7b7f-4cbd-a6d9-584bc9a829fd") : object "kube-system"/"original-pull-secret" not registered Jul 22 00:26:57.751825 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.751791 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkrgp\" (UniqueName: \"kubernetes.io/projected/bbfe455c-9568-48ba-8007-92b4142b845f-kube-api-access-pkrgp\") pod \"network-check-target-6k9fm\" (UID: \"bbfe455c-9568-48ba-8007-92b4142b845f\") " pod="openshift-network-diagnostics/network-check-target-6k9fm" Jul 22 00:26:57.752007 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:26:57.751989 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 00:26:57.752007 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:26:57.752007 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 00:26:57.752118 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:26:57.752020 2574 projected.go:196] Error preparing data for projected volume kube-api-access-pkrgp for pod openshift-network-diagnostics/network-check-target-6k9fm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 00:26:57.752118 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:26:57.752075 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bbfe455c-9568-48ba-8007-92b4142b845f-kube-api-access-pkrgp podName:bbfe455c-9568-48ba-8007-92b4142b845f nodeName:}" failed. No retries permitted until 2026-07-22 00:26:58.752058192 +0000 UTC m=+4.108816050 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-pkrgp" (UniqueName: "kubernetes.io/projected/bbfe455c-9568-48ba-8007-92b4142b845f-kube-api-access-pkrgp") pod "network-check-target-6k9fm" (UID: "bbfe455c-9568-48ba-8007-92b4142b845f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 00:26:57.977113 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.977003 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-21 00:21:56 +0000 UTC" deadline="2028-03-21 05:19:02.048224882 +0000 UTC" Jul 22 00:26:57.977113 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:57.977061 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14596h52m4.071166974s" Jul 22 00:26:58.083107 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:58.083050 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5xz72" Jul 22 00:26:58.083283 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:26:58.083201 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5xz72" podUID="f36a57f8-6e88-43d3-b572-8adf3b0ddb7c" Jul 22 00:26:58.083638 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:58.083617 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sl6b2" Jul 22 00:26:58.083734 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:26:58.083717 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sl6b2" podUID="08b6ba5c-7b7f-4cbd-a6d9-584bc9a829fd" Jul 22 00:26:58.093533 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:58.093503 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7wkvg" event={"ID":"44c9d8f9-4c85-4dfa-8572-2feb0f653720","Type":"ContainerStarted","Data":"ce544cfaf8e884b75e10d889b661f3c769ee6fade8548a6edb0424964c08e020"} Jul 22 00:26:58.094991 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:58.094965 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ctbfz" event={"ID":"5e5e6032-d779-43fe-8dac-33b4123b50bf","Type":"ContainerStarted","Data":"4dd60aaac7a9dc353d80aa38319cfdaa15e12da7ae96d2c2a4fa58e5efcb3f63"} Jul 22 00:26:58.096622 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:58.096585 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hsv57" event={"ID":"17c40464-61b1-49c5-8648-d71165b62469","Type":"ContainerStarted","Data":"428da7c0355fba52b4135ed85932321c86c0d5a7ccbbf9bee9f73e2d79c23cf1"} Jul 22 00:26:58.098368 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:58.098280 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zqvbx" event={"ID":"35037e44-306a-4ec1-9937-d3f4b3b7891a","Type":"ContainerStarted","Data":"b31ff289866020f8e868c3bc1697fbf63c2ac3beec2400f4feddef1f37e334e7"} Jul 22 00:26:58.100508 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:58.100481 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v2ksf" event={"ID":"077d5538-1eab-48a5-97f9-3a623a4b63c9","Type":"ContainerStarted","Data":"2ff6a794af8d35046523bfd71a60e805892a231c0c39c9f93faf70cd517cc411"} Jul 22 00:26:58.102954 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:58.102930 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zk4rp" event={"ID":"d2fac953-0d0d-48cd-be69-240d4d16dd49","Type":"ContainerStarted","Data":"f97218793160dae9ce1baa0c88b0b98d09113c9be399b346f447d200e7a63a67"} Jul 22 00:26:58.104547 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:58.104433 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hr5fn" event={"ID":"947d311a-ec4b-42e7-a0fb-ee5edc0caf8d","Type":"ContainerStarted","Data":"018f6920748d131290fdd1ba3419a67d451a62b4aba09b201e9f4bcd5777071f"} Jul 22 00:26:58.105691 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:58.105629 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjxn9" event={"ID":"352175c8-0733-4744-be74-676a559bb4cd","Type":"ContainerStarted","Data":"407634bf88bee409eed4a68fd12a1bb7e1e63c8598653fa196dc2be5fec58866"} Jul 22 00:26:58.108001 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:58.107954 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-mhnl2" event={"ID":"9b9207f1-4d6c-447c-9b24-c8cba3a2f21e","Type":"ContainerStarted","Data":"a7c9cfa9fef51c7b56b411130ea56d434f6d945807be3da5b929863f3c3485e3"} Jul 22 00:26:58.660994 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:58.660924 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/08b6ba5c-7b7f-4cbd-a6d9-584bc9a829fd-original-pull-secret\") pod \"global-pull-secret-syncer-sl6b2\" (UID: \"08b6ba5c-7b7f-4cbd-a6d9-584bc9a829fd\") " pod="kube-system/global-pull-secret-syncer-sl6b2" Jul 22 00:26:58.661175 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:58.661049 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f36a57f8-6e88-43d3-b572-8adf3b0ddb7c-metrics-certs\") pod \"network-metrics-daemon-5xz72\" (UID: \"f36a57f8-6e88-43d3-b572-8adf3b0ddb7c\") " pod="openshift-multus/network-metrics-daemon-5xz72" Jul 22 00:26:58.661175 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:26:58.661118 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 00:26:58.661175 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:26:58.661168 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 00:26:58.661316 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:26:58.661205 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08b6ba5c-7b7f-4cbd-a6d9-584bc9a829fd-original-pull-secret podName:08b6ba5c-7b7f-4cbd-a6d9-584bc9a829fd nodeName:}" failed. No retries permitted until 2026-07-22 00:27:00.661183424 +0000 UTC m=+6.017941296 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/08b6ba5c-7b7f-4cbd-a6d9-584bc9a829fd-original-pull-secret") pod "global-pull-secret-syncer-sl6b2" (UID: "08b6ba5c-7b7f-4cbd-a6d9-584bc9a829fd") : object "kube-system"/"original-pull-secret" not registered Jul 22 00:26:58.661316 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:26:58.661229 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f36a57f8-6e88-43d3-b572-8adf3b0ddb7c-metrics-certs podName:f36a57f8-6e88-43d3-b572-8adf3b0ddb7c nodeName:}" failed. No retries permitted until 2026-07-22 00:27:00.661218654 +0000 UTC m=+6.017976514 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f36a57f8-6e88-43d3-b572-8adf3b0ddb7c-metrics-certs") pod "network-metrics-daemon-5xz72" (UID: "f36a57f8-6e88-43d3-b572-8adf3b0ddb7c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 00:26:58.762020 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:58.761984 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkrgp\" (UniqueName: \"kubernetes.io/projected/bbfe455c-9568-48ba-8007-92b4142b845f-kube-api-access-pkrgp\") pod \"network-check-target-6k9fm\" (UID: \"bbfe455c-9568-48ba-8007-92b4142b845f\") " pod="openshift-network-diagnostics/network-check-target-6k9fm" Jul 22 00:26:58.762689 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:26:58.762231 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 00:26:58.762689 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:26:58.762258 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 00:26:58.762689 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:26:58.762271 2574 projected.go:196] Error preparing data for projected volume kube-api-access-pkrgp for pod openshift-network-diagnostics/network-check-target-6k9fm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 00:26:58.762689 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:26:58.762336 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bbfe455c-9568-48ba-8007-92b4142b845f-kube-api-access-pkrgp podName:bbfe455c-9568-48ba-8007-92b4142b845f nodeName:}" failed. No retries permitted until 2026-07-22 00:27:00.762316605 +0000 UTC m=+6.119074474 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-pkrgp" (UniqueName: "kubernetes.io/projected/bbfe455c-9568-48ba-8007-92b4142b845f-kube-api-access-pkrgp") pod "network-check-target-6k9fm" (UID: "bbfe455c-9568-48ba-8007-92b4142b845f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 00:26:58.977634 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:58.977525 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-21 00:21:56 +0000 UTC" deadline="2028-01-28 05:18:36.083471365 +0000 UTC" Jul 22 00:26:58.977634 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:58.977570 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13324h51m37.105905934s" Jul 22 00:26:59.083107 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:26:59.082591 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6k9fm" Jul 22 00:26:59.083107 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:26:59.082741 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6k9fm" podUID="bbfe455c-9568-48ba-8007-92b4142b845f" Jul 22 00:27:00.082489 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:00.082456 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sl6b2" Jul 22 00:27:00.083046 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:00.082594 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sl6b2" podUID="08b6ba5c-7b7f-4cbd-a6d9-584bc9a829fd" Jul 22 00:27:00.083046 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:00.082680 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5xz72" Jul 22 00:27:00.083046 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:00.082769 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5xz72" podUID="f36a57f8-6e88-43d3-b572-8adf3b0ddb7c" Jul 22 00:27:00.681926 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:00.681890 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f36a57f8-6e88-43d3-b572-8adf3b0ddb7c-metrics-certs\") pod \"network-metrics-daemon-5xz72\" (UID: \"f36a57f8-6e88-43d3-b572-8adf3b0ddb7c\") " pod="openshift-multus/network-metrics-daemon-5xz72" Jul 22 00:27:00.682101 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:00.682011 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/08b6ba5c-7b7f-4cbd-a6d9-584bc9a829fd-original-pull-secret\") pod \"global-pull-secret-syncer-sl6b2\" (UID: \"08b6ba5c-7b7f-4cbd-a6d9-584bc9a829fd\") " pod="kube-system/global-pull-secret-syncer-sl6b2" Jul 22 00:27:00.682962 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:00.682205 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 00:27:00.682962 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:00.682276 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08b6ba5c-7b7f-4cbd-a6d9-584bc9a829fd-original-pull-secret podName:08b6ba5c-7b7f-4cbd-a6d9-584bc9a829fd nodeName:}" failed. No retries permitted until 2026-07-22 00:27:04.682257032 +0000 UTC m=+10.039014891 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/08b6ba5c-7b7f-4cbd-a6d9-584bc9a829fd-original-pull-secret") pod "global-pull-secret-syncer-sl6b2" (UID: "08b6ba5c-7b7f-4cbd-a6d9-584bc9a829fd") : object "kube-system"/"original-pull-secret" not registered Jul 22 00:27:00.682962 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:00.682660 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 00:27:00.682962 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:00.682707 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f36a57f8-6e88-43d3-b572-8adf3b0ddb7c-metrics-certs podName:f36a57f8-6e88-43d3-b572-8adf3b0ddb7c nodeName:}" failed. No retries permitted until 2026-07-22 00:27:04.682691334 +0000 UTC m=+10.039449195 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f36a57f8-6e88-43d3-b572-8adf3b0ddb7c-metrics-certs") pod "network-metrics-daemon-5xz72" (UID: "f36a57f8-6e88-43d3-b572-8adf3b0ddb7c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 00:27:00.782828 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:00.782780 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkrgp\" (UniqueName: \"kubernetes.io/projected/bbfe455c-9568-48ba-8007-92b4142b845f-kube-api-access-pkrgp\") pod \"network-check-target-6k9fm\" (UID: \"bbfe455c-9568-48ba-8007-92b4142b845f\") " pod="openshift-network-diagnostics/network-check-target-6k9fm" Jul 22 00:27:00.782994 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:00.782982 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 00:27:00.783049 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:00.783003 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 00:27:00.783049 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:00.783016 2574 projected.go:196] Error preparing data for projected volume kube-api-access-pkrgp for pod openshift-network-diagnostics/network-check-target-6k9fm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 00:27:00.783147 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:00.783075 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bbfe455c-9568-48ba-8007-92b4142b845f-kube-api-access-pkrgp podName:bbfe455c-9568-48ba-8007-92b4142b845f nodeName:}" failed. No retries permitted until 2026-07-22 00:27:04.783056871 +0000 UTC m=+10.139814730 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-pkrgp" (UniqueName: "kubernetes.io/projected/bbfe455c-9568-48ba-8007-92b4142b845f-kube-api-access-pkrgp") pod "network-check-target-6k9fm" (UID: "bbfe455c-9568-48ba-8007-92b4142b845f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 00:27:01.082957 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:01.082847 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6k9fm" Jul 22 00:27:01.083357 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:01.082974 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6k9fm" podUID="bbfe455c-9568-48ba-8007-92b4142b845f" Jul 22 00:27:02.083175 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:02.082686 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sl6b2" Jul 22 00:27:02.083175 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:02.082814 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sl6b2" podUID="08b6ba5c-7b7f-4cbd-a6d9-584bc9a829fd" Jul 22 00:27:02.083852 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:02.083197 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5xz72" Jul 22 00:27:02.083852 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:02.083303 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5xz72" podUID="f36a57f8-6e88-43d3-b572-8adf3b0ddb7c" Jul 22 00:27:03.083118 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:03.083032 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6k9fm" Jul 22 00:27:03.083269 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:03.083159 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6k9fm" podUID="bbfe455c-9568-48ba-8007-92b4142b845f" Jul 22 00:27:04.082687 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:04.082638 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5xz72" Jul 22 00:27:04.082867 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:04.082790 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5xz72" podUID="f36a57f8-6e88-43d3-b572-8adf3b0ddb7c" Jul 22 00:27:04.083073 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:04.082644 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sl6b2" Jul 22 00:27:04.083202 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:04.083125 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sl6b2" podUID="08b6ba5c-7b7f-4cbd-a6d9-584bc9a829fd" Jul 22 00:27:04.714622 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:04.713944 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f36a57f8-6e88-43d3-b572-8adf3b0ddb7c-metrics-certs\") pod \"network-metrics-daemon-5xz72\" (UID: \"f36a57f8-6e88-43d3-b572-8adf3b0ddb7c\") " pod="openshift-multus/network-metrics-daemon-5xz72" Jul 22 00:27:04.714622 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:04.714003 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/08b6ba5c-7b7f-4cbd-a6d9-584bc9a829fd-original-pull-secret\") pod \"global-pull-secret-syncer-sl6b2\" (UID: \"08b6ba5c-7b7f-4cbd-a6d9-584bc9a829fd\") " pod="kube-system/global-pull-secret-syncer-sl6b2" Jul 22 00:27:04.714622 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:04.714117 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 00:27:04.714622 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:04.714182 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08b6ba5c-7b7f-4cbd-a6d9-584bc9a829fd-original-pull-secret podName:08b6ba5c-7b7f-4cbd-a6d9-584bc9a829fd nodeName:}" failed. No retries permitted until 2026-07-22 00:27:12.714160874 +0000 UTC m=+18.070918762 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/08b6ba5c-7b7f-4cbd-a6d9-584bc9a829fd-original-pull-secret") pod "global-pull-secret-syncer-sl6b2" (UID: "08b6ba5c-7b7f-4cbd-a6d9-584bc9a829fd") : object "kube-system"/"original-pull-secret" not registered Jul 22 00:27:04.714622 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:04.714550 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 00:27:04.714622 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:04.714613 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f36a57f8-6e88-43d3-b572-8adf3b0ddb7c-metrics-certs podName:f36a57f8-6e88-43d3-b572-8adf3b0ddb7c nodeName:}" failed. No retries permitted until 2026-07-22 00:27:12.714582377 +0000 UTC m=+18.071340237 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f36a57f8-6e88-43d3-b572-8adf3b0ddb7c-metrics-certs") pod "network-metrics-daemon-5xz72" (UID: "f36a57f8-6e88-43d3-b572-8adf3b0ddb7c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 00:27:04.814804 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:04.814748 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkrgp\" (UniqueName: \"kubernetes.io/projected/bbfe455c-9568-48ba-8007-92b4142b845f-kube-api-access-pkrgp\") pod \"network-check-target-6k9fm\" (UID: \"bbfe455c-9568-48ba-8007-92b4142b845f\") " pod="openshift-network-diagnostics/network-check-target-6k9fm" Jul 22 00:27:04.814998 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:04.814942 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 00:27:04.814998 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:04.814971 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 00:27:04.814998 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:04.814985 2574 projected.go:196] Error preparing data for projected volume kube-api-access-pkrgp for pod openshift-network-diagnostics/network-check-target-6k9fm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 00:27:04.815145 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:04.815087 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bbfe455c-9568-48ba-8007-92b4142b845f-kube-api-access-pkrgp podName:bbfe455c-9568-48ba-8007-92b4142b845f nodeName:}" failed. No retries permitted until 2026-07-22 00:27:12.81506703 +0000 UTC m=+18.171824907 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-pkrgp" (UniqueName: "kubernetes.io/projected/bbfe455c-9568-48ba-8007-92b4142b845f-kube-api-access-pkrgp") pod "network-check-target-6k9fm" (UID: "bbfe455c-9568-48ba-8007-92b4142b845f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 00:27:05.083911 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:05.083357 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6k9fm" Jul 22 00:27:05.083911 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:05.083480 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6k9fm" podUID="bbfe455c-9568-48ba-8007-92b4142b845f" Jul 22 00:27:06.083392 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:06.082893 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5xz72" Jul 22 00:27:06.083392 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:06.083044 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5xz72" podUID="f36a57f8-6e88-43d3-b572-8adf3b0ddb7c" Jul 22 00:27:06.083392 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:06.083065 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sl6b2" Jul 22 00:27:06.083392 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:06.083168 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sl6b2" podUID="08b6ba5c-7b7f-4cbd-a6d9-584bc9a829fd" Jul 22 00:27:07.083192 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:07.083164 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6k9fm" Jul 22 00:27:07.083366 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:07.083271 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6k9fm" podUID="bbfe455c-9568-48ba-8007-92b4142b845f" Jul 22 00:27:08.082594 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:08.082515 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5xz72" Jul 22 00:27:08.083038 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:08.082653 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5xz72" podUID="f36a57f8-6e88-43d3-b572-8adf3b0ddb7c" Jul 22 00:27:08.083038 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:08.082699 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sl6b2" Jul 22 00:27:08.083038 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:08.082788 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sl6b2" podUID="08b6ba5c-7b7f-4cbd-a6d9-584bc9a829fd" Jul 22 00:27:09.083068 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:09.083030 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6k9fm" Jul 22 00:27:09.083514 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:09.083449 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6k9fm" podUID="bbfe455c-9568-48ba-8007-92b4142b845f" Jul 22 00:27:10.082990 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:10.082952 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sl6b2" Jul 22 00:27:10.083179 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:10.082954 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5xz72" Jul 22 00:27:10.083179 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:10.083081 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sl6b2" podUID="08b6ba5c-7b7f-4cbd-a6d9-584bc9a829fd" Jul 22 00:27:10.083535 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:10.083194 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5xz72" podUID="f36a57f8-6e88-43d3-b572-8adf3b0ddb7c" Jul 22 00:27:11.082704 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:11.082671 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6k9fm" Jul 22 00:27:11.082888 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:11.082790 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6k9fm" podUID="bbfe455c-9568-48ba-8007-92b4142b845f" Jul 22 00:27:12.083180 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:12.083141 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sl6b2" Jul 22 00:27:12.083700 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:12.083153 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5xz72" Jul 22 00:27:12.083700 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:12.083287 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sl6b2" podUID="08b6ba5c-7b7f-4cbd-a6d9-584bc9a829fd" Jul 22 00:27:12.083700 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:12.083395 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5xz72" podUID="f36a57f8-6e88-43d3-b572-8adf3b0ddb7c" Jul 22 00:27:12.773289 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:12.773250 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f36a57f8-6e88-43d3-b572-8adf3b0ddb7c-metrics-certs\") pod \"network-metrics-daemon-5xz72\" (UID: \"f36a57f8-6e88-43d3-b572-8adf3b0ddb7c\") " pod="openshift-multus/network-metrics-daemon-5xz72" Jul 22 00:27:12.773461 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:12.773311 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/08b6ba5c-7b7f-4cbd-a6d9-584bc9a829fd-original-pull-secret\") pod \"global-pull-secret-syncer-sl6b2\" (UID: \"08b6ba5c-7b7f-4cbd-a6d9-584bc9a829fd\") " pod="kube-system/global-pull-secret-syncer-sl6b2" Jul 22 00:27:12.773461 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:12.773415 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 00:27:12.773461 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:12.773431 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 00:27:12.773592 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:12.773486 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f36a57f8-6e88-43d3-b572-8adf3b0ddb7c-metrics-certs podName:f36a57f8-6e88-43d3-b572-8adf3b0ddb7c nodeName:}" failed. No retries permitted until 2026-07-22 00:27:28.773468092 +0000 UTC m=+34.130225946 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f36a57f8-6e88-43d3-b572-8adf3b0ddb7c-metrics-certs") pod "network-metrics-daemon-5xz72" (UID: "f36a57f8-6e88-43d3-b572-8adf3b0ddb7c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 00:27:12.773592 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:12.773508 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08b6ba5c-7b7f-4cbd-a6d9-584bc9a829fd-original-pull-secret podName:08b6ba5c-7b7f-4cbd-a6d9-584bc9a829fd nodeName:}" failed. No retries permitted until 2026-07-22 00:27:28.773498118 +0000 UTC m=+34.130255978 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/08b6ba5c-7b7f-4cbd-a6d9-584bc9a829fd-original-pull-secret") pod "global-pull-secret-syncer-sl6b2" (UID: "08b6ba5c-7b7f-4cbd-a6d9-584bc9a829fd") : object "kube-system"/"original-pull-secret" not registered Jul 22 00:27:12.874263 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:12.874228 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkrgp\" (UniqueName: \"kubernetes.io/projected/bbfe455c-9568-48ba-8007-92b4142b845f-kube-api-access-pkrgp\") pod \"network-check-target-6k9fm\" (UID: \"bbfe455c-9568-48ba-8007-92b4142b845f\") " pod="openshift-network-diagnostics/network-check-target-6k9fm" Jul 22 00:27:12.874430 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:12.874391 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 00:27:12.874430 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:12.874419 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 00:27:12.874430 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:12.874430 2574 projected.go:196] Error preparing data for projected volume kube-api-access-pkrgp for pod openshift-network-diagnostics/network-check-target-6k9fm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 00:27:12.874548 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:12.874482 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bbfe455c-9568-48ba-8007-92b4142b845f-kube-api-access-pkrgp podName:bbfe455c-9568-48ba-8007-92b4142b845f nodeName:}" failed. No retries permitted until 2026-07-22 00:27:28.87446646 +0000 UTC m=+34.231224327 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-pkrgp" (UniqueName: "kubernetes.io/projected/bbfe455c-9568-48ba-8007-92b4142b845f-kube-api-access-pkrgp") pod "network-check-target-6k9fm" (UID: "bbfe455c-9568-48ba-8007-92b4142b845f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 00:27:13.082732 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:13.082641 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6k9fm" Jul 22 00:27:13.082891 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:13.082785 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6k9fm" podUID="bbfe455c-9568-48ba-8007-92b4142b845f" Jul 22 00:27:14.083461 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:14.083404 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sl6b2" Jul 22 00:27:14.083978 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:14.083404 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5xz72" Jul 22 00:27:14.083978 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:14.083535 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sl6b2" podUID="08b6ba5c-7b7f-4cbd-a6d9-584bc9a829fd" Jul 22 00:27:14.083978 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:14.083651 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5xz72" podUID="f36a57f8-6e88-43d3-b572-8adf3b0ddb7c" Jul 22 00:27:15.083721 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:15.083686 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6k9fm" Jul 22 00:27:15.084204 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:15.083807 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6k9fm" podUID="bbfe455c-9568-48ba-8007-92b4142b845f" Jul 22 00:27:16.082576 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:16.082538 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5xz72" Jul 22 00:27:16.082750 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:16.082590 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sl6b2" Jul 22 00:27:16.082750 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:16.082679 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5xz72" podUID="f36a57f8-6e88-43d3-b572-8adf3b0ddb7c" Jul 22 00:27:16.082831 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:16.082803 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sl6b2" podUID="08b6ba5c-7b7f-4cbd-a6d9-584bc9a829fd" Jul 22 00:27:17.083707 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:17.083488 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6k9fm" Jul 22 00:27:17.084312 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:17.083784 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6k9fm" podUID="bbfe455c-9568-48ba-8007-92b4142b845f" Jul 22 00:27:17.152680 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:17.152648 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hsv57" event={"ID":"17c40464-61b1-49c5-8648-d71165b62469","Type":"ContainerStarted","Data":"9af33fc3d6ebc43728302d88db709ce51ecf3ba27d9735dfb88ede5583a739d6"} Jul 22 00:27:17.158153 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:17.156479 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-192.ec2.internal" event={"ID":"5c5847e7e75764b70e99faa3e06c32f2","Type":"ContainerStarted","Data":"85c2ebf935ea8fcbb810d0b5ac2f82d0f9ac11b1becc9f9e842ae3eca6c20a6b"} Jul 22 00:27:17.160581 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:17.160461 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hr5fn_947d311a-ec4b-42e7-a0fb-ee5edc0caf8d/ovn-acl-logging/0.log" Jul 22 00:27:17.162375 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:17.162346 2574 generic.go:356] "Generic (PLEG): container finished" podID="947d311a-ec4b-42e7-a0fb-ee5edc0caf8d" containerID="d72287f5ea018bd7a3419502aefcff27eb0634d417fc7d567aa7d86b919e09ee" exitCode=1 Jul 22 00:27:17.162955 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:17.162386 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hr5fn" event={"ID":"947d311a-ec4b-42e7-a0fb-ee5edc0caf8d","Type":"ContainerStarted","Data":"05c4d34fc7a0a4bb4868d93ed76dca478101931a5fd32ec55f122dc4612293e7"} Jul 22 00:27:17.162955 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:17.162656 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hr5fn" event={"ID":"947d311a-ec4b-42e7-a0fb-ee5edc0caf8d","Type":"ContainerStarted","Data":"8fab1bb644a9e7d956f7b9aeb5da68800f39debc3efb269ac2d544a9299bf37d"} Jul 22 00:27:17.162955 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:17.162726 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hr5fn" event={"ID":"947d311a-ec4b-42e7-a0fb-ee5edc0caf8d","Type":"ContainerStarted","Data":"c1359e684140f9d6f2754e3dc2bd2ba2082682f41b230e21680cc5d0cc08f566"} Jul 22 00:27:17.162955 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:17.162788 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hr5fn" event={"ID":"947d311a-ec4b-42e7-a0fb-ee5edc0caf8d","Type":"ContainerDied","Data":"d72287f5ea018bd7a3419502aefcff27eb0634d417fc7d567aa7d86b919e09ee"} Jul 22 00:27:17.162955 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:17.162871 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hr5fn" event={"ID":"947d311a-ec4b-42e7-a0fb-ee5edc0caf8d","Type":"ContainerStarted","Data":"10055cdca434c85db812f0af3159582e24a79206881e4eb257852a3c04b30e41"} Jul 22 00:27:17.165227 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:17.165186 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-mhnl2" event={"ID":"9b9207f1-4d6c-447c-9b24-c8cba3a2f21e","Type":"ContainerStarted","Data":"a79cc0910df4a8b4b3c41789cbe08253bec29aa20d714c1684afe89ee8b6f696"} Jul 22 00:27:17.186583 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:17.186042 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-192.ec2.internal" podStartSLOduration=21.186022436000002 podStartE2EDuration="21.186022436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 00:26:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:27:17.185466159 +0000 UTC m=+22.542224062" watchObservedRunningTime="2026-07-22 00:27:17.186022436 +0000 UTC m=+22.542780314" Jul 22 00:27:17.186583 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:17.186289 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-hsv57" podStartSLOduration=2.967258588 podStartE2EDuration="22.186276352s" totalImagesPullingTime="19.219017764s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 00:26:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:27:17.17119212 +0000 UTC m=+22.527949998" watchObservedRunningTime="2026-07-22 00:27:17.186276352 +0000 UTC m=+22.543034230" Jul 22 00:27:18.083480 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:18.083438 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sl6b2" Jul 22 00:27:18.083765 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:18.083549 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sl6b2" podUID="08b6ba5c-7b7f-4cbd-a6d9-584bc9a829fd" Jul 22 00:27:18.083765 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:18.083615 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5xz72" Jul 22 00:27:18.084283 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:18.083826 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5xz72" podUID="f36a57f8-6e88-43d3-b572-8adf3b0ddb7c" Jul 22 00:27:18.168303 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:18.168273 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zk4rp" event={"ID":"d2fac953-0d0d-48cd-be69-240d4d16dd49","Type":"ContainerStarted","Data":"9b23e8b2c3f894283a586c92bdb5328ebfb3c084fb9be9566e94d947ac57849d"} Jul 22 00:27:18.171030 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:18.171011 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hr5fn_947d311a-ec4b-42e7-a0fb-ee5edc0caf8d/ovn-acl-logging/0.log" Jul 22 00:27:18.171376 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:18.171357 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hr5fn" event={"ID":"947d311a-ec4b-42e7-a0fb-ee5edc0caf8d","Type":"ContainerStarted","Data":"492ebd10ad6f6705d8a2098bff42ac909ea888d1abcaba09faba1165d07914a5"} Jul 22 00:27:18.172710 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:18.172687 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjxn9" event={"ID":"352175c8-0733-4744-be74-676a559bb4cd","Type":"ContainerStarted","Data":"08aaa253d0cf235c43af4ec906fa7b1c865b25df599cbde2f45698108aa70c8d"} Jul 22 00:27:18.173935 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:18.173913 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7wkvg" event={"ID":"44c9d8f9-4c85-4dfa-8572-2feb0f653720","Type":"ContainerStarted","Data":"9038146d0bc1433dd60ace62ad6ad0b60680c3bb531de85b16f1dbe174b53079"} Jul 22 00:27:18.175224 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:18.175205 2574 generic.go:356] "Generic (PLEG): container finished" podID="2951ff349dfaeb74342ef57c90cc46ad" containerID="e3334c495475c327e1fb6ba8532b3dc3d7aa6225a6a0fd7c8106f38cf5f69a2f" exitCode=0 Jul 22 00:27:18.175305 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:18.175263 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-192.ec2.internal" event={"ID":"2951ff349dfaeb74342ef57c90cc46ad","Type":"ContainerDied","Data":"e3334c495475c327e1fb6ba8532b3dc3d7aa6225a6a0fd7c8106f38cf5f69a2f"} Jul 22 00:27:18.176665 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:18.176643 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ctbfz" event={"ID":"5e5e6032-d779-43fe-8dac-33b4123b50bf","Type":"ContainerStarted","Data":"528bc5a2b1ea3eee6970018c128f77db8e70dd713df06ef5c097815347c62857"} Jul 22 00:27:18.177979 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:18.177960 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zqvbx" event={"ID":"35037e44-306a-4ec1-9937-d3f4b3b7891a","Type":"ContainerStarted","Data":"6d8cc3ef5fe9d361d9003a634866a5b9fd510094205542c1511fee9b9357d5b9"} Jul 22 00:27:18.179212 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:18.179192 2574 generic.go:356] "Generic (PLEG): container finished" podID="077d5538-1eab-48a5-97f9-3a623a4b63c9" containerID="500b58a1be0d5738a2dde6f85589dbed0286c69f645f8f84d3d58cf76ff212ac" exitCode=0 Jul 22 00:27:18.179300 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:18.179274 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v2ksf" event={"ID":"077d5538-1eab-48a5-97f9-3a623a4b63c9","Type":"ContainerDied","Data":"500b58a1be0d5738a2dde6f85589dbed0286c69f645f8f84d3d58cf76ff212ac"} Jul 22 00:27:18.183051 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:18.183018 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-mhnl2" podStartSLOduration=3.453498184 podStartE2EDuration="23.183006883s" totalImagesPullingTime="19.729508699s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 00:26:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:27:17.204230276 +0000 UTC m=+22.560988167" watchObservedRunningTime="2026-07-22 00:27:18.183006883 +0000 UTC m=+23.539764756" Jul 22 00:27:18.183457 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:18.183437 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-zk4rp" podStartSLOduration=15.737354304 podStartE2EDuration="23.183431034s" totalImagesPullingTime="7.44607673s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:26:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:27:18.183006056 +0000 UTC m=+23.539763953" watchObservedRunningTime="2026-07-22 00:27:18.183431034 +0000 UTC m=+23.540188910" Jul 22 00:27:18.197918 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:18.197855 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-ctbfz" podStartSLOduration=3.956121224 podStartE2EDuration="23.197841625s" totalImagesPullingTime="19.241720401s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 00:26:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:27:18.197207159 +0000 UTC m=+23.553965037" watchObservedRunningTime="2026-07-22 00:27:18.197841625 +0000 UTC m=+23.554599516" Jul 22 00:27:18.210653 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:18.210510 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-7wkvg" podStartSLOduration=14.922331275 podStartE2EDuration="23.210496833s" totalImagesPullingTime="8.288165558s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 00:26:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:27:18.21025041 +0000 UTC m=+23.567008287" watchObservedRunningTime="2026-07-22 00:27:18.210496833 +0000 UTC m=+23.567254710" Jul 22 00:27:18.223836 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:18.223786 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-zqvbx" podStartSLOduration=3.993997361 podStartE2EDuration="23.223773076s" totalImagesPullingTime="19.229775715s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 00:26:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:27:18.223766106 +0000 UTC m=+23.580524001" watchObservedRunningTime="2026-07-22 00:27:18.223773076 +0000 UTC m=+23.580530954" Jul 22 00:27:18.498830 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:18.498794 2574 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 22 00:27:19.003927 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:19.003803 2574 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-22T00:27:18.498812951Z","UUID":"1527c8f8-dbb8-48f6-8bba-a9e56704326b","Handler":null,"Name":"","Endpoint":""} Jul 22 00:27:19.005439 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:19.005410 2574 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 22 00:27:19.005572 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:19.005451 2574 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 22 00:27:19.083690 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:19.083537 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6k9fm" Jul 22 00:27:19.083862 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:19.083691 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6k9fm" podUID="bbfe455c-9568-48ba-8007-92b4142b845f" Jul 22 00:27:19.184297 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:19.184234 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-192.ec2.internal" event={"ID":"2951ff349dfaeb74342ef57c90cc46ad","Type":"ContainerStarted","Data":"9b7348163aefcb7b2f85424788247054fe677d082b968367a2c4048ff49f0aed"} Jul 22 00:27:19.186647 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:19.186617 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjxn9" event={"ID":"352175c8-0733-4744-be74-676a559bb4cd","Type":"ContainerStarted","Data":"e6048d94688847407de571e276af039f1ffa47f7bc7d94fed99cd1b549aa5f85"} Jul 22 00:27:19.201401 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:19.201347 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-192.ec2.internal" podStartSLOduration=23.201329879 podStartE2EDuration="23.201329879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 00:26:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:27:19.201123781 +0000 UTC m=+24.557881659" watchObservedRunningTime="2026-07-22 00:27:19.201329879 +0000 UTC m=+24.558087756" Jul 22 00:27:20.083068 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:20.082994 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5xz72" Jul 22 00:27:20.083263 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:20.082994 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sl6b2" Jul 22 00:27:20.083263 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:20.083096 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5xz72" podUID="f36a57f8-6e88-43d3-b572-8adf3b0ddb7c" Jul 22 00:27:20.083263 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:20.083186 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sl6b2" podUID="08b6ba5c-7b7f-4cbd-a6d9-584bc9a829fd" Jul 22 00:27:20.192017 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:20.191991 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hr5fn_947d311a-ec4b-42e7-a0fb-ee5edc0caf8d/ovn-acl-logging/0.log" Jul 22 00:27:20.192415 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:20.192376 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hr5fn" event={"ID":"947d311a-ec4b-42e7-a0fb-ee5edc0caf8d","Type":"ContainerStarted","Data":"51e33af528e7c834b179d9f96251257e6bd3db07484456f5ecae28a705d1c1fb"} Jul 22 00:27:20.194326 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:20.194301 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjxn9" event={"ID":"352175c8-0733-4744-be74-676a559bb4cd","Type":"ContainerStarted","Data":"91dad1b0fc1d342531f612f57cc351655bfdbcf6072e03f89af64c072f8b73bb"} Jul 22 00:27:20.215824 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:20.215773 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjxn9" podStartSLOduration=8.065335229 podStartE2EDuration="25.215752629s" totalImagesPullingTime="17.1504174s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 00:26:55 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:27:20.215121651 +0000 UTC m=+25.571879531" watchObservedRunningTime="2026-07-22 00:27:20.215752629 +0000 UTC m=+25.572510507" Jul 22 00:27:20.661538 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:20.661501 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-zk4rp" Jul 22 00:27:20.662174 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:20.662154 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-zk4rp" Jul 22 00:27:21.082919 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:21.082837 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6k9fm" Jul 22 00:27:21.083086 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:21.082969 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6k9fm" podUID="bbfe455c-9568-48ba-8007-92b4142b845f" Jul 22 00:27:21.197102 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:21.197060 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-zk4rp" Jul 22 00:27:21.197557 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:21.197502 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-zk4rp" Jul 22 00:27:22.082797 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:22.082702 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5xz72" Jul 22 00:27:22.082974 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:22.082872 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5xz72" podUID="f36a57f8-6e88-43d3-b572-8adf3b0ddb7c" Jul 22 00:27:22.082974 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:22.082880 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sl6b2" Jul 22 00:27:22.082974 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:22.082967 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sl6b2" podUID="08b6ba5c-7b7f-4cbd-a6d9-584bc9a829fd" Jul 22 00:27:23.086259 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:23.086098 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6k9fm" Jul 22 00:27:23.086668 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:23.086333 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6k9fm" podUID="bbfe455c-9568-48ba-8007-92b4142b845f" Jul 22 00:27:23.203889 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:23.203866 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hr5fn_947d311a-ec4b-42e7-a0fb-ee5edc0caf8d/ovn-acl-logging/0.log" Jul 22 00:27:23.204226 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:23.204198 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hr5fn" event={"ID":"947d311a-ec4b-42e7-a0fb-ee5edc0caf8d","Type":"ContainerStarted","Data":"c6e9fa942027bc45e436ca4b1b2f0b6f8600f608885419e2dc967c86de899abe"} Jul 22 00:27:23.204460 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:23.204442 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hr5fn" Jul 22 00:27:23.204519 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:23.204509 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hr5fn" Jul 22 00:27:23.204648 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:23.204634 2574 scope.go:117] "RemoveContainer" containerID="d72287f5ea018bd7a3419502aefcff27eb0634d417fc7d567aa7d86b919e09ee" Jul 22 00:27:23.204726 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:23.204714 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hr5fn" Jul 22 00:27:23.220269 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:23.220228 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hr5fn" Jul 22 00:27:23.220487 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:23.220471 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hr5fn" Jul 22 00:27:24.083167 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:24.083134 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sl6b2" Jul 22 00:27:24.083308 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:24.083239 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sl6b2" podUID="08b6ba5c-7b7f-4cbd-a6d9-584bc9a829fd" Jul 22 00:27:24.083308 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:24.083294 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5xz72" Jul 22 00:27:24.083410 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:24.083392 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5xz72" podUID="f36a57f8-6e88-43d3-b572-8adf3b0ddb7c" Jul 22 00:27:24.208159 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:24.208123 2574 generic.go:356] "Generic (PLEG): container finished" podID="077d5538-1eab-48a5-97f9-3a623a4b63c9" containerID="48545fd1778cb3260eee02133dcc6edaead5cfda2186b3dc0638efa0d9a4bc4f" exitCode=0 Jul 22 00:27:24.209032 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:24.208207 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v2ksf" event={"ID":"077d5538-1eab-48a5-97f9-3a623a4b63c9","Type":"ContainerDied","Data":"48545fd1778cb3260eee02133dcc6edaead5cfda2186b3dc0638efa0d9a4bc4f"} Jul 22 00:27:24.214712 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:24.214693 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hr5fn_947d311a-ec4b-42e7-a0fb-ee5edc0caf8d/ovn-acl-logging/0.log" Jul 22 00:27:24.215050 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:24.215025 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hr5fn" event={"ID":"947d311a-ec4b-42e7-a0fb-ee5edc0caf8d","Type":"ContainerStarted","Data":"bf52e500aa59e76bffcea6840da7a7f874b09e001af24ccc5d5c59ea1c42b082"} Jul 22 00:27:24.254629 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:24.254525 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-hr5fn" podStartSLOduration=9.964626712 podStartE2EDuration="29.254509426s" totalImagesPullingTime="19.289882714s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 00:26:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:27:24.253845212 +0000 UTC m=+29.610603108" watchObservedRunningTime="2026-07-22 00:27:24.254509426 +0000 UTC m=+29.611267303" Jul 22 00:27:25.083647 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:25.083434 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6k9fm" Jul 22 00:27:25.083790 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:25.083711 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6k9fm" podUID="bbfe455c-9568-48ba-8007-92b4142b845f" Jul 22 00:27:25.373777 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:25.373741 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6k9fm"] Jul 22 00:27:25.374137 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:25.373869 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6k9fm" Jul 22 00:27:25.374137 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:25.373980 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6k9fm" podUID="bbfe455c-9568-48ba-8007-92b4142b845f" Jul 22 00:27:25.374660 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:25.374637 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-sl6b2"] Jul 22 00:27:25.374777 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:25.374741 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sl6b2" Jul 22 00:27:25.374897 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:25.374872 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sl6b2" podUID="08b6ba5c-7b7f-4cbd-a6d9-584bc9a829fd" Jul 22 00:27:25.375786 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:25.375765 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5xz72"] Jul 22 00:27:25.375877 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:25.375853 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5xz72" Jul 22 00:27:25.375944 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:25.375931 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5xz72" podUID="f36a57f8-6e88-43d3-b572-8adf3b0ddb7c" Jul 22 00:27:26.221518 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:26.221485 2574 generic.go:356] "Generic (PLEG): container finished" podID="077d5538-1eab-48a5-97f9-3a623a4b63c9" containerID="2210a3680827d0d91ce1c4a04293ed45a1d5c974c69739e65f6d3d79d196cf3e" exitCode=0 Jul 22 00:27:26.221667 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:26.221540 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v2ksf" event={"ID":"077d5538-1eab-48a5-97f9-3a623a4b63c9","Type":"ContainerDied","Data":"2210a3680827d0d91ce1c4a04293ed45a1d5c974c69739e65f6d3d79d196cf3e"} Jul 22 00:27:27.086227 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:27.086193 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6k9fm" Jul 22 00:27:27.086674 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:27.086295 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sl6b2" Jul 22 00:27:27.086674 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:27.086306 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6k9fm" podUID="bbfe455c-9568-48ba-8007-92b4142b845f" Jul 22 00:27:27.086674 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:27.086324 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5xz72" Jul 22 00:27:27.086674 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:27.086391 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sl6b2" podUID="08b6ba5c-7b7f-4cbd-a6d9-584bc9a829fd" Jul 22 00:27:27.086674 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:27.086456 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5xz72" podUID="f36a57f8-6e88-43d3-b572-8adf3b0ddb7c" Jul 22 00:27:28.228249 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:28.228217 2574 generic.go:356] "Generic (PLEG): container finished" podID="077d5538-1eab-48a5-97f9-3a623a4b63c9" containerID="342c51f320da2af0be8cc79f39770fe8ad2a6483d619a2d78ec827f51a429cee" exitCode=0 Jul 22 00:27:28.228743 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:28.228283 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v2ksf" event={"ID":"077d5538-1eab-48a5-97f9-3a623a4b63c9","Type":"ContainerDied","Data":"342c51f320da2af0be8cc79f39770fe8ad2a6483d619a2d78ec827f51a429cee"} Jul 22 00:27:28.793338 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:28.793300 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f36a57f8-6e88-43d3-b572-8adf3b0ddb7c-metrics-certs\") pod \"network-metrics-daemon-5xz72\" (UID: \"f36a57f8-6e88-43d3-b572-8adf3b0ddb7c\") " pod="openshift-multus/network-metrics-daemon-5xz72" Jul 22 00:27:28.793537 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:28.793366 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/08b6ba5c-7b7f-4cbd-a6d9-584bc9a829fd-original-pull-secret\") pod \"global-pull-secret-syncer-sl6b2\" (UID: \"08b6ba5c-7b7f-4cbd-a6d9-584bc9a829fd\") " pod="kube-system/global-pull-secret-syncer-sl6b2" Jul 22 00:27:28.793537 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:28.793421 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 00:27:28.793537 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:28.793476 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 00:27:28.793537 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:28.793499 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f36a57f8-6e88-43d3-b572-8adf3b0ddb7c-metrics-certs podName:f36a57f8-6e88-43d3-b572-8adf3b0ddb7c nodeName:}" failed. No retries permitted until 2026-07-22 00:28:00.793480636 +0000 UTC m=+66.150238511 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f36a57f8-6e88-43d3-b572-8adf3b0ddb7c-metrics-certs") pod "network-metrics-daemon-5xz72" (UID: "f36a57f8-6e88-43d3-b572-8adf3b0ddb7c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 00:27:28.793537 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:28.793526 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08b6ba5c-7b7f-4cbd-a6d9-584bc9a829fd-original-pull-secret podName:08b6ba5c-7b7f-4cbd-a6d9-584bc9a829fd nodeName:}" failed. No retries permitted until 2026-07-22 00:28:00.793510254 +0000 UTC m=+66.150268116 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/08b6ba5c-7b7f-4cbd-a6d9-584bc9a829fd-original-pull-secret") pod "global-pull-secret-syncer-sl6b2" (UID: "08b6ba5c-7b7f-4cbd-a6d9-584bc9a829fd") : object "kube-system"/"original-pull-secret" not registered Jul 22 00:27:28.893970 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:28.893929 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkrgp\" (UniqueName: \"kubernetes.io/projected/bbfe455c-9568-48ba-8007-92b4142b845f-kube-api-access-pkrgp\") pod \"network-check-target-6k9fm\" (UID: \"bbfe455c-9568-48ba-8007-92b4142b845f\") " pod="openshift-network-diagnostics/network-check-target-6k9fm" Jul 22 00:27:28.894197 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:28.894080 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 00:27:28.894197 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:28.894105 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 00:27:28.894197 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:28.894117 2574 projected.go:196] Error preparing data for projected volume kube-api-access-pkrgp for pod openshift-network-diagnostics/network-check-target-6k9fm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 00:27:28.894197 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:28.894180 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bbfe455c-9568-48ba-8007-92b4142b845f-kube-api-access-pkrgp podName:bbfe455c-9568-48ba-8007-92b4142b845f nodeName:}" failed. No retries permitted until 2026-07-22 00:28:00.894161217 +0000 UTC m=+66.250919082 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-pkrgp" (UniqueName: "kubernetes.io/projected/bbfe455c-9568-48ba-8007-92b4142b845f-kube-api-access-pkrgp") pod "network-check-target-6k9fm" (UID: "bbfe455c-9568-48ba-8007-92b4142b845f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 00:27:29.083549 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:29.083465 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5xz72" Jul 22 00:27:29.083549 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:29.083481 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6k9fm" Jul 22 00:27:29.083549 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:29.083465 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sl6b2" Jul 22 00:27:29.083841 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:29.083633 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5xz72" podUID="f36a57f8-6e88-43d3-b572-8adf3b0ddb7c" Jul 22 00:27:29.083841 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:29.083690 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sl6b2" podUID="08b6ba5c-7b7f-4cbd-a6d9-584bc9a829fd" Jul 22 00:27:29.083841 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:29.083761 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6k9fm" podUID="bbfe455c-9568-48ba-8007-92b4142b845f" Jul 22 00:27:29.513134 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:29.513097 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-192.ec2.internal" event="NodeReady" Jul 22 00:27:29.513568 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:29.513249 2574 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 22 00:27:29.553098 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:29.553053 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-9lcsl"] Jul 22 00:27:29.560192 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:29.560157 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-hsxnj"] Jul 22 00:27:29.560348 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:29.560286 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9lcsl" Jul 22 00:27:29.565715 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:29.565198 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nfkpn\"" Jul 22 00:27:29.565715 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:29.565231 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 22 00:27:29.565715 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:29.565543 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 22 00:27:29.570398 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:29.570127 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9lcsl"] Jul 22 00:27:29.570398 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:29.570168 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hsxnj"] Jul 22 00:27:29.570398 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:29.570257 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hsxnj" Jul 22 00:27:29.572825 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:29.572801 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-9frz8\"" Jul 22 00:27:29.573196 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:29.573179 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 22 00:27:29.573291 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:29.573269 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 22 00:27:29.573351 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:29.573179 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 22 00:27:29.700827 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:29.700725 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/646c9f6d-62e7-4a07-8383-554a3c6b8b63-tmp-dir\") pod \"dns-default-9lcsl\" (UID: \"646c9f6d-62e7-4a07-8383-554a3c6b8b63\") " pod="openshift-dns/dns-default-9lcsl" Jul 22 00:27:29.700827 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:29.700773 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/646c9f6d-62e7-4a07-8383-554a3c6b8b63-metrics-tls\") pod \"dns-default-9lcsl\" (UID: \"646c9f6d-62e7-4a07-8383-554a3c6b8b63\") " pod="openshift-dns/dns-default-9lcsl" Jul 22 00:27:29.700827 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:29.700800 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2b90aaae-368a-4ba1-ba82-7e66aa1e518b-cert\") pod \"ingress-canary-hsxnj\" (UID: \"2b90aaae-368a-4ba1-ba82-7e66aa1e518b\") " pod="openshift-ingress-canary/ingress-canary-hsxnj" Jul 22 00:27:29.701101 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:29.700902 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh8xf\" (UniqueName: \"kubernetes.io/projected/646c9f6d-62e7-4a07-8383-554a3c6b8b63-kube-api-access-sh8xf\") pod \"dns-default-9lcsl\" (UID: \"646c9f6d-62e7-4a07-8383-554a3c6b8b63\") " pod="openshift-dns/dns-default-9lcsl" Jul 22 00:27:29.701101 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:29.700941 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/646c9f6d-62e7-4a07-8383-554a3c6b8b63-config-volume\") pod \"dns-default-9lcsl\" (UID: \"646c9f6d-62e7-4a07-8383-554a3c6b8b63\") " pod="openshift-dns/dns-default-9lcsl" Jul 22 00:27:29.701101 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:29.700970 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrzc7\" (UniqueName: \"kubernetes.io/projected/2b90aaae-368a-4ba1-ba82-7e66aa1e518b-kube-api-access-qrzc7\") pod \"ingress-canary-hsxnj\" (UID: \"2b90aaae-368a-4ba1-ba82-7e66aa1e518b\") " pod="openshift-ingress-canary/ingress-canary-hsxnj" Jul 22 00:27:29.802047 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:29.802013 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/646c9f6d-62e7-4a07-8383-554a3c6b8b63-tmp-dir\") pod \"dns-default-9lcsl\" (UID: \"646c9f6d-62e7-4a07-8383-554a3c6b8b63\") " pod="openshift-dns/dns-default-9lcsl" Jul 22 00:27:29.802047 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:29.802058 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/646c9f6d-62e7-4a07-8383-554a3c6b8b63-metrics-tls\") pod \"dns-default-9lcsl\" (UID: \"646c9f6d-62e7-4a07-8383-554a3c6b8b63\") " pod="openshift-dns/dns-default-9lcsl" Jul 22 00:27:29.802291 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:29.802078 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2b90aaae-368a-4ba1-ba82-7e66aa1e518b-cert\") pod \"ingress-canary-hsxnj\" (UID: \"2b90aaae-368a-4ba1-ba82-7e66aa1e518b\") " pod="openshift-ingress-canary/ingress-canary-hsxnj" Jul 22 00:27:29.802291 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:29.802127 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh8xf\" (UniqueName: \"kubernetes.io/projected/646c9f6d-62e7-4a07-8383-554a3c6b8b63-kube-api-access-sh8xf\") pod \"dns-default-9lcsl\" (UID: \"646c9f6d-62e7-4a07-8383-554a3c6b8b63\") " pod="openshift-dns/dns-default-9lcsl" Jul 22 00:27:29.802291 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:29.802148 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/646c9f6d-62e7-4a07-8383-554a3c6b8b63-config-volume\") pod \"dns-default-9lcsl\" (UID: \"646c9f6d-62e7-4a07-8383-554a3c6b8b63\") " pod="openshift-dns/dns-default-9lcsl" Jul 22 00:27:29.802291 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:29.802173 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrzc7\" (UniqueName: \"kubernetes.io/projected/2b90aaae-368a-4ba1-ba82-7e66aa1e518b-kube-api-access-qrzc7\") pod \"ingress-canary-hsxnj\" (UID: \"2b90aaae-368a-4ba1-ba82-7e66aa1e518b\") " pod="openshift-ingress-canary/ingress-canary-hsxnj" Jul 22 00:27:29.802291 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:29.802195 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 00:27:29.802291 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:29.802268 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/646c9f6d-62e7-4a07-8383-554a3c6b8b63-metrics-tls podName:646c9f6d-62e7-4a07-8383-554a3c6b8b63 nodeName:}" failed. No retries permitted until 2026-07-22 00:27:30.302247598 +0000 UTC m=+35.659005473 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/646c9f6d-62e7-4a07-8383-554a3c6b8b63-metrics-tls") pod "dns-default-9lcsl" (UID: "646c9f6d-62e7-4a07-8383-554a3c6b8b63") : secret "dns-default-metrics-tls" not found Jul 22 00:27:29.802568 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:29.802457 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/646c9f6d-62e7-4a07-8383-554a3c6b8b63-tmp-dir\") pod \"dns-default-9lcsl\" (UID: \"646c9f6d-62e7-4a07-8383-554a3c6b8b63\") " pod="openshift-dns/dns-default-9lcsl" Jul 22 00:27:29.803209 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:29.803188 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/646c9f6d-62e7-4a07-8383-554a3c6b8b63-config-volume\") pod \"dns-default-9lcsl\" (UID: \"646c9f6d-62e7-4a07-8383-554a3c6b8b63\") " pod="openshift-dns/dns-default-9lcsl" Jul 22 00:27:29.807083 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:29.806930 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2b90aaae-368a-4ba1-ba82-7e66aa1e518b-cert\") pod \"ingress-canary-hsxnj\" (UID: \"2b90aaae-368a-4ba1-ba82-7e66aa1e518b\") " pod="openshift-ingress-canary/ingress-canary-hsxnj" Jul 22 00:27:29.813493 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:29.813471 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrzc7\" (UniqueName: \"kubernetes.io/projected/2b90aaae-368a-4ba1-ba82-7e66aa1e518b-kube-api-access-qrzc7\") pod \"ingress-canary-hsxnj\" (UID: \"2b90aaae-368a-4ba1-ba82-7e66aa1e518b\") " pod="openshift-ingress-canary/ingress-canary-hsxnj" Jul 22 00:27:29.813633 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:29.813594 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh8xf\" (UniqueName: \"kubernetes.io/projected/646c9f6d-62e7-4a07-8383-554a3c6b8b63-kube-api-access-sh8xf\") pod \"dns-default-9lcsl\" (UID: \"646c9f6d-62e7-4a07-8383-554a3c6b8b63\") " pod="openshift-dns/dns-default-9lcsl" Jul 22 00:27:29.884967 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:29.884934 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hsxnj" Jul 22 00:27:30.113014 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:30.112973 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hsxnj"] Jul 22 00:27:30.117386 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:27:30.117361 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b90aaae_368a_4ba1_ba82_7e66aa1e518b.slice/crio-148d075c78c67850e3dbfbcf0c1f000dd47ad5d3ebccb82e1931342635a957ae WatchSource:0}: Error finding container 148d075c78c67850e3dbfbcf0c1f000dd47ad5d3ebccb82e1931342635a957ae: Status 404 returned error can't find the container with id 148d075c78c67850e3dbfbcf0c1f000dd47ad5d3ebccb82e1931342635a957ae Jul 22 00:27:30.235614 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:30.235503 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hsxnj" event={"ID":"2b90aaae-368a-4ba1-ba82-7e66aa1e518b","Type":"ContainerStarted","Data":"148d075c78c67850e3dbfbcf0c1f000dd47ad5d3ebccb82e1931342635a957ae"} Jul 22 00:27:30.306023 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:30.305942 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/646c9f6d-62e7-4a07-8383-554a3c6b8b63-metrics-tls\") pod \"dns-default-9lcsl\" (UID: \"646c9f6d-62e7-4a07-8383-554a3c6b8b63\") " pod="openshift-dns/dns-default-9lcsl" Jul 22 00:27:30.306190 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:30.306105 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 00:27:30.306250 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:30.306194 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/646c9f6d-62e7-4a07-8383-554a3c6b8b63-metrics-tls podName:646c9f6d-62e7-4a07-8383-554a3c6b8b63 nodeName:}" failed. No retries permitted until 2026-07-22 00:27:31.30617152 +0000 UTC m=+36.662929392 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/646c9f6d-62e7-4a07-8383-554a3c6b8b63-metrics-tls") pod "dns-default-9lcsl" (UID: "646c9f6d-62e7-4a07-8383-554a3c6b8b63") : secret "dns-default-metrics-tls" not found Jul 22 00:27:31.083432 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:31.083393 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5xz72" Jul 22 00:27:31.083432 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:31.083435 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6k9fm" Jul 22 00:27:31.084290 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:31.083393 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sl6b2" Jul 22 00:27:31.087024 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:31.086830 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 22 00:27:31.087024 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:31.086928 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 22 00:27:31.088123 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:31.088103 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dh9z6\"" Jul 22 00:27:31.088123 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:31.088113 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-rsxjh\"" Jul 22 00:27:31.088382 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:31.088361 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 22 00:27:31.088586 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:31.088565 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 22 00:27:31.311923 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:31.311886 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/646c9f6d-62e7-4a07-8383-554a3c6b8b63-metrics-tls\") pod \"dns-default-9lcsl\" (UID: \"646c9f6d-62e7-4a07-8383-554a3c6b8b63\") " pod="openshift-dns/dns-default-9lcsl" Jul 22 00:27:31.312124 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:31.312024 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 00:27:31.312124 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:31.312094 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/646c9f6d-62e7-4a07-8383-554a3c6b8b63-metrics-tls podName:646c9f6d-62e7-4a07-8383-554a3c6b8b63 nodeName:}" failed. No retries permitted until 2026-07-22 00:27:33.312074139 +0000 UTC m=+38.668831997 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/646c9f6d-62e7-4a07-8383-554a3c6b8b63-metrics-tls") pod "dns-default-9lcsl" (UID: "646c9f6d-62e7-4a07-8383-554a3c6b8b63") : secret "dns-default-metrics-tls" not found Jul 22 00:27:33.328198 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:33.328159 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/646c9f6d-62e7-4a07-8383-554a3c6b8b63-metrics-tls\") pod \"dns-default-9lcsl\" (UID: \"646c9f6d-62e7-4a07-8383-554a3c6b8b63\") " pod="openshift-dns/dns-default-9lcsl" Jul 22 00:27:33.328715 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:33.328309 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 00:27:33.328715 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:33.328373 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/646c9f6d-62e7-4a07-8383-554a3c6b8b63-metrics-tls podName:646c9f6d-62e7-4a07-8383-554a3c6b8b63 nodeName:}" failed. No retries permitted until 2026-07-22 00:27:37.328356486 +0000 UTC m=+42.685114346 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/646c9f6d-62e7-4a07-8383-554a3c6b8b63-metrics-tls") pod "dns-default-9lcsl" (UID: "646c9f6d-62e7-4a07-8383-554a3c6b8b63") : secret "dns-default-metrics-tls" not found Jul 22 00:27:34.246571 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:34.246532 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hsxnj" event={"ID":"2b90aaae-368a-4ba1-ba82-7e66aa1e518b","Type":"ContainerStarted","Data":"9205cac743307be0e030e33e1d0d866c36e967985070efd403305461a78b5910"} Jul 22 00:27:34.249068 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:34.249039 2574 generic.go:356] "Generic (PLEG): container finished" podID="077d5538-1eab-48a5-97f9-3a623a4b63c9" containerID="7a5ef4e03313f013856d73f15f4414946812129382cf64b2f6785808ef300f43" exitCode=0 Jul 22 00:27:34.249179 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:34.249081 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v2ksf" event={"ID":"077d5538-1eab-48a5-97f9-3a623a4b63c9","Type":"ContainerDied","Data":"7a5ef4e03313f013856d73f15f4414946812129382cf64b2f6785808ef300f43"} Jul 22 00:27:34.261130 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:34.261078 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-hsxnj" podStartSLOduration=1.922657176 podStartE2EDuration="5.261062378s" totalImagesPullingTime="3.338405202s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:27:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:27:34.260708952 +0000 UTC m=+39.617466862" watchObservedRunningTime="2026-07-22 00:27:34.261062378 +0000 UTC m=+39.617820256" Jul 22 00:27:35.254680 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:35.254646 2574 generic.go:356] "Generic (PLEG): container finished" podID="077d5538-1eab-48a5-97f9-3a623a4b63c9" containerID="3b4f493ae0738a237ac2c68d20213879d5e46769feb767bdbbb36c684ee29bf5" exitCode=0 Jul 22 00:27:35.255100 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:35.254736 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v2ksf" event={"ID":"077d5538-1eab-48a5-97f9-3a623a4b63c9","Type":"ContainerDied","Data":"3b4f493ae0738a237ac2c68d20213879d5e46769feb767bdbbb36c684ee29bf5"} Jul 22 00:27:36.260713 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:36.260670 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v2ksf" event={"ID":"077d5538-1eab-48a5-97f9-3a623a4b63c9","Type":"ContainerStarted","Data":"164f42709e1f33bbf2ce1864ef0cf2756ed7694698c28258d9c1e869feb22a74"} Jul 22 00:27:36.283311 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:36.283255 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-v2ksf" podStartSLOduration=9.927653129 podStartE2EDuration="41.283238314s" totalImagesPullingTime="31.355585185s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 00:26:55 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:27:36.281432665 +0000 UTC m=+41.638190553" watchObservedRunningTime="2026-07-22 00:27:36.283238314 +0000 UTC m=+41.639996192" Jul 22 00:27:37.357580 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:37.357542 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/646c9f6d-62e7-4a07-8383-554a3c6b8b63-metrics-tls\") pod \"dns-default-9lcsl\" (UID: \"646c9f6d-62e7-4a07-8383-554a3c6b8b63\") " pod="openshift-dns/dns-default-9lcsl" Jul 22 00:27:37.357988 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:37.357710 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 00:27:37.357988 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:37.357779 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/646c9f6d-62e7-4a07-8383-554a3c6b8b63-metrics-tls podName:646c9f6d-62e7-4a07-8383-554a3c6b8b63 nodeName:}" failed. No retries permitted until 2026-07-22 00:27:45.357760263 +0000 UTC m=+50.714518137 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/646c9f6d-62e7-4a07-8383-554a3c6b8b63-metrics-tls") pod "dns-default-9lcsl" (UID: "646c9f6d-62e7-4a07-8383-554a3c6b8b63") : secret "dns-default-metrics-tls" not found Jul 22 00:27:45.411986 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:45.411945 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/646c9f6d-62e7-4a07-8383-554a3c6b8b63-metrics-tls\") pod \"dns-default-9lcsl\" (UID: \"646c9f6d-62e7-4a07-8383-554a3c6b8b63\") " pod="openshift-dns/dns-default-9lcsl" Jul 22 00:27:45.412443 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:45.412067 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 00:27:45.412443 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:27:45.412119 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/646c9f6d-62e7-4a07-8383-554a3c6b8b63-metrics-tls podName:646c9f6d-62e7-4a07-8383-554a3c6b8b63 nodeName:}" failed. No retries permitted until 2026-07-22 00:28:01.412104943 +0000 UTC m=+66.768862798 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/646c9f6d-62e7-4a07-8383-554a3c6b8b63-metrics-tls") pod "dns-default-9lcsl" (UID: "646c9f6d-62e7-4a07-8383-554a3c6b8b63") : secret "dns-default-metrics-tls" not found Jul 22 00:27:55.227962 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:55.227928 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hr5fn" Jul 22 00:27:56.573338 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:56.573301 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86b5c84d8-txnd8"] Jul 22 00:27:56.599309 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:56.599277 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86b5c84d8-txnd8"] Jul 22 00:27:56.599524 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:56.599319 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ff8fbf64f-d7jxg"] Jul 22 00:27:56.599524 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:56.599485 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86b5c84d8-txnd8" Jul 22 00:27:56.603568 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:56.603545 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 22 00:27:56.603568 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:56.603560 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 22 00:27:56.603788 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:56.603590 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 22 00:27:56.603788 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:56.603590 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 22 00:27:56.603788 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:56.603632 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-wcfcp\"" Jul 22 00:27:56.611863 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:56.611846 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ff8fbf64f-d7jxg"] Jul 22 00:27:56.611964 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:56.611928 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ff8fbf64f-d7jxg" Jul 22 00:27:56.614348 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:56.614332 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 22 00:27:56.690097 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:56.690059 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/20bcbb88-28d1-411f-8abb-a3901542b77a-tmp\") pod \"klusterlet-addon-workmgr-5ff8fbf64f-d7jxg\" (UID: \"20bcbb88-28d1-411f-8abb-a3901542b77a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ff8fbf64f-d7jxg" Jul 22 00:27:56.690263 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:56.690126 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69lkq\" (UniqueName: \"kubernetes.io/projected/20bcbb88-28d1-411f-8abb-a3901542b77a-kube-api-access-69lkq\") pod \"klusterlet-addon-workmgr-5ff8fbf64f-d7jxg\" (UID: \"20bcbb88-28d1-411f-8abb-a3901542b77a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ff8fbf64f-d7jxg" Jul 22 00:27:56.690263 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:56.690172 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvzgd\" (UniqueName: \"kubernetes.io/projected/54f2f0f7-b6f3-4545-b67f-8ce39c79ba4f-kube-api-access-rvzgd\") pod \"managed-serviceaccount-addon-agent-86b5c84d8-txnd8\" (UID: \"54f2f0f7-b6f3-4545-b67f-8ce39c79ba4f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86b5c84d8-txnd8" Jul 22 00:27:56.690263 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:56.690218 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/20bcbb88-28d1-411f-8abb-a3901542b77a-klusterlet-config\") pod \"klusterlet-addon-workmgr-5ff8fbf64f-d7jxg\" (UID: \"20bcbb88-28d1-411f-8abb-a3901542b77a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ff8fbf64f-d7jxg" Jul 22 00:27:56.690263 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:56.690258 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/54f2f0f7-b6f3-4545-b67f-8ce39c79ba4f-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-86b5c84d8-txnd8\" (UID: \"54f2f0f7-b6f3-4545-b67f-8ce39c79ba4f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86b5c84d8-txnd8" Jul 22 00:27:56.791370 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:56.791334 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/54f2f0f7-b6f3-4545-b67f-8ce39c79ba4f-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-86b5c84d8-txnd8\" (UID: \"54f2f0f7-b6f3-4545-b67f-8ce39c79ba4f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86b5c84d8-txnd8" Jul 22 00:27:56.791542 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:56.791379 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/20bcbb88-28d1-411f-8abb-a3901542b77a-tmp\") pod \"klusterlet-addon-workmgr-5ff8fbf64f-d7jxg\" (UID: \"20bcbb88-28d1-411f-8abb-a3901542b77a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ff8fbf64f-d7jxg" Jul 22 00:27:56.791542 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:56.791437 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69lkq\" (UniqueName: \"kubernetes.io/projected/20bcbb88-28d1-411f-8abb-a3901542b77a-kube-api-access-69lkq\") pod \"klusterlet-addon-workmgr-5ff8fbf64f-d7jxg\" (UID: \"20bcbb88-28d1-411f-8abb-a3901542b77a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ff8fbf64f-d7jxg" Jul 22 00:27:56.791542 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:56.791455 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvzgd\" (UniqueName: \"kubernetes.io/projected/54f2f0f7-b6f3-4545-b67f-8ce39c79ba4f-kube-api-access-rvzgd\") pod \"managed-serviceaccount-addon-agent-86b5c84d8-txnd8\" (UID: \"54f2f0f7-b6f3-4545-b67f-8ce39c79ba4f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86b5c84d8-txnd8" Jul 22 00:27:56.791542 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:56.791482 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/20bcbb88-28d1-411f-8abb-a3901542b77a-klusterlet-config\") pod \"klusterlet-addon-workmgr-5ff8fbf64f-d7jxg\" (UID: \"20bcbb88-28d1-411f-8abb-a3901542b77a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ff8fbf64f-d7jxg" Jul 22 00:27:56.791876 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:56.791851 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/20bcbb88-28d1-411f-8abb-a3901542b77a-tmp\") pod \"klusterlet-addon-workmgr-5ff8fbf64f-d7jxg\" (UID: \"20bcbb88-28d1-411f-8abb-a3901542b77a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ff8fbf64f-d7jxg" Jul 22 00:27:56.795011 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:56.794990 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/20bcbb88-28d1-411f-8abb-a3901542b77a-klusterlet-config\") pod \"klusterlet-addon-workmgr-5ff8fbf64f-d7jxg\" (UID: \"20bcbb88-28d1-411f-8abb-a3901542b77a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ff8fbf64f-d7jxg" Jul 22 00:27:56.795103 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:56.795048 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/54f2f0f7-b6f3-4545-b67f-8ce39c79ba4f-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-86b5c84d8-txnd8\" (UID: \"54f2f0f7-b6f3-4545-b67f-8ce39c79ba4f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86b5c84d8-txnd8" Jul 22 00:27:56.799344 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:56.799302 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69lkq\" (UniqueName: \"kubernetes.io/projected/20bcbb88-28d1-411f-8abb-a3901542b77a-kube-api-access-69lkq\") pod \"klusterlet-addon-workmgr-5ff8fbf64f-d7jxg\" (UID: \"20bcbb88-28d1-411f-8abb-a3901542b77a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ff8fbf64f-d7jxg" Jul 22 00:27:56.799436 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:56.799388 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvzgd\" (UniqueName: \"kubernetes.io/projected/54f2f0f7-b6f3-4545-b67f-8ce39c79ba4f-kube-api-access-rvzgd\") pod \"managed-serviceaccount-addon-agent-86b5c84d8-txnd8\" (UID: \"54f2f0f7-b6f3-4545-b67f-8ce39c79ba4f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86b5c84d8-txnd8" Jul 22 00:27:56.915448 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:56.915413 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86b5c84d8-txnd8" Jul 22 00:27:56.922532 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:56.922506 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ff8fbf64f-d7jxg" Jul 22 00:27:57.068467 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:57.068435 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86b5c84d8-txnd8"] Jul 22 00:27:57.071735 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:27:57.071708 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod54f2f0f7_b6f3_4545_b67f_8ce39c79ba4f.slice/crio-25b3a0f00ce6fe724b82848e812ffdf04f7aee0224319848ef86c489089d5aba WatchSource:0}: Error finding container 25b3a0f00ce6fe724b82848e812ffdf04f7aee0224319848ef86c489089d5aba: Status 404 returned error can't find the container with id 25b3a0f00ce6fe724b82848e812ffdf04f7aee0224319848ef86c489089d5aba Jul 22 00:27:57.086431 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:57.086408 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ff8fbf64f-d7jxg"] Jul 22 00:27:57.086939 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:27:57.086917 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20bcbb88_28d1_411f_8abb_a3901542b77a.slice/crio-874ad884e0d573981b523f84912cc522e6ae7003ac17421d8972bfb366170151 WatchSource:0}: Error finding container 874ad884e0d573981b523f84912cc522e6ae7003ac17421d8972bfb366170151: Status 404 returned error can't find the container with id 874ad884e0d573981b523f84912cc522e6ae7003ac17421d8972bfb366170151 Jul 22 00:27:57.308328 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:57.308243 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ff8fbf64f-d7jxg" event={"ID":"20bcbb88-28d1-411f-8abb-a3901542b77a","Type":"ContainerStarted","Data":"874ad884e0d573981b523f84912cc522e6ae7003ac17421d8972bfb366170151"} Jul 22 00:27:57.309263 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:27:57.309242 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86b5c84d8-txnd8" event={"ID":"54f2f0f7-b6f3-4545-b67f-8ce39c79ba4f","Type":"ContainerStarted","Data":"25b3a0f00ce6fe724b82848e812ffdf04f7aee0224319848ef86c489089d5aba"} Jul 22 00:28:00.820369 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:28:00.820329 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f36a57f8-6e88-43d3-b572-8adf3b0ddb7c-metrics-certs\") pod \"network-metrics-daemon-5xz72\" (UID: \"f36a57f8-6e88-43d3-b572-8adf3b0ddb7c\") " pod="openshift-multus/network-metrics-daemon-5xz72" Jul 22 00:28:00.820888 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:28:00.820401 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/08b6ba5c-7b7f-4cbd-a6d9-584bc9a829fd-original-pull-secret\") pod \"global-pull-secret-syncer-sl6b2\" (UID: \"08b6ba5c-7b7f-4cbd-a6d9-584bc9a829fd\") " pod="kube-system/global-pull-secret-syncer-sl6b2" Jul 22 00:28:00.823644 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:28:00.823614 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 22 00:28:00.823745 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:28:00.823628 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 22 00:28:00.831084 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:28:00.831054 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 22 00:28:00.831195 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:28:00.831156 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f36a57f8-6e88-43d3-b572-8adf3b0ddb7c-metrics-certs podName:f36a57f8-6e88-43d3-b572-8adf3b0ddb7c nodeName:}" failed. No retries permitted until 2026-07-22 00:29:04.831134012 +0000 UTC m=+130.187891875 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f36a57f8-6e88-43d3-b572-8adf3b0ddb7c-metrics-certs") pod "network-metrics-daemon-5xz72" (UID: "f36a57f8-6e88-43d3-b572-8adf3b0ddb7c") : secret "metrics-daemon-secret" not found Jul 22 00:28:00.833343 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:28:00.833318 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/08b6ba5c-7b7f-4cbd-a6d9-584bc9a829fd-original-pull-secret\") pod \"global-pull-secret-syncer-sl6b2\" (UID: \"08b6ba5c-7b7f-4cbd-a6d9-584bc9a829fd\") " pod="kube-system/global-pull-secret-syncer-sl6b2" Jul 22 00:28:00.920997 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:28:00.920956 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkrgp\" (UniqueName: \"kubernetes.io/projected/bbfe455c-9568-48ba-8007-92b4142b845f-kube-api-access-pkrgp\") pod \"network-check-target-6k9fm\" (UID: \"bbfe455c-9568-48ba-8007-92b4142b845f\") " pod="openshift-network-diagnostics/network-check-target-6k9fm" Jul 22 00:28:00.924220 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:28:00.924195 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 22 00:28:00.933821 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:28:00.933781 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 22 00:28:00.945002 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:28:00.944947 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkrgp\" (UniqueName: \"kubernetes.io/projected/bbfe455c-9568-48ba-8007-92b4142b845f-kube-api-access-pkrgp\") pod \"network-check-target-6k9fm\" (UID: \"bbfe455c-9568-48ba-8007-92b4142b845f\") " pod="openshift-network-diagnostics/network-check-target-6k9fm" Jul 22 00:28:01.106663 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:28:01.106635 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sl6b2" Jul 22 00:28:01.115847 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:28:01.115822 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-rsxjh\"" Jul 22 00:28:01.124090 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:28:01.124066 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6k9fm" Jul 22 00:28:01.269761 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:28:01.269683 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-sl6b2"] Jul 22 00:28:01.274061 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:28:01.274034 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08b6ba5c_7b7f_4cbd_a6d9_584bc9a829fd.slice/crio-fe5ca974437c35b3fb89c7ff8d45bea6576c774f02b6ebce23ed2365151fb61e WatchSource:0}: Error finding container fe5ca974437c35b3fb89c7ff8d45bea6576c774f02b6ebce23ed2365151fb61e: Status 404 returned error can't find the container with id fe5ca974437c35b3fb89c7ff8d45bea6576c774f02b6ebce23ed2365151fb61e Jul 22 00:28:01.284493 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:28:01.284464 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6k9fm"] Jul 22 00:28:01.287186 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:28:01.287162 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbfe455c_9568_48ba_8007_92b4142b845f.slice/crio-b97a7e78183994beb6eeec5503ecfe5c8657b116f04a341884c568beec4af0a1 WatchSource:0}: Error finding container b97a7e78183994beb6eeec5503ecfe5c8657b116f04a341884c568beec4af0a1: Status 404 returned error can't find the container with id b97a7e78183994beb6eeec5503ecfe5c8657b116f04a341884c568beec4af0a1 Jul 22 00:28:01.321797 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:28:01.321765 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6k9fm" event={"ID":"bbfe455c-9568-48ba-8007-92b4142b845f","Type":"ContainerStarted","Data":"b97a7e78183994beb6eeec5503ecfe5c8657b116f04a341884c568beec4af0a1"} Jul 22 00:28:01.323180 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:28:01.323106 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86b5c84d8-txnd8" event={"ID":"54f2f0f7-b6f3-4545-b67f-8ce39c79ba4f","Type":"ContainerStarted","Data":"f2e4738aa7517da0b1afe771eb5b54d9667d3da9dbc78891fa4f500b58d1afe3"} Jul 22 00:28:01.324137 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:28:01.324116 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-sl6b2" event={"ID":"08b6ba5c-7b7f-4cbd-a6d9-584bc9a829fd","Type":"ContainerStarted","Data":"fe5ca974437c35b3fb89c7ff8d45bea6576c774f02b6ebce23ed2365151fb61e"} Jul 22 00:28:01.337163 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:28:01.337125 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86b5c84d8-txnd8" podStartSLOduration=1.373763052 podStartE2EDuration="5.337113075s" totalImagesPullingTime="3.963350023s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:27:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:28:01.336511124 +0000 UTC m=+66.693269020" watchObservedRunningTime="2026-07-22 00:28:01.337113075 +0000 UTC m=+66.693870951" Jul 22 00:28:01.424428 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:28:01.424393 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/646c9f6d-62e7-4a07-8383-554a3c6b8b63-metrics-tls\") pod \"dns-default-9lcsl\" (UID: \"646c9f6d-62e7-4a07-8383-554a3c6b8b63\") " pod="openshift-dns/dns-default-9lcsl" Jul 22 00:28:01.424589 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:28:01.424528 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 00:28:01.424655 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:28:01.424622 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/646c9f6d-62e7-4a07-8383-554a3c6b8b63-metrics-tls podName:646c9f6d-62e7-4a07-8383-554a3c6b8b63 nodeName:}" failed. No retries permitted until 2026-07-22 00:28:33.424584233 +0000 UTC m=+98.781342087 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/646c9f6d-62e7-4a07-8383-554a3c6b8b63-metrics-tls") pod "dns-default-9lcsl" (UID: "646c9f6d-62e7-4a07-8383-554a3c6b8b63") : secret "dns-default-metrics-tls" not found Jul 22 00:28:05.336667 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:28:05.336570 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6k9fm" event={"ID":"bbfe455c-9568-48ba-8007-92b4142b845f","Type":"ContainerStarted","Data":"d4fec0faa5a5fcd27a15c6d8c740d43054eb99c2cd261886f555bf35efa87118"} Jul 22 00:28:05.337044 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:28:05.336723 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-6k9fm" Jul 22 00:28:05.354968 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:28:05.354901 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-6k9fm" podStartSLOduration=66.707179385 podStartE2EDuration="1m10.35488561s" totalImagesPullingTime="3.647706225s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:26:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:28:05.354455925 +0000 UTC m=+70.711213824" watchObservedRunningTime="2026-07-22 00:28:05.35488561 +0000 UTC m=+70.711643541" Jul 22 00:28:07.343286 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:28:07.343251 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-sl6b2" event={"ID":"08b6ba5c-7b7f-4cbd-a6d9-584bc9a829fd","Type":"ContainerStarted","Data":"271c50908faef105208443535c59b5c13f6967b9a680d42b08388c02e23a08fe"} Jul 22 00:28:07.357997 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:28:07.357947 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-sl6b2" podStartSLOduration=66.776594593 podStartE2EDuration="1m12.357931908s" totalImagesPullingTime="5.581337315s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 00:26:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:28:07.357085525 +0000 UTC m=+72.713843425" watchObservedRunningTime="2026-07-22 00:28:07.357931908 +0000 UTC m=+72.714689784" Jul 22 00:28:33.453866 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:28:33.453824 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/646c9f6d-62e7-4a07-8383-554a3c6b8b63-metrics-tls\") pod \"dns-default-9lcsl\" (UID: \"646c9f6d-62e7-4a07-8383-554a3c6b8b63\") " pod="openshift-dns/dns-default-9lcsl" Jul 22 00:28:33.454371 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:28:33.453946 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 00:28:33.454371 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:28:33.454001 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/646c9f6d-62e7-4a07-8383-554a3c6b8b63-metrics-tls podName:646c9f6d-62e7-4a07-8383-554a3c6b8b63 nodeName:}" failed. No retries permitted until 2026-07-22 00:29:37.453984802 +0000 UTC m=+162.810742660 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/646c9f6d-62e7-4a07-8383-554a3c6b8b63-metrics-tls") pod "dns-default-9lcsl" (UID: "646c9f6d-62e7-4a07-8383-554a3c6b8b63") : secret "dns-default-metrics-tls" not found Jul 22 00:28:36.342471 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:28:36.342436 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-6k9fm" Jul 22 00:28:48.441815 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:28:48.441777 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ff8fbf64f-d7jxg" event={"ID":"20bcbb88-28d1-411f-8abb-a3901542b77a","Type":"ContainerStarted","Data":"f32f5cdffec5524fe8e0df5045924d29143d0aea4bfff3bf562921f0e9b373b7"} Jul 22 00:28:48.442224 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:28:48.441943 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ff8fbf64f-d7jxg" Jul 22 00:28:48.443407 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:28:48.443391 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ff8fbf64f-d7jxg" Jul 22 00:28:48.457875 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:28:48.457834 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ff8fbf64f-d7jxg" podStartSLOduration=1.4917325909999999 podStartE2EDuration="52.45782315s" totalImagesPullingTime="50.966090559s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:27:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:28:48.45732987 +0000 UTC m=+113.814087788" watchObservedRunningTime="2026-07-22 00:28:48.45782315 +0000 UTC m=+113.814581053" Jul 22 00:29:04.866000 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:29:04.865952 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f36a57f8-6e88-43d3-b572-8adf3b0ddb7c-metrics-certs\") pod \"network-metrics-daemon-5xz72\" (UID: \"f36a57f8-6e88-43d3-b572-8adf3b0ddb7c\") " pod="openshift-multus/network-metrics-daemon-5xz72" Jul 22 00:29:04.866501 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:29:04.866112 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 22 00:29:04.866501 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:29:04.866190 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f36a57f8-6e88-43d3-b572-8adf3b0ddb7c-metrics-certs podName:f36a57f8-6e88-43d3-b572-8adf3b0ddb7c nodeName:}" failed. No retries permitted until 2026-07-22 00:31:06.86617306 +0000 UTC m=+252.222930914 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f36a57f8-6e88-43d3-b572-8adf3b0ddb7c-metrics-certs") pod "network-metrics-daemon-5xz72" (UID: "f36a57f8-6e88-43d3-b572-8adf3b0ddb7c") : secret "metrics-daemon-secret" not found Jul 22 00:29:32.572797 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:29:32.572761 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-9lcsl" podUID="646c9f6d-62e7-4a07-8383-554a3c6b8b63" Jul 22 00:29:33.571286 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:29:33.571255 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9lcsl" Jul 22 00:29:34.098433 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:29:34.098383 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-5xz72" podUID="f36a57f8-6e88-43d3-b572-8adf3b0ddb7c" Jul 22 00:29:37.491765 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:29:37.491703 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/646c9f6d-62e7-4a07-8383-554a3c6b8b63-metrics-tls\") pod \"dns-default-9lcsl\" (UID: \"646c9f6d-62e7-4a07-8383-554a3c6b8b63\") " pod="openshift-dns/dns-default-9lcsl" Jul 22 00:29:37.492301 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:29:37.491878 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 00:29:37.492301 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:29:37.491994 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/646c9f6d-62e7-4a07-8383-554a3c6b8b63-metrics-tls podName:646c9f6d-62e7-4a07-8383-554a3c6b8b63 nodeName:}" failed. No retries permitted until 2026-07-22 00:31:39.491972167 +0000 UTC m=+284.848730022 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/646c9f6d-62e7-4a07-8383-554a3c6b8b63-metrics-tls") pod "dns-default-9lcsl" (UID: "646c9f6d-62e7-4a07-8383-554a3c6b8b63") : secret "dns-default-metrics-tls" not found Jul 22 00:29:45.083673 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:29:45.083631 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5xz72" Jul 22 00:30:01.651273 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:01.651230 2574 generic.go:356] "Generic (PLEG): container finished" podID="54f2f0f7-b6f3-4545-b67f-8ce39c79ba4f" containerID="f2e4738aa7517da0b1afe771eb5b54d9667d3da9dbc78891fa4f500b58d1afe3" exitCode=255 Jul 22 00:30:01.651656 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:01.651306 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86b5c84d8-txnd8" event={"ID":"54f2f0f7-b6f3-4545-b67f-8ce39c79ba4f","Type":"ContainerDied","Data":"f2e4738aa7517da0b1afe771eb5b54d9667d3da9dbc78891fa4f500b58d1afe3"} Jul 22 00:30:01.657014 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:01.656997 2574 scope.go:117] "RemoveContainer" containerID="f2e4738aa7517da0b1afe771eb5b54d9667d3da9dbc78891fa4f500b58d1afe3" Jul 22 00:30:02.655648 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:02.655594 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86b5c84d8-txnd8" event={"ID":"54f2f0f7-b6f3-4545-b67f-8ce39c79ba4f","Type":"ContainerStarted","Data":"20d5330a1454f163311d9152979115f82e009484c2203945144325631febf18a"} Jul 22 00:30:48.442668 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:48.442556 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ff8fbf64f-d7jxg" podUID="20bcbb88-28d1-411f-8abb-a3901542b77a" containerName="acm-agent" probeResult="failure" output="Get \"http://10.133.0.9:8000/readyz\": dial tcp 10.133.0.9:8000: connect: connection refused" Jul 22 00:30:48.722911 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:48.722829 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-hb69v"] Jul 22 00:30:48.725763 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:48.725746 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5b776d6f4f-45tmn"] Jul 22 00:30:48.725898 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:48.725881 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-hb69v" Jul 22 00:30:48.728305 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:48.728288 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b776d6f4f-45tmn" Jul 22 00:30:48.728707 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:48.728662 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 22 00:30:48.729214 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:48.729195 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 22 00:30:48.729308 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:48.729233 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 22 00:30:48.730261 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:48.730244 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 22 00:30:48.730343 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:48.730264 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-jkfdk\"" Jul 22 00:30:48.732040 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:48.732022 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-kstbg\"" Jul 22 00:30:48.732464 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:48.732439 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 22 00:30:48.732562 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:48.732448 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 22 00:30:48.732562 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:48.732498 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 22 00:30:48.734382 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:48.734365 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 22 00:30:48.739118 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:48.738992 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 22 00:30:48.748067 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:48.748043 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-hb69v"] Jul 22 00:30:48.750632 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:48.750584 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5b776d6f4f-45tmn"] Jul 22 00:30:48.784627 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:48.784583 2574 generic.go:356] "Generic (PLEG): container finished" podID="20bcbb88-28d1-411f-8abb-a3901542b77a" containerID="f32f5cdffec5524fe8e0df5045924d29143d0aea4bfff3bf562921f0e9b373b7" exitCode=1 Jul 22 00:30:48.784760 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:48.784632 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ff8fbf64f-d7jxg" event={"ID":"20bcbb88-28d1-411f-8abb-a3901542b77a","Type":"ContainerDied","Data":"f32f5cdffec5524fe8e0df5045924d29143d0aea4bfff3bf562921f0e9b373b7"} Jul 22 00:30:48.784932 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:48.784917 2574 scope.go:117] "RemoveContainer" containerID="f32f5cdffec5524fe8e0df5045924d29143d0aea4bfff3bf562921f0e9b373b7" Jul 22 00:30:48.871387 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:48.871341 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a29d90f9-887b-4d2e-a145-c985530029f8-installation-pull-secrets\") pod \"image-registry-5b776d6f4f-45tmn\" (UID: \"a29d90f9-887b-4d2e-a145-c985530029f8\") " pod="openshift-image-registry/image-registry-5b776d6f4f-45tmn" Jul 22 00:30:48.871525 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:48.871450 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a29d90f9-887b-4d2e-a145-c985530029f8-bound-sa-token\") pod \"image-registry-5b776d6f4f-45tmn\" (UID: \"a29d90f9-887b-4d2e-a145-c985530029f8\") " pod="openshift-image-registry/image-registry-5b776d6f4f-45tmn" Jul 22 00:30:48.871525 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:48.871489 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bf7b370a-c67c-4c3c-b3ba-7533131eace2-config\") pod \"console-operator-565b7b569d-hb69v\" (UID: \"bf7b370a-c67c-4c3c-b3ba-7533131eace2\") " pod="openshift-console-operator/console-operator-565b7b569d-hb69v" Jul 22 00:30:48.871525 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:48.871517 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bf7b370a-c67c-4c3c-b3ba-7533131eace2-trusted-ca\") pod \"console-operator-565b7b569d-hb69v\" (UID: \"bf7b370a-c67c-4c3c-b3ba-7533131eace2\") " pod="openshift-console-operator/console-operator-565b7b569d-hb69v" Jul 22 00:30:48.871681 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:48.871552 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a29d90f9-887b-4d2e-a145-c985530029f8-registry-certificates\") pod \"image-registry-5b776d6f4f-45tmn\" (UID: \"a29d90f9-887b-4d2e-a145-c985530029f8\") " pod="openshift-image-registry/image-registry-5b776d6f4f-45tmn" Jul 22 00:30:48.871737 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:48.871706 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a29d90f9-887b-4d2e-a145-c985530029f8-registry-tls\") pod \"image-registry-5b776d6f4f-45tmn\" (UID: \"a29d90f9-887b-4d2e-a145-c985530029f8\") " pod="openshift-image-registry/image-registry-5b776d6f4f-45tmn" Jul 22 00:30:48.871791 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:48.871736 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx8gc\" (UniqueName: \"kubernetes.io/projected/bf7b370a-c67c-4c3c-b3ba-7533131eace2-kube-api-access-gx8gc\") pod \"console-operator-565b7b569d-hb69v\" (UID: \"bf7b370a-c67c-4c3c-b3ba-7533131eace2\") " pod="openshift-console-operator/console-operator-565b7b569d-hb69v" Jul 22 00:30:48.871837 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:48.871806 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a29d90f9-887b-4d2e-a145-c985530029f8-ca-trust-extracted\") pod \"image-registry-5b776d6f4f-45tmn\" (UID: \"a29d90f9-887b-4d2e-a145-c985530029f8\") " pod="openshift-image-registry/image-registry-5b776d6f4f-45tmn" Jul 22 00:30:48.871892 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:48.871851 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a29d90f9-887b-4d2e-a145-c985530029f8-image-registry-private-configuration\") pod \"image-registry-5b776d6f4f-45tmn\" (UID: \"a29d90f9-887b-4d2e-a145-c985530029f8\") " pod="openshift-image-registry/image-registry-5b776d6f4f-45tmn" Jul 22 00:30:48.871892 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:48.871877 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a29d90f9-887b-4d2e-a145-c985530029f8-trusted-ca\") pod \"image-registry-5b776d6f4f-45tmn\" (UID: \"a29d90f9-887b-4d2e-a145-c985530029f8\") " pod="openshift-image-registry/image-registry-5b776d6f4f-45tmn" Jul 22 00:30:48.871988 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:48.871939 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lv2dc\" (UniqueName: \"kubernetes.io/projected/a29d90f9-887b-4d2e-a145-c985530029f8-kube-api-access-lv2dc\") pod \"image-registry-5b776d6f4f-45tmn\" (UID: \"a29d90f9-887b-4d2e-a145-c985530029f8\") " pod="openshift-image-registry/image-registry-5b776d6f4f-45tmn" Jul 22 00:30:48.871988 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:48.871971 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bf7b370a-c67c-4c3c-b3ba-7533131eace2-serving-cert\") pod \"console-operator-565b7b569d-hb69v\" (UID: \"bf7b370a-c67c-4c3c-b3ba-7533131eace2\") " pod="openshift-console-operator/console-operator-565b7b569d-hb69v" Jul 22 00:30:48.972741 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:48.972710 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bf7b370a-c67c-4c3c-b3ba-7533131eace2-trusted-ca\") pod \"console-operator-565b7b569d-hb69v\" (UID: \"bf7b370a-c67c-4c3c-b3ba-7533131eace2\") " pod="openshift-console-operator/console-operator-565b7b569d-hb69v" Jul 22 00:30:48.972741 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:48.972744 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a29d90f9-887b-4d2e-a145-c985530029f8-registry-certificates\") pod \"image-registry-5b776d6f4f-45tmn\" (UID: \"a29d90f9-887b-4d2e-a145-c985530029f8\") " pod="openshift-image-registry/image-registry-5b776d6f4f-45tmn" Jul 22 00:30:48.972977 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:48.972933 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a29d90f9-887b-4d2e-a145-c985530029f8-registry-tls\") pod \"image-registry-5b776d6f4f-45tmn\" (UID: \"a29d90f9-887b-4d2e-a145-c985530029f8\") " pod="openshift-image-registry/image-registry-5b776d6f4f-45tmn" Jul 22 00:30:48.973026 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:48.972976 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx8gc\" (UniqueName: \"kubernetes.io/projected/bf7b370a-c67c-4c3c-b3ba-7533131eace2-kube-api-access-gx8gc\") pod \"console-operator-565b7b569d-hb69v\" (UID: \"bf7b370a-c67c-4c3c-b3ba-7533131eace2\") " pod="openshift-console-operator/console-operator-565b7b569d-hb69v" Jul 22 00:30:48.973070 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:48.973027 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a29d90f9-887b-4d2e-a145-c985530029f8-ca-trust-extracted\") pod \"image-registry-5b776d6f4f-45tmn\" (UID: \"a29d90f9-887b-4d2e-a145-c985530029f8\") " pod="openshift-image-registry/image-registry-5b776d6f4f-45tmn" Jul 22 00:30:48.973070 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:30:48.973058 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 00:30:48.973070 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:48.973065 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a29d90f9-887b-4d2e-a145-c985530029f8-image-registry-private-configuration\") pod \"image-registry-5b776d6f4f-45tmn\" (UID: \"a29d90f9-887b-4d2e-a145-c985530029f8\") " pod="openshift-image-registry/image-registry-5b776d6f4f-45tmn" Jul 22 00:30:48.973208 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:30:48.973075 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b776d6f4f-45tmn: secret "image-registry-tls" not found Jul 22 00:30:48.973208 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:48.973090 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a29d90f9-887b-4d2e-a145-c985530029f8-trusted-ca\") pod \"image-registry-5b776d6f4f-45tmn\" (UID: \"a29d90f9-887b-4d2e-a145-c985530029f8\") " pod="openshift-image-registry/image-registry-5b776d6f4f-45tmn" Jul 22 00:30:48.973208 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:30:48.973126 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a29d90f9-887b-4d2e-a145-c985530029f8-registry-tls podName:a29d90f9-887b-4d2e-a145-c985530029f8 nodeName:}" failed. No retries permitted until 2026-07-22 00:30:49.473110611 +0000 UTC m=+234.829868492 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a29d90f9-887b-4d2e-a145-c985530029f8-registry-tls") pod "image-registry-5b776d6f4f-45tmn" (UID: "a29d90f9-887b-4d2e-a145-c985530029f8") : secret "image-registry-tls" not found Jul 22 00:30:48.973208 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:48.973145 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lv2dc\" (UniqueName: \"kubernetes.io/projected/a29d90f9-887b-4d2e-a145-c985530029f8-kube-api-access-lv2dc\") pod \"image-registry-5b776d6f4f-45tmn\" (UID: \"a29d90f9-887b-4d2e-a145-c985530029f8\") " pod="openshift-image-registry/image-registry-5b776d6f4f-45tmn" Jul 22 00:30:48.973208 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:48.973171 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bf7b370a-c67c-4c3c-b3ba-7533131eace2-serving-cert\") pod \"console-operator-565b7b569d-hb69v\" (UID: \"bf7b370a-c67c-4c3c-b3ba-7533131eace2\") " pod="openshift-console-operator/console-operator-565b7b569d-hb69v" Jul 22 00:30:48.973208 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:48.973210 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a29d90f9-887b-4d2e-a145-c985530029f8-installation-pull-secrets\") pod \"image-registry-5b776d6f4f-45tmn\" (UID: \"a29d90f9-887b-4d2e-a145-c985530029f8\") " pod="openshift-image-registry/image-registry-5b776d6f4f-45tmn" Jul 22 00:30:48.973498 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:48.973245 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a29d90f9-887b-4d2e-a145-c985530029f8-bound-sa-token\") pod \"image-registry-5b776d6f4f-45tmn\" (UID: \"a29d90f9-887b-4d2e-a145-c985530029f8\") " pod="openshift-image-registry/image-registry-5b776d6f4f-45tmn" Jul 22 00:30:48.973498 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:48.973264 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bf7b370a-c67c-4c3c-b3ba-7533131eace2-config\") pod \"console-operator-565b7b569d-hb69v\" (UID: \"bf7b370a-c67c-4c3c-b3ba-7533131eace2\") " pod="openshift-console-operator/console-operator-565b7b569d-hb69v" Jul 22 00:30:48.973498 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:48.973306 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a29d90f9-887b-4d2e-a145-c985530029f8-registry-certificates\") pod \"image-registry-5b776d6f4f-45tmn\" (UID: \"a29d90f9-887b-4d2e-a145-c985530029f8\") " pod="openshift-image-registry/image-registry-5b776d6f4f-45tmn" Jul 22 00:30:48.973747 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:48.973698 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bf7b370a-c67c-4c3c-b3ba-7533131eace2-trusted-ca\") pod \"console-operator-565b7b569d-hb69v\" (UID: \"bf7b370a-c67c-4c3c-b3ba-7533131eace2\") " pod="openshift-console-operator/console-operator-565b7b569d-hb69v" Jul 22 00:30:48.973976 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:48.973954 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bf7b370a-c67c-4c3c-b3ba-7533131eace2-config\") pod \"console-operator-565b7b569d-hb69v\" (UID: \"bf7b370a-c67c-4c3c-b3ba-7533131eace2\") " pod="openshift-console-operator/console-operator-565b7b569d-hb69v" Jul 22 00:30:48.974052 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:48.973985 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a29d90f9-887b-4d2e-a145-c985530029f8-ca-trust-extracted\") pod \"image-registry-5b776d6f4f-45tmn\" (UID: \"a29d90f9-887b-4d2e-a145-c985530029f8\") " pod="openshift-image-registry/image-registry-5b776d6f4f-45tmn" Jul 22 00:30:48.974052 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:48.973995 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a29d90f9-887b-4d2e-a145-c985530029f8-trusted-ca\") pod \"image-registry-5b776d6f4f-45tmn\" (UID: \"a29d90f9-887b-4d2e-a145-c985530029f8\") " pod="openshift-image-registry/image-registry-5b776d6f4f-45tmn" Jul 22 00:30:48.975701 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:48.975682 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a29d90f9-887b-4d2e-a145-c985530029f8-image-registry-private-configuration\") pod \"image-registry-5b776d6f4f-45tmn\" (UID: \"a29d90f9-887b-4d2e-a145-c985530029f8\") " pod="openshift-image-registry/image-registry-5b776d6f4f-45tmn" Jul 22 00:30:48.975970 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:48.975952 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bf7b370a-c67c-4c3c-b3ba-7533131eace2-serving-cert\") pod \"console-operator-565b7b569d-hb69v\" (UID: \"bf7b370a-c67c-4c3c-b3ba-7533131eace2\") " pod="openshift-console-operator/console-operator-565b7b569d-hb69v" Jul 22 00:30:48.976103 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:48.976081 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a29d90f9-887b-4d2e-a145-c985530029f8-installation-pull-secrets\") pod \"image-registry-5b776d6f4f-45tmn\" (UID: \"a29d90f9-887b-4d2e-a145-c985530029f8\") " pod="openshift-image-registry/image-registry-5b776d6f4f-45tmn" Jul 22 00:30:48.986053 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:48.986033 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lv2dc\" (UniqueName: \"kubernetes.io/projected/a29d90f9-887b-4d2e-a145-c985530029f8-kube-api-access-lv2dc\") pod \"image-registry-5b776d6f4f-45tmn\" (UID: \"a29d90f9-887b-4d2e-a145-c985530029f8\") " pod="openshift-image-registry/image-registry-5b776d6f4f-45tmn" Jul 22 00:30:48.986136 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:48.986035 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx8gc\" (UniqueName: \"kubernetes.io/projected/bf7b370a-c67c-4c3c-b3ba-7533131eace2-kube-api-access-gx8gc\") pod \"console-operator-565b7b569d-hb69v\" (UID: \"bf7b370a-c67c-4c3c-b3ba-7533131eace2\") " pod="openshift-console-operator/console-operator-565b7b569d-hb69v" Jul 22 00:30:48.986136 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:48.986056 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a29d90f9-887b-4d2e-a145-c985530029f8-bound-sa-token\") pod \"image-registry-5b776d6f4f-45tmn\" (UID: \"a29d90f9-887b-4d2e-a145-c985530029f8\") " pod="openshift-image-registry/image-registry-5b776d6f4f-45tmn" Jul 22 00:30:49.036041 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:49.036018 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-hb69v" Jul 22 00:30:49.155673 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:49.155645 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-hb69v"] Jul 22 00:30:49.158275 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:30:49.158247 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf7b370a_c67c_4c3c_b3ba_7533131eace2.slice/crio-07223334ee7a157d671a58ed9e21c8be3cf0364d172a1f4bd7b9d2959294b0f7 WatchSource:0}: Error finding container 07223334ee7a157d671a58ed9e21c8be3cf0364d172a1f4bd7b9d2959294b0f7: Status 404 returned error can't find the container with id 07223334ee7a157d671a58ed9e21c8be3cf0364d172a1f4bd7b9d2959294b0f7 Jul 22 00:30:49.476987 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:49.476958 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a29d90f9-887b-4d2e-a145-c985530029f8-registry-tls\") pod \"image-registry-5b776d6f4f-45tmn\" (UID: \"a29d90f9-887b-4d2e-a145-c985530029f8\") " pod="openshift-image-registry/image-registry-5b776d6f4f-45tmn" Jul 22 00:30:49.477354 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:30:49.477098 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 00:30:49.477354 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:30:49.477113 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b776d6f4f-45tmn: secret "image-registry-tls" not found Jul 22 00:30:49.477354 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:30:49.477162 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a29d90f9-887b-4d2e-a145-c985530029f8-registry-tls podName:a29d90f9-887b-4d2e-a145-c985530029f8 nodeName:}" failed. No retries permitted until 2026-07-22 00:30:50.47714818 +0000 UTC m=+235.833906039 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a29d90f9-887b-4d2e-a145-c985530029f8-registry-tls") pod "image-registry-5b776d6f4f-45tmn" (UID: "a29d90f9-887b-4d2e-a145-c985530029f8") : secret "image-registry-tls" not found Jul 22 00:30:49.788925 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:49.788828 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-hb69v" event={"ID":"bf7b370a-c67c-4c3c-b3ba-7533131eace2","Type":"ContainerStarted","Data":"07223334ee7a157d671a58ed9e21c8be3cf0364d172a1f4bd7b9d2959294b0f7"} Jul 22 00:30:49.790374 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:49.790353 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ff8fbf64f-d7jxg" event={"ID":"20bcbb88-28d1-411f-8abb-a3901542b77a","Type":"ContainerStarted","Data":"edb7d6469a5d41c984e4fe15d7c2bc1e29effc92fab17af29bd9dd6bbecadf9c"} Jul 22 00:30:49.790650 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:49.790630 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ff8fbf64f-d7jxg" Jul 22 00:30:49.791190 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:49.791170 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5ff8fbf64f-d7jxg" Jul 22 00:30:50.486567 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:50.486530 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a29d90f9-887b-4d2e-a145-c985530029f8-registry-tls\") pod \"image-registry-5b776d6f4f-45tmn\" (UID: \"a29d90f9-887b-4d2e-a145-c985530029f8\") " pod="openshift-image-registry/image-registry-5b776d6f4f-45tmn" Jul 22 00:30:50.487055 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:30:50.486731 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 00:30:50.487055 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:30:50.486754 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b776d6f4f-45tmn: secret "image-registry-tls" not found Jul 22 00:30:50.487055 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:30:50.486827 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a29d90f9-887b-4d2e-a145-c985530029f8-registry-tls podName:a29d90f9-887b-4d2e-a145-c985530029f8 nodeName:}" failed. No retries permitted until 2026-07-22 00:30:52.486805605 +0000 UTC m=+237.843563464 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a29d90f9-887b-4d2e-a145-c985530029f8-registry-tls") pod "image-registry-5b776d6f4f-45tmn" (UID: "a29d90f9-887b-4d2e-a145-c985530029f8") : secret "image-registry-tls" not found Jul 22 00:30:52.502231 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:52.502139 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a29d90f9-887b-4d2e-a145-c985530029f8-registry-tls\") pod \"image-registry-5b776d6f4f-45tmn\" (UID: \"a29d90f9-887b-4d2e-a145-c985530029f8\") " pod="openshift-image-registry/image-registry-5b776d6f4f-45tmn" Jul 22 00:30:52.502635 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:30:52.502262 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 00:30:52.502635 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:30:52.502274 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b776d6f4f-45tmn: secret "image-registry-tls" not found Jul 22 00:30:52.502635 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:30:52.502323 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a29d90f9-887b-4d2e-a145-c985530029f8-registry-tls podName:a29d90f9-887b-4d2e-a145-c985530029f8 nodeName:}" failed. No retries permitted until 2026-07-22 00:30:56.502309858 +0000 UTC m=+241.859067713 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a29d90f9-887b-4d2e-a145-c985530029f8-registry-tls") pod "image-registry-5b776d6f4f-45tmn" (UID: "a29d90f9-887b-4d2e-a145-c985530029f8") : secret "image-registry-tls" not found Jul 22 00:30:52.801437 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:52.801358 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-hb69v_bf7b370a-c67c-4c3c-b3ba-7533131eace2/console-operator/0.log" Jul 22 00:30:52.801437 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:52.801398 2574 generic.go:356] "Generic (PLEG): container finished" podID="bf7b370a-c67c-4c3c-b3ba-7533131eace2" containerID="1cc65f6e60de7f90747f6d72beeecb1e854a47757ad002f16c7e7b19cdc52b1a" exitCode=255 Jul 22 00:30:52.801437 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:52.801427 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-hb69v" event={"ID":"bf7b370a-c67c-4c3c-b3ba-7533131eace2","Type":"ContainerDied","Data":"1cc65f6e60de7f90747f6d72beeecb1e854a47757ad002f16c7e7b19cdc52b1a"} Jul 22 00:30:52.801743 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:52.801725 2574 scope.go:117] "RemoveContainer" containerID="1cc65f6e60de7f90747f6d72beeecb1e854a47757ad002f16c7e7b19cdc52b1a" Jul 22 00:30:53.805916 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:53.805883 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-hb69v_bf7b370a-c67c-4c3c-b3ba-7533131eace2/console-operator/1.log" Jul 22 00:30:53.806373 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:53.806256 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-hb69v_bf7b370a-c67c-4c3c-b3ba-7533131eace2/console-operator/0.log" Jul 22 00:30:53.806373 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:53.806292 2574 generic.go:356] "Generic (PLEG): container finished" podID="bf7b370a-c67c-4c3c-b3ba-7533131eace2" containerID="cb77595fbcf38242ad3c8f54c6aa35855fbdeb9ea66d1967dda95bb095d95b2e" exitCode=255 Jul 22 00:30:53.806373 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:53.806339 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-hb69v" event={"ID":"bf7b370a-c67c-4c3c-b3ba-7533131eace2","Type":"ContainerDied","Data":"cb77595fbcf38242ad3c8f54c6aa35855fbdeb9ea66d1967dda95bb095d95b2e"} Jul 22 00:30:53.806483 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:53.806394 2574 scope.go:117] "RemoveContainer" containerID="1cc65f6e60de7f90747f6d72beeecb1e854a47757ad002f16c7e7b19cdc52b1a" Jul 22 00:30:53.806640 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:53.806617 2574 scope.go:117] "RemoveContainer" containerID="cb77595fbcf38242ad3c8f54c6aa35855fbdeb9ea66d1967dda95bb095d95b2e" Jul 22 00:30:53.806832 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:30:53.806811 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-hb69v_openshift-console-operator(bf7b370a-c67c-4c3c-b3ba-7533131eace2)\"" pod="openshift-console-operator/console-operator-565b7b569d-hb69v" podUID="bf7b370a-c67c-4c3c-b3ba-7533131eace2" Jul 22 00:30:53.992289 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:53.992254 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-844qz"] Jul 22 00:30:53.996530 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:53.996515 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-844qz" Jul 22 00:30:53.999387 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:53.999356 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 22 00:30:53.999387 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:53.999366 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qsbbh\"" Jul 22 00:30:53.999565 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:53.999391 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 22 00:30:53.999565 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:53.999368 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 22 00:30:53.999710 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:53.999695 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 22 00:30:54.004122 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:54.004096 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-844qz"] Jul 22 00:30:54.115079 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:54.115007 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxpf6\" (UniqueName: \"kubernetes.io/projected/cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f-kube-api-access-cxpf6\") pod \"insights-runtime-extractor-844qz\" (UID: \"cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f\") " pod="openshift-insights/insights-runtime-extractor-844qz" Jul 22 00:30:54.115218 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:54.115098 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-844qz\" (UID: \"cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f\") " pod="openshift-insights/insights-runtime-extractor-844qz" Jul 22 00:30:54.115218 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:54.115120 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f-crio-socket\") pod \"insights-runtime-extractor-844qz\" (UID: \"cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f\") " pod="openshift-insights/insights-runtime-extractor-844qz" Jul 22 00:30:54.115218 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:54.115138 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f-data-volume\") pod \"insights-runtime-extractor-844qz\" (UID: \"cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f\") " pod="openshift-insights/insights-runtime-extractor-844qz" Jul 22 00:30:54.115218 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:54.115162 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-844qz\" (UID: \"cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f\") " pod="openshift-insights/insights-runtime-extractor-844qz" Jul 22 00:30:54.215776 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:54.215729 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-844qz\" (UID: \"cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f\") " pod="openshift-insights/insights-runtime-extractor-844qz" Jul 22 00:30:54.215951 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:54.215787 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f-crio-socket\") pod \"insights-runtime-extractor-844qz\" (UID: \"cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f\") " pod="openshift-insights/insights-runtime-extractor-844qz" Jul 22 00:30:54.215951 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:54.215817 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f-data-volume\") pod \"insights-runtime-extractor-844qz\" (UID: \"cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f\") " pod="openshift-insights/insights-runtime-extractor-844qz" Jul 22 00:30:54.215951 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:30:54.215843 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 00:30:54.215951 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:54.215853 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-844qz\" (UID: \"cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f\") " pod="openshift-insights/insights-runtime-extractor-844qz" Jul 22 00:30:54.215951 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:54.215913 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f-crio-socket\") pod \"insights-runtime-extractor-844qz\" (UID: \"cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f\") " pod="openshift-insights/insights-runtime-extractor-844qz" Jul 22 00:30:54.215951 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:30:54.215922 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f-kube-rbac-proxy-cm podName:cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f nodeName:}" failed. No retries permitted until 2026-07-22 00:30:54.715902477 +0000 UTC m=+240.072660332 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-844qz" (UID: "cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f") : configmap "kube-rbac-proxy" not found Jul 22 00:30:54.215951 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:30:54.215938 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 00:30:54.216245 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:54.215960 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxpf6\" (UniqueName: \"kubernetes.io/projected/cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f-kube-api-access-cxpf6\") pod \"insights-runtime-extractor-844qz\" (UID: \"cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f\") " pod="openshift-insights/insights-runtime-extractor-844qz" Jul 22 00:30:54.216245 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:30:54.215989 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f-insights-runtime-extractor-tls podName:cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f nodeName:}" failed. No retries permitted until 2026-07-22 00:30:54.715971551 +0000 UTC m=+240.072729431 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-844qz" (UID: "cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f") : secret "insights-runtime-extractor-tls" not found Jul 22 00:30:54.216245 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:54.216149 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f-data-volume\") pod \"insights-runtime-extractor-844qz\" (UID: \"cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f\") " pod="openshift-insights/insights-runtime-extractor-844qz" Jul 22 00:30:54.224348 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:54.224322 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxpf6\" (UniqueName: \"kubernetes.io/projected/cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f-kube-api-access-cxpf6\") pod \"insights-runtime-extractor-844qz\" (UID: \"cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f\") " pod="openshift-insights/insights-runtime-extractor-844qz" Jul 22 00:30:54.719720 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:54.719684 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-844qz\" (UID: \"cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f\") " pod="openshift-insights/insights-runtime-extractor-844qz" Jul 22 00:30:54.719986 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:54.719736 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-844qz\" (UID: \"cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f\") " pod="openshift-insights/insights-runtime-extractor-844qz" Jul 22 00:30:54.719986 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:30:54.719827 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 00:30:54.719986 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:30:54.719849 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 00:30:54.719986 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:30:54.719904 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f-insights-runtime-extractor-tls podName:cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f nodeName:}" failed. No retries permitted until 2026-07-22 00:30:55.719885957 +0000 UTC m=+241.076643812 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-844qz" (UID: "cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f") : secret "insights-runtime-extractor-tls" not found Jul 22 00:30:54.719986 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:30:54.719919 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f-kube-rbac-proxy-cm podName:cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f nodeName:}" failed. No retries permitted until 2026-07-22 00:30:55.719912996 +0000 UTC m=+241.076670850 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-844qz" (UID: "cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f") : configmap "kube-rbac-proxy" not found Jul 22 00:30:54.814212 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:54.814182 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-hb69v_bf7b370a-c67c-4c3c-b3ba-7533131eace2/console-operator/1.log" Jul 22 00:30:54.814594 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:54.814497 2574 scope.go:117] "RemoveContainer" containerID="cb77595fbcf38242ad3c8f54c6aa35855fbdeb9ea66d1967dda95bb095d95b2e" Jul 22 00:30:54.814683 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:30:54.814664 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-hb69v_openshift-console-operator(bf7b370a-c67c-4c3c-b3ba-7533131eace2)\"" pod="openshift-console-operator/console-operator-565b7b569d-hb69v" podUID="bf7b370a-c67c-4c3c-b3ba-7533131eace2" Jul 22 00:30:55.727832 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:55.727790 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-844qz\" (UID: \"cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f\") " pod="openshift-insights/insights-runtime-extractor-844qz" Jul 22 00:30:55.728040 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:55.727891 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-844qz\" (UID: \"cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f\") " pod="openshift-insights/insights-runtime-extractor-844qz" Jul 22 00:30:55.728040 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:30:55.727949 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 00:30:55.728040 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:30:55.727975 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 00:30:55.728040 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:30:55.728016 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f-insights-runtime-extractor-tls podName:cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f nodeName:}" failed. No retries permitted until 2026-07-22 00:30:57.727999462 +0000 UTC m=+243.084757317 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-844qz" (UID: "cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f") : secret "insights-runtime-extractor-tls" not found Jul 22 00:30:55.728040 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:30:55.728031 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f-kube-rbac-proxy-cm podName:cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f nodeName:}" failed. No retries permitted until 2026-07-22 00:30:57.7280245 +0000 UTC m=+243.084782355 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-844qz" (UID: "cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f") : configmap "kube-rbac-proxy" not found Jul 22 00:30:56.533538 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:56.533505 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a29d90f9-887b-4d2e-a145-c985530029f8-registry-tls\") pod \"image-registry-5b776d6f4f-45tmn\" (UID: \"a29d90f9-887b-4d2e-a145-c985530029f8\") " pod="openshift-image-registry/image-registry-5b776d6f4f-45tmn" Jul 22 00:30:56.533944 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:30:56.533638 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 00:30:56.533944 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:30:56.533651 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b776d6f4f-45tmn: secret "image-registry-tls" not found Jul 22 00:30:56.533944 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:30:56.533700 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a29d90f9-887b-4d2e-a145-c985530029f8-registry-tls podName:a29d90f9-887b-4d2e-a145-c985530029f8 nodeName:}" failed. No retries permitted until 2026-07-22 00:31:04.533685951 +0000 UTC m=+249.890443819 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a29d90f9-887b-4d2e-a145-c985530029f8-registry-tls") pod "image-registry-5b776d6f4f-45tmn" (UID: "a29d90f9-887b-4d2e-a145-c985530029f8") : secret "image-registry-tls" not found Jul 22 00:30:56.711034 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:56.711000 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-58598695f7-l22cd"] Jul 22 00:30:56.714004 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:56.713989 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-l22cd" Jul 22 00:30:56.716778 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:56.716752 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 22 00:30:56.716897 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:56.716794 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 22 00:30:56.716897 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:56.716793 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 22 00:30:56.717913 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:56.717886 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 22 00:30:56.717913 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:56.717895 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-c58nl\"" Jul 22 00:30:56.721127 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:56.721108 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-l22cd"] Jul 22 00:30:56.836081 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:56.836003 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ea53daeb-109f-4594-8cdd-de4ead6508e3-signing-cabundle\") pod \"service-ca-58598695f7-l22cd\" (UID: \"ea53daeb-109f-4594-8cdd-de4ead6508e3\") " pod="openshift-service-ca/service-ca-58598695f7-l22cd" Jul 22 00:30:56.836081 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:56.836036 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ea53daeb-109f-4594-8cdd-de4ead6508e3-signing-key\") pod \"service-ca-58598695f7-l22cd\" (UID: \"ea53daeb-109f-4594-8cdd-de4ead6508e3\") " pod="openshift-service-ca/service-ca-58598695f7-l22cd" Jul 22 00:30:56.836081 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:56.836071 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpnxf\" (UniqueName: \"kubernetes.io/projected/ea53daeb-109f-4594-8cdd-de4ead6508e3-kube-api-access-mpnxf\") pod \"service-ca-58598695f7-l22cd\" (UID: \"ea53daeb-109f-4594-8cdd-de4ead6508e3\") " pod="openshift-service-ca/service-ca-58598695f7-l22cd" Jul 22 00:30:56.937030 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:56.937000 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ea53daeb-109f-4594-8cdd-de4ead6508e3-signing-cabundle\") pod \"service-ca-58598695f7-l22cd\" (UID: \"ea53daeb-109f-4594-8cdd-de4ead6508e3\") " pod="openshift-service-ca/service-ca-58598695f7-l22cd" Jul 22 00:30:56.937030 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:56.937034 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ea53daeb-109f-4594-8cdd-de4ead6508e3-signing-key\") pod \"service-ca-58598695f7-l22cd\" (UID: \"ea53daeb-109f-4594-8cdd-de4ead6508e3\") " pod="openshift-service-ca/service-ca-58598695f7-l22cd" Jul 22 00:30:56.937229 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:56.937110 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpnxf\" (UniqueName: \"kubernetes.io/projected/ea53daeb-109f-4594-8cdd-de4ead6508e3-kube-api-access-mpnxf\") pod \"service-ca-58598695f7-l22cd\" (UID: \"ea53daeb-109f-4594-8cdd-de4ead6508e3\") " pod="openshift-service-ca/service-ca-58598695f7-l22cd" Jul 22 00:30:56.937751 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:56.937733 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ea53daeb-109f-4594-8cdd-de4ead6508e3-signing-cabundle\") pod \"service-ca-58598695f7-l22cd\" (UID: \"ea53daeb-109f-4594-8cdd-de4ead6508e3\") " pod="openshift-service-ca/service-ca-58598695f7-l22cd" Jul 22 00:30:56.939334 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:56.939318 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ea53daeb-109f-4594-8cdd-de4ead6508e3-signing-key\") pod \"service-ca-58598695f7-l22cd\" (UID: \"ea53daeb-109f-4594-8cdd-de4ead6508e3\") " pod="openshift-service-ca/service-ca-58598695f7-l22cd" Jul 22 00:30:56.945091 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:56.945066 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpnxf\" (UniqueName: \"kubernetes.io/projected/ea53daeb-109f-4594-8cdd-de4ead6508e3-kube-api-access-mpnxf\") pod \"service-ca-58598695f7-l22cd\" (UID: \"ea53daeb-109f-4594-8cdd-de4ead6508e3\") " pod="openshift-service-ca/service-ca-58598695f7-l22cd" Jul 22 00:30:56.984911 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:56.984887 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zqvbx_35037e44-306a-4ec1-9937-d3f4b3b7891a/dns-node-resolver/0.log" Jul 22 00:30:57.023225 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:57.023199 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-l22cd" Jul 22 00:30:57.146228 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:57.146199 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-l22cd"] Jul 22 00:30:57.149252 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:30:57.149226 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea53daeb_109f_4594_8cdd_de4ead6508e3.slice/crio-738d94bc148ccda45e19ffe80be5c036a944d0f567d2cb2fa755677a066f29ec WatchSource:0}: Error finding container 738d94bc148ccda45e19ffe80be5c036a944d0f567d2cb2fa755677a066f29ec: Status 404 returned error can't find the container with id 738d94bc148ccda45e19ffe80be5c036a944d0f567d2cb2fa755677a066f29ec Jul 22 00:30:57.586060 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:57.586029 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7wkvg_44c9d8f9-4c85-4dfa-8572-2feb0f653720/node-ca/0.log" Jul 22 00:30:57.742941 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:57.742908 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-844qz\" (UID: \"cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f\") " pod="openshift-insights/insights-runtime-extractor-844qz" Jul 22 00:30:57.743105 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:57.742960 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-844qz\" (UID: \"cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f\") " pod="openshift-insights/insights-runtime-extractor-844qz" Jul 22 00:30:57.743105 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:30:57.743050 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 00:30:57.743105 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:30:57.743070 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 00:30:57.743256 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:30:57.743125 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f-kube-rbac-proxy-cm podName:cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f nodeName:}" failed. No retries permitted until 2026-07-22 00:31:01.743104294 +0000 UTC m=+247.099862148 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-844qz" (UID: "cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f") : configmap "kube-rbac-proxy" not found Jul 22 00:30:57.743256 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:30:57.743150 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f-insights-runtime-extractor-tls podName:cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f nodeName:}" failed. No retries permitted until 2026-07-22 00:31:01.74314003 +0000 UTC m=+247.099897911 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-844qz" (UID: "cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f") : secret "insights-runtime-extractor-tls" not found Jul 22 00:30:57.827629 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:57.827579 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-l22cd" event={"ID":"ea53daeb-109f-4594-8cdd-de4ead6508e3","Type":"ContainerStarted","Data":"738d94bc148ccda45e19ffe80be5c036a944d0f567d2cb2fa755677a066f29ec"} Jul 22 00:30:58.585209 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:58.585178 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hsxnj_2b90aaae-368a-4ba1-ba82-7e66aa1e518b/serve-healthcheck-canary/0.log" Jul 22 00:30:59.036678 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:59.036642 2574 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-565b7b569d-hb69v" Jul 22 00:30:59.037089 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:59.036710 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-hb69v" Jul 22 00:30:59.037152 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:59.037133 2574 scope.go:117] "RemoveContainer" containerID="cb77595fbcf38242ad3c8f54c6aa35855fbdeb9ea66d1967dda95bb095d95b2e" Jul 22 00:30:59.037353 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:30:59.037326 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-hb69v_openshift-console-operator(bf7b370a-c67c-4c3c-b3ba-7533131eace2)\"" pod="openshift-console-operator/console-operator-565b7b569d-hb69v" podUID="bf7b370a-c67c-4c3c-b3ba-7533131eace2" Jul 22 00:30:59.835472 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:59.835431 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-l22cd" event={"ID":"ea53daeb-109f-4594-8cdd-de4ead6508e3","Type":"ContainerStarted","Data":"7a0e3559ce8acf1e8a67f820b4f1a644aac54e2a2186bb5a35f0b0126df16464"} Jul 22 00:30:59.856063 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:30:59.856012 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-58598695f7-l22cd" podStartSLOduration=1.6952940600000002 podStartE2EDuration="3.855996309s" totalImagesPullingTime="2.160702249s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:30:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:30:59.85543981 +0000 UTC m=+245.212197699" watchObservedRunningTime="2026-07-22 00:30:59.855996309 +0000 UTC m=+245.212754186" Jul 22 00:31:01.777472 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:01.777438 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-844qz\" (UID: \"cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f\") " pod="openshift-insights/insights-runtime-extractor-844qz" Jul 22 00:31:01.777902 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:01.777508 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-844qz\" (UID: \"cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f\") " pod="openshift-insights/insights-runtime-extractor-844qz" Jul 22 00:31:01.777902 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:31:01.777573 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 00:31:01.777902 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:31:01.777652 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f-kube-rbac-proxy-cm podName:cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f nodeName:}" failed. No retries permitted until 2026-07-22 00:31:09.777635509 +0000 UTC m=+255.134393365 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-844qz" (UID: "cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f") : configmap "kube-rbac-proxy" not found Jul 22 00:31:01.777902 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:31:01.777653 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 00:31:01.777902 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:31:01.777704 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f-insights-runtime-extractor-tls podName:cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f nodeName:}" failed. No retries permitted until 2026-07-22 00:31:09.777688755 +0000 UTC m=+255.134446623 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-844qz" (UID: "cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f") : secret "insights-runtime-extractor-tls" not found Jul 22 00:31:04.595048 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:04.595009 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a29d90f9-887b-4d2e-a145-c985530029f8-registry-tls\") pod \"image-registry-5b776d6f4f-45tmn\" (UID: \"a29d90f9-887b-4d2e-a145-c985530029f8\") " pod="openshift-image-registry/image-registry-5b776d6f4f-45tmn" Jul 22 00:31:04.595514 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:31:04.595134 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 00:31:04.595514 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:31:04.595145 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b776d6f4f-45tmn: secret "image-registry-tls" not found Jul 22 00:31:04.595514 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:31:04.595194 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a29d90f9-887b-4d2e-a145-c985530029f8-registry-tls podName:a29d90f9-887b-4d2e-a145-c985530029f8 nodeName:}" failed. No retries permitted until 2026-07-22 00:31:20.595180248 +0000 UTC m=+265.951938108 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a29d90f9-887b-4d2e-a145-c985530029f8-registry-tls") pod "image-registry-5b776d6f4f-45tmn" (UID: "a29d90f9-887b-4d2e-a145-c985530029f8") : secret "image-registry-tls" not found Jul 22 00:31:06.913119 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:06.913084 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f36a57f8-6e88-43d3-b572-8adf3b0ddb7c-metrics-certs\") pod \"network-metrics-daemon-5xz72\" (UID: \"f36a57f8-6e88-43d3-b572-8adf3b0ddb7c\") " pod="openshift-multus/network-metrics-daemon-5xz72" Jul 22 00:31:06.913548 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:31:06.913265 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 22 00:31:06.913548 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:31:06.913343 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f36a57f8-6e88-43d3-b572-8adf3b0ddb7c-metrics-certs podName:f36a57f8-6e88-43d3-b572-8adf3b0ddb7c nodeName:}" failed. No retries permitted until 2026-07-22 00:33:08.913322205 +0000 UTC m=+374.270080067 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f36a57f8-6e88-43d3-b572-8adf3b0ddb7c-metrics-certs") pod "network-metrics-daemon-5xz72" (UID: "f36a57f8-6e88-43d3-b572-8adf3b0ddb7c") : secret "metrics-daemon-secret" not found Jul 22 00:31:09.834974 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:09.834932 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-844qz\" (UID: \"cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f\") " pod="openshift-insights/insights-runtime-extractor-844qz" Jul 22 00:31:09.835550 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:09.834997 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-844qz\" (UID: \"cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f\") " pod="openshift-insights/insights-runtime-extractor-844qz" Jul 22 00:31:09.835550 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:31:09.835132 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 00:31:09.835550 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:31:09.835202 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f-insights-runtime-extractor-tls podName:cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f nodeName:}" failed. No retries permitted until 2026-07-22 00:31:25.835181867 +0000 UTC m=+271.191939724 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-844qz" (UID: "cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f") : secret "insights-runtime-extractor-tls" not found Jul 22 00:31:09.835550 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:09.835459 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-844qz\" (UID: \"cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f\") " pod="openshift-insights/insights-runtime-extractor-844qz" Jul 22 00:31:10.082890 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:10.082860 2574 scope.go:117] "RemoveContainer" containerID="cb77595fbcf38242ad3c8f54c6aa35855fbdeb9ea66d1967dda95bb095d95b2e" Jul 22 00:31:10.870508 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:10.870480 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-hb69v_bf7b370a-c67c-4c3c-b3ba-7533131eace2/console-operator/1.log" Jul 22 00:31:10.870909 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:10.870551 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-hb69v" event={"ID":"bf7b370a-c67c-4c3c-b3ba-7533131eace2","Type":"ContainerStarted","Data":"05bb6168ca07684ec93bfa819b0c89c6c4e4ff1513e8f371d0314deca9d0da8b"} Jul 22 00:31:10.870909 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:10.870837 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-hb69v" Jul 22 00:31:10.875424 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:10.875406 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-565b7b569d-hb69v" Jul 22 00:31:10.887494 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:10.887450 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-565b7b569d-hb69v" podStartSLOduration=19.865756428 podStartE2EDuration="22.887439719s" totalImagesPullingTime="3.021683291s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:30:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:31:10.887257877 +0000 UTC m=+256.244015801" watchObservedRunningTime="2026-07-22 00:31:10.887439719 +0000 UTC m=+256.244197596" Jul 22 00:31:19.422714 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:19.422633 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5b776d6f4f-45tmn"] Jul 22 00:31:19.423096 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:31:19.422870 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-5b776d6f4f-45tmn" podUID="a29d90f9-887b-4d2e-a145-c985530029f8" Jul 22 00:31:19.517716 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:19.517685 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-779f6b595d-ml5n9"] Jul 22 00:31:19.520538 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:19.520520 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-779f6b595d-ml5n9" Jul 22 00:31:19.529993 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:19.529970 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-779f6b595d-ml5n9"] Jul 22 00:31:19.608334 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:19.608308 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/22bd50aa-4014-4172-b5f1-90cdcbe3cfb0-ca-trust-extracted\") pod \"image-registry-779f6b595d-ml5n9\" (UID: \"22bd50aa-4014-4172-b5f1-90cdcbe3cfb0\") " pod="openshift-image-registry/image-registry-779f6b595d-ml5n9" Jul 22 00:31:19.608334 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:19.608338 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-767z5\" (UniqueName: \"kubernetes.io/projected/22bd50aa-4014-4172-b5f1-90cdcbe3cfb0-kube-api-access-767z5\") pod \"image-registry-779f6b595d-ml5n9\" (UID: \"22bd50aa-4014-4172-b5f1-90cdcbe3cfb0\") " pod="openshift-image-registry/image-registry-779f6b595d-ml5n9" Jul 22 00:31:19.608549 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:19.608364 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/22bd50aa-4014-4172-b5f1-90cdcbe3cfb0-registry-tls\") pod \"image-registry-779f6b595d-ml5n9\" (UID: \"22bd50aa-4014-4172-b5f1-90cdcbe3cfb0\") " pod="openshift-image-registry/image-registry-779f6b595d-ml5n9" Jul 22 00:31:19.608549 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:19.608415 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/22bd50aa-4014-4172-b5f1-90cdcbe3cfb0-trusted-ca\") pod \"image-registry-779f6b595d-ml5n9\" (UID: \"22bd50aa-4014-4172-b5f1-90cdcbe3cfb0\") " pod="openshift-image-registry/image-registry-779f6b595d-ml5n9" Jul 22 00:31:19.608549 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:19.608466 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/22bd50aa-4014-4172-b5f1-90cdcbe3cfb0-image-registry-private-configuration\") pod \"image-registry-779f6b595d-ml5n9\" (UID: \"22bd50aa-4014-4172-b5f1-90cdcbe3cfb0\") " pod="openshift-image-registry/image-registry-779f6b595d-ml5n9" Jul 22 00:31:19.608549 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:19.608506 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/22bd50aa-4014-4172-b5f1-90cdcbe3cfb0-registry-certificates\") pod \"image-registry-779f6b595d-ml5n9\" (UID: \"22bd50aa-4014-4172-b5f1-90cdcbe3cfb0\") " pod="openshift-image-registry/image-registry-779f6b595d-ml5n9" Jul 22 00:31:19.608712 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:19.608559 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/22bd50aa-4014-4172-b5f1-90cdcbe3cfb0-installation-pull-secrets\") pod \"image-registry-779f6b595d-ml5n9\" (UID: \"22bd50aa-4014-4172-b5f1-90cdcbe3cfb0\") " pod="openshift-image-registry/image-registry-779f6b595d-ml5n9" Jul 22 00:31:19.608712 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:19.608582 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/22bd50aa-4014-4172-b5f1-90cdcbe3cfb0-bound-sa-token\") pod \"image-registry-779f6b595d-ml5n9\" (UID: \"22bd50aa-4014-4172-b5f1-90cdcbe3cfb0\") " pod="openshift-image-registry/image-registry-779f6b595d-ml5n9" Jul 22 00:31:19.709980 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:19.709899 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/22bd50aa-4014-4172-b5f1-90cdcbe3cfb0-ca-trust-extracted\") pod \"image-registry-779f6b595d-ml5n9\" (UID: \"22bd50aa-4014-4172-b5f1-90cdcbe3cfb0\") " pod="openshift-image-registry/image-registry-779f6b595d-ml5n9" Jul 22 00:31:19.709980 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:19.709933 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-767z5\" (UniqueName: \"kubernetes.io/projected/22bd50aa-4014-4172-b5f1-90cdcbe3cfb0-kube-api-access-767z5\") pod \"image-registry-779f6b595d-ml5n9\" (UID: \"22bd50aa-4014-4172-b5f1-90cdcbe3cfb0\") " pod="openshift-image-registry/image-registry-779f6b595d-ml5n9" Jul 22 00:31:19.709980 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:19.709969 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/22bd50aa-4014-4172-b5f1-90cdcbe3cfb0-registry-tls\") pod \"image-registry-779f6b595d-ml5n9\" (UID: \"22bd50aa-4014-4172-b5f1-90cdcbe3cfb0\") " pod="openshift-image-registry/image-registry-779f6b595d-ml5n9" Jul 22 00:31:19.710255 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:19.709992 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/22bd50aa-4014-4172-b5f1-90cdcbe3cfb0-trusted-ca\") pod \"image-registry-779f6b595d-ml5n9\" (UID: \"22bd50aa-4014-4172-b5f1-90cdcbe3cfb0\") " pod="openshift-image-registry/image-registry-779f6b595d-ml5n9" Jul 22 00:31:19.710255 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:19.710030 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/22bd50aa-4014-4172-b5f1-90cdcbe3cfb0-image-registry-private-configuration\") pod \"image-registry-779f6b595d-ml5n9\" (UID: \"22bd50aa-4014-4172-b5f1-90cdcbe3cfb0\") " pod="openshift-image-registry/image-registry-779f6b595d-ml5n9" Jul 22 00:31:19.710255 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:19.710075 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/22bd50aa-4014-4172-b5f1-90cdcbe3cfb0-registry-certificates\") pod \"image-registry-779f6b595d-ml5n9\" (UID: \"22bd50aa-4014-4172-b5f1-90cdcbe3cfb0\") " pod="openshift-image-registry/image-registry-779f6b595d-ml5n9" Jul 22 00:31:19.710255 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:19.710123 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/22bd50aa-4014-4172-b5f1-90cdcbe3cfb0-installation-pull-secrets\") pod \"image-registry-779f6b595d-ml5n9\" (UID: \"22bd50aa-4014-4172-b5f1-90cdcbe3cfb0\") " pod="openshift-image-registry/image-registry-779f6b595d-ml5n9" Jul 22 00:31:19.710255 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:19.710148 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/22bd50aa-4014-4172-b5f1-90cdcbe3cfb0-bound-sa-token\") pod \"image-registry-779f6b595d-ml5n9\" (UID: \"22bd50aa-4014-4172-b5f1-90cdcbe3cfb0\") " pod="openshift-image-registry/image-registry-779f6b595d-ml5n9" Jul 22 00:31:19.710481 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:19.710404 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/22bd50aa-4014-4172-b5f1-90cdcbe3cfb0-ca-trust-extracted\") pod \"image-registry-779f6b595d-ml5n9\" (UID: \"22bd50aa-4014-4172-b5f1-90cdcbe3cfb0\") " pod="openshift-image-registry/image-registry-779f6b595d-ml5n9" Jul 22 00:31:19.711058 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:19.711033 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/22bd50aa-4014-4172-b5f1-90cdcbe3cfb0-registry-certificates\") pod \"image-registry-779f6b595d-ml5n9\" (UID: \"22bd50aa-4014-4172-b5f1-90cdcbe3cfb0\") " pod="openshift-image-registry/image-registry-779f6b595d-ml5n9" Jul 22 00:31:19.711174 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:19.711155 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/22bd50aa-4014-4172-b5f1-90cdcbe3cfb0-trusted-ca\") pod \"image-registry-779f6b595d-ml5n9\" (UID: \"22bd50aa-4014-4172-b5f1-90cdcbe3cfb0\") " pod="openshift-image-registry/image-registry-779f6b595d-ml5n9" Jul 22 00:31:19.712503 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:19.712477 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/22bd50aa-4014-4172-b5f1-90cdcbe3cfb0-image-registry-private-configuration\") pod \"image-registry-779f6b595d-ml5n9\" (UID: \"22bd50aa-4014-4172-b5f1-90cdcbe3cfb0\") " pod="openshift-image-registry/image-registry-779f6b595d-ml5n9" Jul 22 00:31:19.712917 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:19.712895 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/22bd50aa-4014-4172-b5f1-90cdcbe3cfb0-registry-tls\") pod \"image-registry-779f6b595d-ml5n9\" (UID: \"22bd50aa-4014-4172-b5f1-90cdcbe3cfb0\") " pod="openshift-image-registry/image-registry-779f6b595d-ml5n9" Jul 22 00:31:19.713000 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:19.712903 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/22bd50aa-4014-4172-b5f1-90cdcbe3cfb0-installation-pull-secrets\") pod \"image-registry-779f6b595d-ml5n9\" (UID: \"22bd50aa-4014-4172-b5f1-90cdcbe3cfb0\") " pod="openshift-image-registry/image-registry-779f6b595d-ml5n9" Jul 22 00:31:19.720153 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:19.720129 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-767z5\" (UniqueName: \"kubernetes.io/projected/22bd50aa-4014-4172-b5f1-90cdcbe3cfb0-kube-api-access-767z5\") pod \"image-registry-779f6b595d-ml5n9\" (UID: \"22bd50aa-4014-4172-b5f1-90cdcbe3cfb0\") " pod="openshift-image-registry/image-registry-779f6b595d-ml5n9" Jul 22 00:31:19.720325 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:19.720309 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/22bd50aa-4014-4172-b5f1-90cdcbe3cfb0-bound-sa-token\") pod \"image-registry-779f6b595d-ml5n9\" (UID: \"22bd50aa-4014-4172-b5f1-90cdcbe3cfb0\") " pod="openshift-image-registry/image-registry-779f6b595d-ml5n9" Jul 22 00:31:19.831685 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:19.831657 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-kstbg\"" Jul 22 00:31:19.839806 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:19.839787 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-779f6b595d-ml5n9" Jul 22 00:31:19.899717 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:19.899577 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b776d6f4f-45tmn" Jul 22 00:31:19.907571 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:19.907546 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b776d6f4f-45tmn" Jul 22 00:31:19.969948 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:19.969753 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-779f6b595d-ml5n9"] Jul 22 00:31:19.972463 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:31:19.972438 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22bd50aa_4014_4172_b5f1_90cdcbe3cfb0.slice/crio-2e9d38ae8c0c7e8e714d5f83e5b62c36c422db36d69aa08fcdf03f3eebc6e15b WatchSource:0}: Error finding container 2e9d38ae8c0c7e8e714d5f83e5b62c36c422db36d69aa08fcdf03f3eebc6e15b: Status 404 returned error can't find the container with id 2e9d38ae8c0c7e8e714d5f83e5b62c36c422db36d69aa08fcdf03f3eebc6e15b Jul 22 00:31:20.013386 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:20.013366 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a29d90f9-887b-4d2e-a145-c985530029f8-installation-pull-secrets\") pod \"a29d90f9-887b-4d2e-a145-c985530029f8\" (UID: \"a29d90f9-887b-4d2e-a145-c985530029f8\") " Jul 22 00:31:20.013473 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:20.013394 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lv2dc\" (UniqueName: \"kubernetes.io/projected/a29d90f9-887b-4d2e-a145-c985530029f8-kube-api-access-lv2dc\") pod \"a29d90f9-887b-4d2e-a145-c985530029f8\" (UID: \"a29d90f9-887b-4d2e-a145-c985530029f8\") " Jul 22 00:31:20.013473 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:20.013443 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a29d90f9-887b-4d2e-a145-c985530029f8-image-registry-private-configuration\") pod \"a29d90f9-887b-4d2e-a145-c985530029f8\" (UID: \"a29d90f9-887b-4d2e-a145-c985530029f8\") " Jul 22 00:31:20.013546 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:20.013519 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a29d90f9-887b-4d2e-a145-c985530029f8-bound-sa-token\") pod \"a29d90f9-887b-4d2e-a145-c985530029f8\" (UID: \"a29d90f9-887b-4d2e-a145-c985530029f8\") " Jul 22 00:31:20.013591 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:20.013544 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a29d90f9-887b-4d2e-a145-c985530029f8-trusted-ca\") pod \"a29d90f9-887b-4d2e-a145-c985530029f8\" (UID: \"a29d90f9-887b-4d2e-a145-c985530029f8\") " Jul 22 00:31:20.013591 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:20.013569 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a29d90f9-887b-4d2e-a145-c985530029f8-registry-certificates\") pod \"a29d90f9-887b-4d2e-a145-c985530029f8\" (UID: \"a29d90f9-887b-4d2e-a145-c985530029f8\") " Jul 22 00:31:20.013706 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:20.013614 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a29d90f9-887b-4d2e-a145-c985530029f8-ca-trust-extracted\") pod \"a29d90f9-887b-4d2e-a145-c985530029f8\" (UID: \"a29d90f9-887b-4d2e-a145-c985530029f8\") " Jul 22 00:31:20.013970 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:20.013941 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a29d90f9-887b-4d2e-a145-c985530029f8-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "a29d90f9-887b-4d2e-a145-c985530029f8" (UID: "a29d90f9-887b-4d2e-a145-c985530029f8"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 00:31:20.014070 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:20.014011 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a29d90f9-887b-4d2e-a145-c985530029f8-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "a29d90f9-887b-4d2e-a145-c985530029f8" (UID: "a29d90f9-887b-4d2e-a145-c985530029f8"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 00:31:20.014070 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:20.014034 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a29d90f9-887b-4d2e-a145-c985530029f8-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "a29d90f9-887b-4d2e-a145-c985530029f8" (UID: "a29d90f9-887b-4d2e-a145-c985530029f8"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 00:31:20.014191 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:20.014119 2574 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a29d90f9-887b-4d2e-a145-c985530029f8-trusted-ca\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 00:31:20.014191 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:20.014144 2574 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a29d90f9-887b-4d2e-a145-c985530029f8-registry-certificates\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 00:31:20.014191 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:20.014159 2574 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a29d90f9-887b-4d2e-a145-c985530029f8-ca-trust-extracted\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 00:31:20.015643 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:20.015616 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a29d90f9-887b-4d2e-a145-c985530029f8-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "a29d90f9-887b-4d2e-a145-c985530029f8" (UID: "a29d90f9-887b-4d2e-a145-c985530029f8"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 00:31:20.015722 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:20.015656 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a29d90f9-887b-4d2e-a145-c985530029f8-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "a29d90f9-887b-4d2e-a145-c985530029f8" (UID: "a29d90f9-887b-4d2e-a145-c985530029f8"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 00:31:20.015722 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:20.015685 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a29d90f9-887b-4d2e-a145-c985530029f8-kube-api-access-lv2dc" (OuterVolumeSpecName: "kube-api-access-lv2dc") pod "a29d90f9-887b-4d2e-a145-c985530029f8" (UID: "a29d90f9-887b-4d2e-a145-c985530029f8"). InnerVolumeSpecName "kube-api-access-lv2dc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 00:31:20.015789 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:20.015756 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a29d90f9-887b-4d2e-a145-c985530029f8-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "a29d90f9-887b-4d2e-a145-c985530029f8" (UID: "a29d90f9-887b-4d2e-a145-c985530029f8"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 00:31:20.115123 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:20.115084 2574 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a29d90f9-887b-4d2e-a145-c985530029f8-image-registry-private-configuration\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 00:31:20.115123 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:20.115119 2574 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a29d90f9-887b-4d2e-a145-c985530029f8-bound-sa-token\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 00:31:20.115306 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:20.115135 2574 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a29d90f9-887b-4d2e-a145-c985530029f8-installation-pull-secrets\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 00:31:20.115306 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:20.115152 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lv2dc\" (UniqueName: \"kubernetes.io/projected/a29d90f9-887b-4d2e-a145-c985530029f8-kube-api-access-lv2dc\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 00:31:20.620417 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:20.620369 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a29d90f9-887b-4d2e-a145-c985530029f8-registry-tls\") pod \"image-registry-5b776d6f4f-45tmn\" (UID: \"a29d90f9-887b-4d2e-a145-c985530029f8\") " pod="openshift-image-registry/image-registry-5b776d6f4f-45tmn" Jul 22 00:31:20.622713 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:20.622694 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a29d90f9-887b-4d2e-a145-c985530029f8-registry-tls\") pod \"image-registry-5b776d6f4f-45tmn\" (UID: \"a29d90f9-887b-4d2e-a145-c985530029f8\") " pod="openshift-image-registry/image-registry-5b776d6f4f-45tmn" Jul 22 00:31:20.720784 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:20.720759 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a29d90f9-887b-4d2e-a145-c985530029f8-registry-tls\") pod \"a29d90f9-887b-4d2e-a145-c985530029f8\" (UID: \"a29d90f9-887b-4d2e-a145-c985530029f8\") " Jul 22 00:31:20.722750 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:20.722721 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a29d90f9-887b-4d2e-a145-c985530029f8-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "a29d90f9-887b-4d2e-a145-c985530029f8" (UID: "a29d90f9-887b-4d2e-a145-c985530029f8"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 00:31:20.822315 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:20.822269 2574 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a29d90f9-887b-4d2e-a145-c985530029f8-registry-tls\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 00:31:20.904073 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:20.904048 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b776d6f4f-45tmn" Jul 22 00:31:20.904073 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:20.904061 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-779f6b595d-ml5n9" event={"ID":"22bd50aa-4014-4172-b5f1-90cdcbe3cfb0","Type":"ContainerStarted","Data":"3393a7f86e51ea8ebdfd96a438a1666cad69cb2c79167d1b617b058d058424ea"} Jul 22 00:31:20.904278 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:20.904106 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-779f6b595d-ml5n9" event={"ID":"22bd50aa-4014-4172-b5f1-90cdcbe3cfb0","Type":"ContainerStarted","Data":"2e9d38ae8c0c7e8e714d5f83e5b62c36c422db36d69aa08fcdf03f3eebc6e15b"} Jul 22 00:31:20.904278 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:20.904224 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-779f6b595d-ml5n9" Jul 22 00:31:20.923467 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:20.923322 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-779f6b595d-ml5n9" podStartSLOduration=1.923306236 podStartE2EDuration="1.923306236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:31:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:31:20.922800587 +0000 UTC m=+266.279558507" watchObservedRunningTime="2026-07-22 00:31:20.923306236 +0000 UTC m=+266.280064116" Jul 22 00:31:20.952815 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:20.952790 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5b776d6f4f-45tmn"] Jul 22 00:31:20.957014 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:20.956969 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5b776d6f4f-45tmn"] Jul 22 00:31:21.086399 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:21.086356 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a29d90f9-887b-4d2e-a145-c985530029f8" path="/var/lib/kubelet/pods/a29d90f9-887b-4d2e-a145-c985530029f8/volumes" Jul 22 00:31:25.861452 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:25.861411 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-844qz\" (UID: \"cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f\") " pod="openshift-insights/insights-runtime-extractor-844qz" Jul 22 00:31:25.863684 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:25.863665 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-844qz\" (UID: \"cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f\") " pod="openshift-insights/insights-runtime-extractor-844qz" Jul 22 00:31:26.108040 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:26.108003 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qsbbh\"" Jul 22 00:31:26.116044 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:26.115999 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-844qz" Jul 22 00:31:26.238825 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:26.238791 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-844qz"] Jul 22 00:31:26.241801 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:31:26.241775 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcce653d8_6ab6_49b0_9a93_b03a8cf6ec0f.slice/crio-84757187b84c31c71dea325462f6717b568c2b9e1405e56718406295692343ee WatchSource:0}: Error finding container 84757187b84c31c71dea325462f6717b568c2b9e1405e56718406295692343ee: Status 404 returned error can't find the container with id 84757187b84c31c71dea325462f6717b568c2b9e1405e56718406295692343ee Jul 22 00:31:26.548203 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:26.548169 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-hvttw"] Jul 22 00:31:26.552718 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:26.552700 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-hvttw" Jul 22 00:31:26.555200 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:26.555178 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 22 00:31:26.555315 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:26.555202 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 22 00:31:26.556415 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:26.556390 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 22 00:31:26.556415 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:26.556407 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 22 00:31:26.556562 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:26.556392 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 22 00:31:26.556562 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:26.556463 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-2fhs9\"" Jul 22 00:31:26.559980 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:26.559657 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-hvttw"] Jul 22 00:31:26.666485 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:26.666451 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtfxq\" (UniqueName: \"kubernetes.io/projected/2bc91e32-84b8-4e76-b19f-d0de3c4bddcc-kube-api-access-rtfxq\") pod \"prometheus-operator-5945b5cfd-hvttw\" (UID: \"2bc91e32-84b8-4e76-b19f-d0de3c4bddcc\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-hvttw" Jul 22 00:31:26.666679 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:26.666536 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/2bc91e32-84b8-4e76-b19f-d0de3c4bddcc-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-hvttw\" (UID: \"2bc91e32-84b8-4e76-b19f-d0de3c4bddcc\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-hvttw" Jul 22 00:31:26.666679 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:26.666593 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2bc91e32-84b8-4e76-b19f-d0de3c4bddcc-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-hvttw\" (UID: \"2bc91e32-84b8-4e76-b19f-d0de3c4bddcc\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-hvttw" Jul 22 00:31:26.666810 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:26.666683 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2bc91e32-84b8-4e76-b19f-d0de3c4bddcc-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-hvttw\" (UID: \"2bc91e32-84b8-4e76-b19f-d0de3c4bddcc\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-hvttw" Jul 22 00:31:26.768004 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:26.767968 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2bc91e32-84b8-4e76-b19f-d0de3c4bddcc-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-hvttw\" (UID: \"2bc91e32-84b8-4e76-b19f-d0de3c4bddcc\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-hvttw" Jul 22 00:31:26.768176 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:26.768051 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2bc91e32-84b8-4e76-b19f-d0de3c4bddcc-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-hvttw\" (UID: \"2bc91e32-84b8-4e76-b19f-d0de3c4bddcc\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-hvttw" Jul 22 00:31:26.768176 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:26.768087 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtfxq\" (UniqueName: \"kubernetes.io/projected/2bc91e32-84b8-4e76-b19f-d0de3c4bddcc-kube-api-access-rtfxq\") pod \"prometheus-operator-5945b5cfd-hvttw\" (UID: \"2bc91e32-84b8-4e76-b19f-d0de3c4bddcc\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-hvttw" Jul 22 00:31:26.768176 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:26.768143 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/2bc91e32-84b8-4e76-b19f-d0de3c4bddcc-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-hvttw\" (UID: \"2bc91e32-84b8-4e76-b19f-d0de3c4bddcc\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-hvttw" Jul 22 00:31:26.768861 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:26.768834 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2bc91e32-84b8-4e76-b19f-d0de3c4bddcc-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-hvttw\" (UID: \"2bc91e32-84b8-4e76-b19f-d0de3c4bddcc\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-hvttw" Jul 22 00:31:26.772164 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:26.772123 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2bc91e32-84b8-4e76-b19f-d0de3c4bddcc-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-hvttw\" (UID: \"2bc91e32-84b8-4e76-b19f-d0de3c4bddcc\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-hvttw" Jul 22 00:31:26.773419 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:26.773393 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/2bc91e32-84b8-4e76-b19f-d0de3c4bddcc-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-hvttw\" (UID: \"2bc91e32-84b8-4e76-b19f-d0de3c4bddcc\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-hvttw" Jul 22 00:31:26.777332 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:26.777290 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtfxq\" (UniqueName: \"kubernetes.io/projected/2bc91e32-84b8-4e76-b19f-d0de3c4bddcc-kube-api-access-rtfxq\") pod \"prometheus-operator-5945b5cfd-hvttw\" (UID: \"2bc91e32-84b8-4e76-b19f-d0de3c4bddcc\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-hvttw" Jul 22 00:31:26.862405 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:26.862323 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-hvttw" Jul 22 00:31:26.924970 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:26.924865 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-844qz" event={"ID":"cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f","Type":"ContainerStarted","Data":"84757187b84c31c71dea325462f6717b568c2b9e1405e56718406295692343ee"} Jul 22 00:31:27.024096 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:27.024065 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-hvttw"] Jul 22 00:31:27.027615 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:31:27.027562 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bc91e32_84b8_4e76_b19f_d0de3c4bddcc.slice/crio-77bcb254a79be6ecc02217c22680150368482f61eed8d74d76ffb37cfd552c1f WatchSource:0}: Error finding container 77bcb254a79be6ecc02217c22680150368482f61eed8d74d76ffb37cfd552c1f: Status 404 returned error can't find the container with id 77bcb254a79be6ecc02217c22680150368482f61eed8d74d76ffb37cfd552c1f Jul 22 00:31:27.929922 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:27.929889 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-hvttw" event={"ID":"2bc91e32-84b8-4e76-b19f-d0de3c4bddcc","Type":"ContainerStarted","Data":"77bcb254a79be6ecc02217c22680150368482f61eed8d74d76ffb37cfd552c1f"} Jul 22 00:31:27.985503 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:27.985469 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-69f8ffd6f5-vm74q"] Jul 22 00:31:27.988818 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:27.988783 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69f8ffd6f5-vm74q" Jul 22 00:31:27.993917 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:27.993869 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-t9krq\"" Jul 22 00:31:27.994215 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:27.994186 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 22 00:31:27.995174 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:27.995039 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 22 00:31:27.995174 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:27.995080 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 22 00:31:27.995378 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:27.995260 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 22 00:31:27.995378 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:27.995290 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 22 00:31:27.995551 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:27.995534 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 22 00:31:28.000706 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:28.000367 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 22 00:31:28.002517 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:28.002460 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-69f8ffd6f5-vm74q"] Jul 22 00:31:28.003194 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:28.003019 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 22 00:31:28.080114 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:28.080074 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c04f1ab4-85e1-4865-a055-dd4d84315477-oauth-serving-cert\") pod \"console-69f8ffd6f5-vm74q\" (UID: \"c04f1ab4-85e1-4865-a055-dd4d84315477\") " pod="openshift-console/console-69f8ffd6f5-vm74q" Jul 22 00:31:28.080305 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:28.080151 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c04f1ab4-85e1-4865-a055-dd4d84315477-console-serving-cert\") pod \"console-69f8ffd6f5-vm74q\" (UID: \"c04f1ab4-85e1-4865-a055-dd4d84315477\") " pod="openshift-console/console-69f8ffd6f5-vm74q" Jul 22 00:31:28.080305 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:28.080176 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c04f1ab4-85e1-4865-a055-dd4d84315477-service-ca\") pod \"console-69f8ffd6f5-vm74q\" (UID: \"c04f1ab4-85e1-4865-a055-dd4d84315477\") " pod="openshift-console/console-69f8ffd6f5-vm74q" Jul 22 00:31:28.080305 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:28.080216 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c04f1ab4-85e1-4865-a055-dd4d84315477-trusted-ca-bundle\") pod \"console-69f8ffd6f5-vm74q\" (UID: \"c04f1ab4-85e1-4865-a055-dd4d84315477\") " pod="openshift-console/console-69f8ffd6f5-vm74q" Jul 22 00:31:28.080305 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:28.080289 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8h8s\" (UniqueName: \"kubernetes.io/projected/c04f1ab4-85e1-4865-a055-dd4d84315477-kube-api-access-w8h8s\") pod \"console-69f8ffd6f5-vm74q\" (UID: \"c04f1ab4-85e1-4865-a055-dd4d84315477\") " pod="openshift-console/console-69f8ffd6f5-vm74q" Jul 22 00:31:28.080531 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:28.080351 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c04f1ab4-85e1-4865-a055-dd4d84315477-console-config\") pod \"console-69f8ffd6f5-vm74q\" (UID: \"c04f1ab4-85e1-4865-a055-dd4d84315477\") " pod="openshift-console/console-69f8ffd6f5-vm74q" Jul 22 00:31:28.080531 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:28.080381 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c04f1ab4-85e1-4865-a055-dd4d84315477-console-oauth-config\") pod \"console-69f8ffd6f5-vm74q\" (UID: \"c04f1ab4-85e1-4865-a055-dd4d84315477\") " pod="openshift-console/console-69f8ffd6f5-vm74q" Jul 22 00:31:28.181697 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:28.181596 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c04f1ab4-85e1-4865-a055-dd4d84315477-oauth-serving-cert\") pod \"console-69f8ffd6f5-vm74q\" (UID: \"c04f1ab4-85e1-4865-a055-dd4d84315477\") " pod="openshift-console/console-69f8ffd6f5-vm74q" Jul 22 00:31:28.181697 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:28.181691 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c04f1ab4-85e1-4865-a055-dd4d84315477-console-serving-cert\") pod \"console-69f8ffd6f5-vm74q\" (UID: \"c04f1ab4-85e1-4865-a055-dd4d84315477\") " pod="openshift-console/console-69f8ffd6f5-vm74q" Jul 22 00:31:28.181926 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:28.181716 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c04f1ab4-85e1-4865-a055-dd4d84315477-service-ca\") pod \"console-69f8ffd6f5-vm74q\" (UID: \"c04f1ab4-85e1-4865-a055-dd4d84315477\") " pod="openshift-console/console-69f8ffd6f5-vm74q" Jul 22 00:31:28.181926 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:28.181756 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c04f1ab4-85e1-4865-a055-dd4d84315477-trusted-ca-bundle\") pod \"console-69f8ffd6f5-vm74q\" (UID: \"c04f1ab4-85e1-4865-a055-dd4d84315477\") " pod="openshift-console/console-69f8ffd6f5-vm74q" Jul 22 00:31:28.181926 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:28.181821 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8h8s\" (UniqueName: \"kubernetes.io/projected/c04f1ab4-85e1-4865-a055-dd4d84315477-kube-api-access-w8h8s\") pod \"console-69f8ffd6f5-vm74q\" (UID: \"c04f1ab4-85e1-4865-a055-dd4d84315477\") " pod="openshift-console/console-69f8ffd6f5-vm74q" Jul 22 00:31:28.181926 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:28.181859 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c04f1ab4-85e1-4865-a055-dd4d84315477-console-config\") pod \"console-69f8ffd6f5-vm74q\" (UID: \"c04f1ab4-85e1-4865-a055-dd4d84315477\") " pod="openshift-console/console-69f8ffd6f5-vm74q" Jul 22 00:31:28.181926 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:28.181890 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c04f1ab4-85e1-4865-a055-dd4d84315477-console-oauth-config\") pod \"console-69f8ffd6f5-vm74q\" (UID: \"c04f1ab4-85e1-4865-a055-dd4d84315477\") " pod="openshift-console/console-69f8ffd6f5-vm74q" Jul 22 00:31:28.182439 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:28.182415 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c04f1ab4-85e1-4865-a055-dd4d84315477-oauth-serving-cert\") pod \"console-69f8ffd6f5-vm74q\" (UID: \"c04f1ab4-85e1-4865-a055-dd4d84315477\") " pod="openshift-console/console-69f8ffd6f5-vm74q" Jul 22 00:31:28.183274 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:28.182953 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c04f1ab4-85e1-4865-a055-dd4d84315477-trusted-ca-bundle\") pod \"console-69f8ffd6f5-vm74q\" (UID: \"c04f1ab4-85e1-4865-a055-dd4d84315477\") " pod="openshift-console/console-69f8ffd6f5-vm74q" Jul 22 00:31:28.183274 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:28.183045 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c04f1ab4-85e1-4865-a055-dd4d84315477-service-ca\") pod \"console-69f8ffd6f5-vm74q\" (UID: \"c04f1ab4-85e1-4865-a055-dd4d84315477\") " pod="openshift-console/console-69f8ffd6f5-vm74q" Jul 22 00:31:28.183274 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:28.183221 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c04f1ab4-85e1-4865-a055-dd4d84315477-console-config\") pod \"console-69f8ffd6f5-vm74q\" (UID: \"c04f1ab4-85e1-4865-a055-dd4d84315477\") " pod="openshift-console/console-69f8ffd6f5-vm74q" Jul 22 00:31:28.185114 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:28.185090 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c04f1ab4-85e1-4865-a055-dd4d84315477-console-oauth-config\") pod \"console-69f8ffd6f5-vm74q\" (UID: \"c04f1ab4-85e1-4865-a055-dd4d84315477\") " pod="openshift-console/console-69f8ffd6f5-vm74q" Jul 22 00:31:28.185657 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:28.185634 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c04f1ab4-85e1-4865-a055-dd4d84315477-console-serving-cert\") pod \"console-69f8ffd6f5-vm74q\" (UID: \"c04f1ab4-85e1-4865-a055-dd4d84315477\") " pod="openshift-console/console-69f8ffd6f5-vm74q" Jul 22 00:31:28.190848 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:28.190813 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8h8s\" (UniqueName: \"kubernetes.io/projected/c04f1ab4-85e1-4865-a055-dd4d84315477-kube-api-access-w8h8s\") pod \"console-69f8ffd6f5-vm74q\" (UID: \"c04f1ab4-85e1-4865-a055-dd4d84315477\") " pod="openshift-console/console-69f8ffd6f5-vm74q" Jul 22 00:31:28.303395 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:28.303154 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69f8ffd6f5-vm74q" Jul 22 00:31:28.728977 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:28.728880 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-69f8ffd6f5-vm74q"] Jul 22 00:31:28.935507 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:28.935467 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-hvttw" event={"ID":"2bc91e32-84b8-4e76-b19f-d0de3c4bddcc","Type":"ContainerStarted","Data":"5670e96adae2374b7b61e76bc7e51a1b01977f1d852b2e87e9df12f48c39e74d"} Jul 22 00:31:30.756582 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:31:30.756548 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc04f1ab4_85e1_4865_a055_dd4d84315477.slice/crio-fb895e0a26a511d0cd7b89c6002c76981cfd977e2f3f9e47ace1a2f447fb0d6d WatchSource:0}: Error finding container fb895e0a26a511d0cd7b89c6002c76981cfd977e2f3f9e47ace1a2f447fb0d6d: Status 404 returned error can't find the container with id fb895e0a26a511d0cd7b89c6002c76981cfd977e2f3f9e47ace1a2f447fb0d6d Jul 22 00:31:30.946218 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:30.946180 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69f8ffd6f5-vm74q" event={"ID":"c04f1ab4-85e1-4865-a055-dd4d84315477","Type":"ContainerStarted","Data":"fb895e0a26a511d0cd7b89c6002c76981cfd977e2f3f9e47ace1a2f447fb0d6d"} Jul 22 00:31:31.950195 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:31.950163 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-844qz" event={"ID":"cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f","Type":"ContainerStarted","Data":"e9003c92b16632a38888aa423353cbbedabff17fe8c86e2df87679f41647559a"} Jul 22 00:31:31.951960 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:31.951934 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-hvttw" event={"ID":"2bc91e32-84b8-4e76-b19f-d0de3c4bddcc","Type":"ContainerStarted","Data":"26df7155572e5590ed920b8ee1c0d7c994f1d7746495c052d22b22977c19bc25"} Jul 22 00:31:31.968250 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:31.968202 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-5945b5cfd-hvttw" podStartSLOduration=4.366516268 podStartE2EDuration="5.968182583s" totalImagesPullingTime="1.601666315s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:31:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:31:31.966767779 +0000 UTC m=+277.323525680" watchObservedRunningTime="2026-07-22 00:31:31.968182583 +0000 UTC m=+277.324940536" Jul 22 00:31:33.890885 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:33.890846 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-fsnlx"] Jul 22 00:31:33.899916 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:33.899886 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-fsnlx" Jul 22 00:31:33.905020 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:33.904198 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-xvq7l"] Jul 22 00:31:33.907222 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:33.906111 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-5wwtn\"" Jul 22 00:31:33.907222 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:33.906352 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 22 00:31:33.907222 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:33.906632 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 22 00:31:33.907222 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:33.906817 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 22 00:31:33.910738 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:33.910646 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-fsnlx"] Jul 22 00:31:33.910836 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:33.910789 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xvq7l" Jul 22 00:31:33.914239 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:33.913532 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 22 00:31:33.914239 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:33.913765 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 22 00:31:33.914239 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:33.913821 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 22 00:31:33.914239 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:33.914078 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-49khf\"" Jul 22 00:31:34.035237 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:34.035051 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4d7dbfcd-8818-437c-b363-6c8abd0c52b6-sys\") pod \"node-exporter-xvq7l\" (UID: \"4d7dbfcd-8818-437c-b363-6c8abd0c52b6\") " pod="openshift-monitoring/node-exporter-xvq7l" Jul 22 00:31:34.035237 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:34.035126 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/65e8b186-5032-46e9-b65c-5fbc08f5d68d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-fsnlx\" (UID: \"65e8b186-5032-46e9-b65c-5fbc08f5d68d\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-fsnlx" Jul 22 00:31:34.035237 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:34.035155 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4d7dbfcd-8818-437c-b363-6c8abd0c52b6-metrics-client-ca\") pod \"node-exporter-xvq7l\" (UID: \"4d7dbfcd-8818-437c-b363-6c8abd0c52b6\") " pod="openshift-monitoring/node-exporter-xvq7l" Jul 22 00:31:34.035538 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:34.035247 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/65e8b186-5032-46e9-b65c-5fbc08f5d68d-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-fsnlx\" (UID: \"65e8b186-5032-46e9-b65c-5fbc08f5d68d\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-fsnlx" Jul 22 00:31:34.035538 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:34.035271 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4d7dbfcd-8818-437c-b363-6c8abd0c52b6-root\") pod \"node-exporter-xvq7l\" (UID: \"4d7dbfcd-8818-437c-b363-6c8abd0c52b6\") " pod="openshift-monitoring/node-exporter-xvq7l" Jul 22 00:31:34.035538 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:34.035298 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4d7dbfcd-8818-437c-b363-6c8abd0c52b6-node-exporter-wtmp\") pod \"node-exporter-xvq7l\" (UID: \"4d7dbfcd-8818-437c-b363-6c8abd0c52b6\") " pod="openshift-monitoring/node-exporter-xvq7l" Jul 22 00:31:34.035538 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:34.035323 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqd88\" (UniqueName: \"kubernetes.io/projected/4d7dbfcd-8818-437c-b363-6c8abd0c52b6-kube-api-access-jqd88\") pod \"node-exporter-xvq7l\" (UID: \"4d7dbfcd-8818-437c-b363-6c8abd0c52b6\") " pod="openshift-monitoring/node-exporter-xvq7l" Jul 22 00:31:34.035538 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:34.035393 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/65e8b186-5032-46e9-b65c-5fbc08f5d68d-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-fsnlx\" (UID: \"65e8b186-5032-46e9-b65c-5fbc08f5d68d\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-fsnlx" Jul 22 00:31:34.035538 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:34.035426 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4d7dbfcd-8818-437c-b363-6c8abd0c52b6-node-exporter-accelerators-collector-config\") pod \"node-exporter-xvq7l\" (UID: \"4d7dbfcd-8818-437c-b363-6c8abd0c52b6\") " pod="openshift-monitoring/node-exporter-xvq7l" Jul 22 00:31:34.035538 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:34.035462 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4d7dbfcd-8818-437c-b363-6c8abd0c52b6-node-exporter-tls\") pod \"node-exporter-xvq7l\" (UID: \"4d7dbfcd-8818-437c-b363-6c8abd0c52b6\") " pod="openshift-monitoring/node-exporter-xvq7l" Jul 22 00:31:34.035538 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:34.035489 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4d7dbfcd-8818-437c-b363-6c8abd0c52b6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xvq7l\" (UID: \"4d7dbfcd-8818-437c-b363-6c8abd0c52b6\") " pod="openshift-monitoring/node-exporter-xvq7l" Jul 22 00:31:34.035538 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:34.035528 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k64x4\" (UniqueName: \"kubernetes.io/projected/65e8b186-5032-46e9-b65c-5fbc08f5d68d-kube-api-access-k64x4\") pod \"kube-state-metrics-56c5594674-fsnlx\" (UID: \"65e8b186-5032-46e9-b65c-5fbc08f5d68d\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-fsnlx" Jul 22 00:31:34.035984 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:34.035561 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4d7dbfcd-8818-437c-b363-6c8abd0c52b6-node-exporter-textfile\") pod \"node-exporter-xvq7l\" (UID: \"4d7dbfcd-8818-437c-b363-6c8abd0c52b6\") " pod="openshift-monitoring/node-exporter-xvq7l" Jul 22 00:31:34.035984 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:34.035584 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/65e8b186-5032-46e9-b65c-5fbc08f5d68d-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-fsnlx\" (UID: \"65e8b186-5032-46e9-b65c-5fbc08f5d68d\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-fsnlx" Jul 22 00:31:34.035984 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:34.035638 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/65e8b186-5032-46e9-b65c-5fbc08f5d68d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-fsnlx\" (UID: \"65e8b186-5032-46e9-b65c-5fbc08f5d68d\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-fsnlx" Jul 22 00:31:34.136813 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:34.136114 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/65e8b186-5032-46e9-b65c-5fbc08f5d68d-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-fsnlx\" (UID: \"65e8b186-5032-46e9-b65c-5fbc08f5d68d\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-fsnlx" Jul 22 00:31:34.136813 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:34.136164 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4d7dbfcd-8818-437c-b363-6c8abd0c52b6-node-exporter-accelerators-collector-config\") pod \"node-exporter-xvq7l\" (UID: \"4d7dbfcd-8818-437c-b363-6c8abd0c52b6\") " pod="openshift-monitoring/node-exporter-xvq7l" Jul 22 00:31:34.136813 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:34.136193 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4d7dbfcd-8818-437c-b363-6c8abd0c52b6-node-exporter-tls\") pod \"node-exporter-xvq7l\" (UID: \"4d7dbfcd-8818-437c-b363-6c8abd0c52b6\") " pod="openshift-monitoring/node-exporter-xvq7l" Jul 22 00:31:34.136813 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:31:34.136329 2574 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 22 00:31:34.136813 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:34.136350 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4d7dbfcd-8818-437c-b363-6c8abd0c52b6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xvq7l\" (UID: \"4d7dbfcd-8818-437c-b363-6c8abd0c52b6\") " pod="openshift-monitoring/node-exporter-xvq7l" Jul 22 00:31:34.136813 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:31:34.136402 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65e8b186-5032-46e9-b65c-5fbc08f5d68d-kube-state-metrics-tls podName:65e8b186-5032-46e9-b65c-5fbc08f5d68d nodeName:}" failed. No retries permitted until 2026-07-22 00:31:34.636378676 +0000 UTC m=+279.993136547 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/65e8b186-5032-46e9-b65c-5fbc08f5d68d-kube-state-metrics-tls") pod "kube-state-metrics-56c5594674-fsnlx" (UID: "65e8b186-5032-46e9-b65c-5fbc08f5d68d") : secret "kube-state-metrics-tls" not found Jul 22 00:31:34.136813 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:34.136438 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k64x4\" (UniqueName: \"kubernetes.io/projected/65e8b186-5032-46e9-b65c-5fbc08f5d68d-kube-api-access-k64x4\") pod \"kube-state-metrics-56c5594674-fsnlx\" (UID: \"65e8b186-5032-46e9-b65c-5fbc08f5d68d\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-fsnlx" Jul 22 00:31:34.136813 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:34.136478 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4d7dbfcd-8818-437c-b363-6c8abd0c52b6-node-exporter-textfile\") pod \"node-exporter-xvq7l\" (UID: \"4d7dbfcd-8818-437c-b363-6c8abd0c52b6\") " pod="openshift-monitoring/node-exporter-xvq7l" Jul 22 00:31:34.136813 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:34.136503 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/65e8b186-5032-46e9-b65c-5fbc08f5d68d-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-fsnlx\" (UID: \"65e8b186-5032-46e9-b65c-5fbc08f5d68d\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-fsnlx" Jul 22 00:31:34.136813 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:34.136535 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/65e8b186-5032-46e9-b65c-5fbc08f5d68d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-fsnlx\" (UID: \"65e8b186-5032-46e9-b65c-5fbc08f5d68d\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-fsnlx" Jul 22 00:31:34.136813 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:34.136576 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4d7dbfcd-8818-437c-b363-6c8abd0c52b6-sys\") pod \"node-exporter-xvq7l\" (UID: \"4d7dbfcd-8818-437c-b363-6c8abd0c52b6\") " pod="openshift-monitoring/node-exporter-xvq7l" Jul 22 00:31:34.136813 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:34.136640 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/65e8b186-5032-46e9-b65c-5fbc08f5d68d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-fsnlx\" (UID: \"65e8b186-5032-46e9-b65c-5fbc08f5d68d\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-fsnlx" Jul 22 00:31:34.136813 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:34.136665 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4d7dbfcd-8818-437c-b363-6c8abd0c52b6-metrics-client-ca\") pod \"node-exporter-xvq7l\" (UID: \"4d7dbfcd-8818-437c-b363-6c8abd0c52b6\") " pod="openshift-monitoring/node-exporter-xvq7l" Jul 22 00:31:34.136813 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:34.136718 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/65e8b186-5032-46e9-b65c-5fbc08f5d68d-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-fsnlx\" (UID: \"65e8b186-5032-46e9-b65c-5fbc08f5d68d\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-fsnlx" Jul 22 00:31:34.136813 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:34.136741 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4d7dbfcd-8818-437c-b363-6c8abd0c52b6-root\") pod \"node-exporter-xvq7l\" (UID: \"4d7dbfcd-8818-437c-b363-6c8abd0c52b6\") " pod="openshift-monitoring/node-exporter-xvq7l" Jul 22 00:31:34.136813 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:34.136768 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4d7dbfcd-8818-437c-b363-6c8abd0c52b6-node-exporter-wtmp\") pod \"node-exporter-xvq7l\" (UID: \"4d7dbfcd-8818-437c-b363-6c8abd0c52b6\") " pod="openshift-monitoring/node-exporter-xvq7l" Jul 22 00:31:34.137795 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:34.136800 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqd88\" (UniqueName: \"kubernetes.io/projected/4d7dbfcd-8818-437c-b363-6c8abd0c52b6-kube-api-access-jqd88\") pod \"node-exporter-xvq7l\" (UID: \"4d7dbfcd-8818-437c-b363-6c8abd0c52b6\") " pod="openshift-monitoring/node-exporter-xvq7l" Jul 22 00:31:34.137795 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:34.137267 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/65e8b186-5032-46e9-b65c-5fbc08f5d68d-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-fsnlx\" (UID: \"65e8b186-5032-46e9-b65c-5fbc08f5d68d\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-fsnlx" Jul 22 00:31:34.137795 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:34.137295 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4d7dbfcd-8818-437c-b363-6c8abd0c52b6-root\") pod \"node-exporter-xvq7l\" (UID: \"4d7dbfcd-8818-437c-b363-6c8abd0c52b6\") " pod="openshift-monitoring/node-exporter-xvq7l" Jul 22 00:31:34.137795 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:34.137333 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4d7dbfcd-8818-437c-b363-6c8abd0c52b6-sys\") pod \"node-exporter-xvq7l\" (UID: \"4d7dbfcd-8818-437c-b363-6c8abd0c52b6\") " pod="openshift-monitoring/node-exporter-xvq7l" Jul 22 00:31:34.137795 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:34.137431 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4d7dbfcd-8818-437c-b363-6c8abd0c52b6-node-exporter-wtmp\") pod \"node-exporter-xvq7l\" (UID: \"4d7dbfcd-8818-437c-b363-6c8abd0c52b6\") " pod="openshift-monitoring/node-exporter-xvq7l" Jul 22 00:31:34.137795 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:34.137513 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4d7dbfcd-8818-437c-b363-6c8abd0c52b6-node-exporter-textfile\") pod \"node-exporter-xvq7l\" (UID: \"4d7dbfcd-8818-437c-b363-6c8abd0c52b6\") " pod="openshift-monitoring/node-exporter-xvq7l" Jul 22 00:31:34.138110 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:34.137940 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4d7dbfcd-8818-437c-b363-6c8abd0c52b6-metrics-client-ca\") pod \"node-exporter-xvq7l\" (UID: \"4d7dbfcd-8818-437c-b363-6c8abd0c52b6\") " pod="openshift-monitoring/node-exporter-xvq7l" Jul 22 00:31:34.138110 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:34.137966 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4d7dbfcd-8818-437c-b363-6c8abd0c52b6-node-exporter-accelerators-collector-config\") pod \"node-exporter-xvq7l\" (UID: \"4d7dbfcd-8818-437c-b363-6c8abd0c52b6\") " pod="openshift-monitoring/node-exporter-xvq7l" Jul 22 00:31:34.138110 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:34.137965 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/65e8b186-5032-46e9-b65c-5fbc08f5d68d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-fsnlx\" (UID: \"65e8b186-5032-46e9-b65c-5fbc08f5d68d\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-fsnlx" Jul 22 00:31:34.138110 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:31:34.138031 2574 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 22 00:31:34.138110 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:31:34.138095 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d7dbfcd-8818-437c-b363-6c8abd0c52b6-node-exporter-tls podName:4d7dbfcd-8818-437c-b363-6c8abd0c52b6 nodeName:}" failed. No retries permitted until 2026-07-22 00:31:34.638080663 +0000 UTC m=+279.994838528 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/4d7dbfcd-8818-437c-b363-6c8abd0c52b6-node-exporter-tls") pod "node-exporter-xvq7l" (UID: "4d7dbfcd-8818-437c-b363-6c8abd0c52b6") : secret "node-exporter-tls" not found Jul 22 00:31:34.138346 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:34.138130 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/65e8b186-5032-46e9-b65c-5fbc08f5d68d-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-fsnlx\" (UID: \"65e8b186-5032-46e9-b65c-5fbc08f5d68d\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-fsnlx" Jul 22 00:31:34.141833 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:34.141491 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/65e8b186-5032-46e9-b65c-5fbc08f5d68d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-fsnlx\" (UID: \"65e8b186-5032-46e9-b65c-5fbc08f5d68d\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-fsnlx" Jul 22 00:31:34.141970 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:34.141887 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4d7dbfcd-8818-437c-b363-6c8abd0c52b6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xvq7l\" (UID: \"4d7dbfcd-8818-437c-b363-6c8abd0c52b6\") " pod="openshift-monitoring/node-exporter-xvq7l" Jul 22 00:31:34.147254 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:34.147207 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k64x4\" (UniqueName: \"kubernetes.io/projected/65e8b186-5032-46e9-b65c-5fbc08f5d68d-kube-api-access-k64x4\") pod \"kube-state-metrics-56c5594674-fsnlx\" (UID: \"65e8b186-5032-46e9-b65c-5fbc08f5d68d\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-fsnlx" Jul 22 00:31:34.149402 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:34.149298 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqd88\" (UniqueName: \"kubernetes.io/projected/4d7dbfcd-8818-437c-b363-6c8abd0c52b6-kube-api-access-jqd88\") pod \"node-exporter-xvq7l\" (UID: \"4d7dbfcd-8818-437c-b363-6c8abd0c52b6\") " pod="openshift-monitoring/node-exporter-xvq7l" Jul 22 00:31:34.642077 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:34.642039 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/65e8b186-5032-46e9-b65c-5fbc08f5d68d-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-fsnlx\" (UID: \"65e8b186-5032-46e9-b65c-5fbc08f5d68d\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-fsnlx" Jul 22 00:31:34.642258 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:34.642091 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4d7dbfcd-8818-437c-b363-6c8abd0c52b6-node-exporter-tls\") pod \"node-exporter-xvq7l\" (UID: \"4d7dbfcd-8818-437c-b363-6c8abd0c52b6\") " pod="openshift-monitoring/node-exporter-xvq7l" Jul 22 00:31:34.642326 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:31:34.642268 2574 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 22 00:31:34.642381 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:31:34.642351 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d7dbfcd-8818-437c-b363-6c8abd0c52b6-node-exporter-tls podName:4d7dbfcd-8818-437c-b363-6c8abd0c52b6 nodeName:}" failed. No retries permitted until 2026-07-22 00:31:35.642328217 +0000 UTC m=+280.999086071 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/4d7dbfcd-8818-437c-b363-6c8abd0c52b6-node-exporter-tls") pod "node-exporter-xvq7l" (UID: "4d7dbfcd-8818-437c-b363-6c8abd0c52b6") : secret "node-exporter-tls" not found Jul 22 00:31:34.644651 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:34.644622 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/65e8b186-5032-46e9-b65c-5fbc08f5d68d-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-fsnlx\" (UID: \"65e8b186-5032-46e9-b65c-5fbc08f5d68d\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-fsnlx" Jul 22 00:31:34.821654 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:34.821594 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-fsnlx" Jul 22 00:31:35.651379 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:35.651340 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4d7dbfcd-8818-437c-b363-6c8abd0c52b6-node-exporter-tls\") pod \"node-exporter-xvq7l\" (UID: \"4d7dbfcd-8818-437c-b363-6c8abd0c52b6\") " pod="openshift-monitoring/node-exporter-xvq7l" Jul 22 00:31:35.653968 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:35.653935 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4d7dbfcd-8818-437c-b363-6c8abd0c52b6-node-exporter-tls\") pod \"node-exporter-xvq7l\" (UID: \"4d7dbfcd-8818-437c-b363-6c8abd0c52b6\") " pod="openshift-monitoring/node-exporter-xvq7l" Jul 22 00:31:35.730328 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:35.730286 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xvq7l" Jul 22 00:31:36.389306 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:31:36.388893 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d7dbfcd_8818_437c_b363_6c8abd0c52b6.slice/crio-df6c8818c4cffa2b8eea61f72c1befdf39d172a8472ea4e9beeb2d6768ce9830 WatchSource:0}: Error finding container df6c8818c4cffa2b8eea61f72c1befdf39d172a8472ea4e9beeb2d6768ce9830: Status 404 returned error can't find the container with id df6c8818c4cffa2b8eea61f72c1befdf39d172a8472ea4e9beeb2d6768ce9830 Jul 22 00:31:36.520513 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:36.520487 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-fsnlx"] Jul 22 00:31:36.524187 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:31:36.524167 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65e8b186_5032_46e9_b65c_5fbc08f5d68d.slice/crio-be8a5a6b24b52d8213f125baefce0db2c39bbeb08bc267d36c07f02705af3cd7 WatchSource:0}: Error finding container be8a5a6b24b52d8213f125baefce0db2c39bbeb08bc267d36c07f02705af3cd7: Status 404 returned error can't find the container with id be8a5a6b24b52d8213f125baefce0db2c39bbeb08bc267d36c07f02705af3cd7 Jul 22 00:31:36.572537 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:31:36.572493 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-9lcsl" podUID="646c9f6d-62e7-4a07-8383-554a3c6b8b63" Jul 22 00:31:36.974675 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:36.974631 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69f8ffd6f5-vm74q" event={"ID":"c04f1ab4-85e1-4865-a055-dd4d84315477","Type":"ContainerStarted","Data":"e0fb1c7571d906f16a5c6dace75c07c96fb52c278489a13d1a4894fa91d20631"} Jul 22 00:31:36.975852 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:36.975825 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-fsnlx" event={"ID":"65e8b186-5032-46e9-b65c-5fbc08f5d68d","Type":"ContainerStarted","Data":"be8a5a6b24b52d8213f125baefce0db2c39bbeb08bc267d36c07f02705af3cd7"} Jul 22 00:31:36.977534 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:36.977506 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-844qz" event={"ID":"cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f","Type":"ContainerStarted","Data":"625c2ec287544519472ad2a2b909a883716986800948f9f3a3bd07b9ee556064"} Jul 22 00:31:36.978633 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:36.978613 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9lcsl" Jul 22 00:31:36.978734 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:36.978594 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xvq7l" event={"ID":"4d7dbfcd-8818-437c-b363-6c8abd0c52b6","Type":"ContainerStarted","Data":"df6c8818c4cffa2b8eea61f72c1befdf39d172a8472ea4e9beeb2d6768ce9830"} Jul 22 00:31:36.991160 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:36.991123 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-69f8ffd6f5-vm74q" podStartSLOduration=4.360894056 podStartE2EDuration="9.991106657s" totalImagesPullingTime="5.630212601s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:31:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:31:36.990340699 +0000 UTC m=+282.347098577" watchObservedRunningTime="2026-07-22 00:31:36.991106657 +0000 UTC m=+282.347864535" Jul 22 00:31:37.985423 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:37.985326 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xvq7l" event={"ID":"4d7dbfcd-8818-437c-b363-6c8abd0c52b6","Type":"ContainerStarted","Data":"e438d2fb0554799ea63f216de8b42d4b483e2007b653a2289372698310b19c4b"} Jul 22 00:31:38.310698 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:38.305565 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-69f8ffd6f5-vm74q" Jul 22 00:31:38.310698 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:38.306005 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-69f8ffd6f5-vm74q" Jul 22 00:31:38.314481 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:38.314421 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-69f8ffd6f5-vm74q" Jul 22 00:31:38.991289 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:38.991243 2574 generic.go:356] "Generic (PLEG): container finished" podID="4d7dbfcd-8818-437c-b363-6c8abd0c52b6" containerID="e438d2fb0554799ea63f216de8b42d4b483e2007b653a2289372698310b19c4b" exitCode=0 Jul 22 00:31:38.991782 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:38.991385 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xvq7l" event={"ID":"4d7dbfcd-8818-437c-b363-6c8abd0c52b6","Type":"ContainerDied","Data":"e438d2fb0554799ea63f216de8b42d4b483e2007b653a2289372698310b19c4b"} Jul 22 00:31:38.994051 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:38.994018 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-fsnlx" event={"ID":"65e8b186-5032-46e9-b65c-5fbc08f5d68d","Type":"ContainerStarted","Data":"0c46105dc2433e6fe35b79e76a6724585502dd85220b1e27601ecb6212f661f5"} Jul 22 00:31:38.994159 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:38.994083 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-fsnlx" event={"ID":"65e8b186-5032-46e9-b65c-5fbc08f5d68d","Type":"ContainerStarted","Data":"6c89eb094484013a3a62d2343afe6df2f6ec842002b5154a5fab5b5201dd43c0"} Jul 22 00:31:38.994159 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:38.994113 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-fsnlx" event={"ID":"65e8b186-5032-46e9-b65c-5fbc08f5d68d","Type":"ContainerStarted","Data":"25f0d73b8f99af852b976df3fbb48430af4a748821bf376bb4e3a787cd17ee80"} Jul 22 00:31:38.999387 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:38.999358 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-69f8ffd6f5-vm74q" Jul 22 00:31:39.044980 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:39.044894 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56c5594674-fsnlx" podStartSLOduration=4.423798232 podStartE2EDuration="6.044874755s" totalImagesPullingTime="1.621076523s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:31:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:31:39.043786967 +0000 UTC m=+284.400544844" watchObservedRunningTime="2026-07-22 00:31:39.044874755 +0000 UTC m=+284.401632673" Jul 22 00:31:39.586462 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:39.586421 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/646c9f6d-62e7-4a07-8383-554a3c6b8b63-metrics-tls\") pod \"dns-default-9lcsl\" (UID: \"646c9f6d-62e7-4a07-8383-554a3c6b8b63\") " pod="openshift-dns/dns-default-9lcsl" Jul 22 00:31:39.589025 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:39.589001 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/646c9f6d-62e7-4a07-8383-554a3c6b8b63-metrics-tls\") pod \"dns-default-9lcsl\" (UID: \"646c9f6d-62e7-4a07-8383-554a3c6b8b63\") " pod="openshift-dns/dns-default-9lcsl" Jul 22 00:31:39.681596 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:39.681569 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nfkpn\"" Jul 22 00:31:39.689834 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:39.689805 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9lcsl" Jul 22 00:31:39.847672 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:39.847579 2574 patch_prober.go:28] interesting pod/image-registry-779f6b595d-ml5n9 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 22 00:31:39.847809 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:39.847668 2574 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-779f6b595d-ml5n9" podUID="22bd50aa-4014-4172-b5f1-90cdcbe3cfb0" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 22 00:31:40.749817 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:40.749793 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9lcsl"] Jul 22 00:31:40.752110 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:31:40.752088 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod646c9f6d_62e7_4a07_8383_554a3c6b8b63.slice/crio-3f56efd300a8053855e246ceed9671f1ede92acca4d5cb5b7b501b25d91edd8f WatchSource:0}: Error finding container 3f56efd300a8053855e246ceed9671f1ede92acca4d5cb5b7b501b25d91edd8f: Status 404 returned error can't find the container with id 3f56efd300a8053855e246ceed9671f1ede92acca4d5cb5b7b501b25d91edd8f Jul 22 00:31:41.004282 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:41.004248 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-844qz" event={"ID":"cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f","Type":"ContainerStarted","Data":"c127d9467788f51b4ef7e1b9b69b5527ba330f97e6729fce9e481e32a29ac3a1"} Jul 22 00:31:41.010572 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:41.010538 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xvq7l" event={"ID":"4d7dbfcd-8818-437c-b363-6c8abd0c52b6","Type":"ContainerStarted","Data":"f7686c5cf5d038d978620646d435b70328c508db1746fa43a43f97cc62ae895e"} Jul 22 00:31:41.010765 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:41.010658 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xvq7l" event={"ID":"4d7dbfcd-8818-437c-b363-6c8abd0c52b6","Type":"ContainerStarted","Data":"43bf3ec5a4505e83cc9ad6e15815626123ddeda0b83b0c2cb481a7a1cf627d6a"} Jul 22 00:31:41.011788 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:41.011764 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9lcsl" event={"ID":"646c9f6d-62e7-4a07-8383-554a3c6b8b63","Type":"ContainerStarted","Data":"3f56efd300a8053855e246ceed9671f1ede92acca4d5cb5b7b501b25d91edd8f"} Jul 22 00:31:41.023423 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:41.023376 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-844qz" podStartSLOduration=33.695880218 podStartE2EDuration="48.023364338s" totalImagesPullingTime="14.32748412s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 00:30:53 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:31:41.022503387 +0000 UTC m=+286.379261277" watchObservedRunningTime="2026-07-22 00:31:41.023364338 +0000 UTC m=+286.380122214" Jul 22 00:31:41.038853 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:41.038808 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-xvq7l" podStartSLOduration=6.681954734 podStartE2EDuration="8.038796699s" totalImagesPullingTime="1.278986976s" totalInitContainerRuntime="77.854989ms" isStatefulPod=true podCreationTimestamp="2026-07-22 00:31:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:31:41.038272928 +0000 UTC m=+286.395030807" watchObservedRunningTime="2026-07-22 00:31:41.038796699 +0000 UTC m=+286.395554586" Jul 22 00:31:41.912359 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:41.912329 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-779f6b595d-ml5n9" Jul 22 00:31:44.026527 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:44.026492 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9lcsl" event={"ID":"646c9f6d-62e7-4a07-8383-554a3c6b8b63","Type":"ContainerStarted","Data":"3873f847fcb7e305939995036c75b8eac0ec84cfe3697a344869b71edf44cf1f"} Jul 22 00:31:44.026876 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:44.026545 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9lcsl" event={"ID":"646c9f6d-62e7-4a07-8383-554a3c6b8b63","Type":"ContainerStarted","Data":"f86a63c5c7473a1e00fad1920ad71c78171814a20562b277577d83948f2f4b6a"} Jul 22 00:31:44.026876 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:44.026622 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-9lcsl" Jul 22 00:31:44.042577 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:44.042526 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-9lcsl" podStartSLOduration=251.973863606 podStartE2EDuration="4m15.042511567s" totalImagesPullingTime="3.068647961s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:27:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:31:44.041461646 +0000 UTC m=+289.398219548" watchObservedRunningTime="2026-07-22 00:31:44.042511567 +0000 UTC m=+289.399269440" Jul 22 00:31:48.084310 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:31:48.084268 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-5xz72" podUID="f36a57f8-6e88-43d3-b572-8adf3b0ddb7c" Jul 22 00:31:49.941271 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:49.941234 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-69f8ffd6f5-vm74q"] Jul 22 00:31:54.033276 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:54.033244 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-9lcsl" Jul 22 00:31:54.943985 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:54.943957 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-hb69v_bf7b370a-c67c-4c3c-b3ba-7533131eace2/console-operator/1.log" Jul 22 00:31:54.945333 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:54.945307 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-hb69v_bf7b370a-c67c-4c3c-b3ba-7533131eace2/console-operator/1.log" Jul 22 00:31:54.948715 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:54.948697 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hr5fn_947d311a-ec4b-42e7-a0fb-ee5edc0caf8d/ovn-acl-logging/0.log" Jul 22 00:31:54.949945 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:54.949924 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hr5fn_947d311a-ec4b-42e7-a0fb-ee5edc0caf8d/ovn-acl-logging/0.log" Jul 22 00:31:54.956076 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:54.956059 2574 kubelet.go:1648] "Image garbage collection succeeded" Jul 22 00:31:59.960614 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:59.960569 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9lcsl_646c9f6d-62e7-4a07-8383-554a3c6b8b63/dns/0.log" Jul 22 00:31:59.967315 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:31:59.967294 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9lcsl_646c9f6d-62e7-4a07-8383-554a3c6b8b63/kube-rbac-proxy/0.log" Jul 22 00:32:00.915534 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:32:00.915499 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zqvbx_35037e44-306a-4ec1-9937-d3f4b3b7891a/dns-node-resolver/0.log" Jul 22 00:32:02.083099 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:32:02.083060 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5xz72" Jul 22 00:32:14.962794 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:32:14.962732 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-69f8ffd6f5-vm74q" podUID="c04f1ab4-85e1-4865-a055-dd4d84315477" containerName="console" containerID="cri-o://e0fb1c7571d906f16a5c6dace75c07c96fb52c278489a13d1a4894fa91d20631" gracePeriod=15 Jul 22 00:32:15.140768 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:32:15.140733 2574 generic.go:356] "Generic (PLEG): container finished" podID="c04f1ab4-85e1-4865-a055-dd4d84315477" containerID="e0fb1c7571d906f16a5c6dace75c07c96fb52c278489a13d1a4894fa91d20631" exitCode=0 Jul 22 00:32:15.140917 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:32:15.140858 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69f8ffd6f5-vm74q" event={"ID":"c04f1ab4-85e1-4865-a055-dd4d84315477","Type":"ContainerDied","Data":"e0fb1c7571d906f16a5c6dace75c07c96fb52c278489a13d1a4894fa91d20631"} Jul 22 00:32:15.210445 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:32:15.210424 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69f8ffd6f5-vm74q" Jul 22 00:32:15.279783 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:32:15.279705 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c04f1ab4-85e1-4865-a055-dd4d84315477-trusted-ca-bundle\") pod \"c04f1ab4-85e1-4865-a055-dd4d84315477\" (UID: \"c04f1ab4-85e1-4865-a055-dd4d84315477\") " Jul 22 00:32:15.279783 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:32:15.279755 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w8h8s\" (UniqueName: \"kubernetes.io/projected/c04f1ab4-85e1-4865-a055-dd4d84315477-kube-api-access-w8h8s\") pod \"c04f1ab4-85e1-4865-a055-dd4d84315477\" (UID: \"c04f1ab4-85e1-4865-a055-dd4d84315477\") " Jul 22 00:32:15.279783 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:32:15.279780 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c04f1ab4-85e1-4865-a055-dd4d84315477-oauth-serving-cert\") pod \"c04f1ab4-85e1-4865-a055-dd4d84315477\" (UID: \"c04f1ab4-85e1-4865-a055-dd4d84315477\") " Jul 22 00:32:15.280050 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:32:15.279796 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c04f1ab4-85e1-4865-a055-dd4d84315477-console-serving-cert\") pod \"c04f1ab4-85e1-4865-a055-dd4d84315477\" (UID: \"c04f1ab4-85e1-4865-a055-dd4d84315477\") " Jul 22 00:32:15.280050 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:32:15.279904 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c04f1ab4-85e1-4865-a055-dd4d84315477-console-oauth-config\") pod \"c04f1ab4-85e1-4865-a055-dd4d84315477\" (UID: \"c04f1ab4-85e1-4865-a055-dd4d84315477\") " Jul 22 00:32:15.280050 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:32:15.279981 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c04f1ab4-85e1-4865-a055-dd4d84315477-console-config\") pod \"c04f1ab4-85e1-4865-a055-dd4d84315477\" (UID: \"c04f1ab4-85e1-4865-a055-dd4d84315477\") " Jul 22 00:32:15.280050 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:32:15.280004 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c04f1ab4-85e1-4865-a055-dd4d84315477-service-ca\") pod \"c04f1ab4-85e1-4865-a055-dd4d84315477\" (UID: \"c04f1ab4-85e1-4865-a055-dd4d84315477\") " Jul 22 00:32:15.280253 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:32:15.280134 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c04f1ab4-85e1-4865-a055-dd4d84315477-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "c04f1ab4-85e1-4865-a055-dd4d84315477" (UID: "c04f1ab4-85e1-4865-a055-dd4d84315477"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 00:32:15.280308 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:32:15.280279 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c04f1ab4-85e1-4865-a055-dd4d84315477-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c04f1ab4-85e1-4865-a055-dd4d84315477" (UID: "c04f1ab4-85e1-4865-a055-dd4d84315477"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 00:32:15.280388 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:32:15.280359 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c04f1ab4-85e1-4865-a055-dd4d84315477-console-config" (OuterVolumeSpecName: "console-config") pod "c04f1ab4-85e1-4865-a055-dd4d84315477" (UID: "c04f1ab4-85e1-4865-a055-dd4d84315477"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 00:32:15.280388 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:32:15.280379 2574 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c04f1ab4-85e1-4865-a055-dd4d84315477-trusted-ca-bundle\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 00:32:15.280497 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:32:15.280426 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c04f1ab4-85e1-4865-a055-dd4d84315477-service-ca" (OuterVolumeSpecName: "service-ca") pod "c04f1ab4-85e1-4865-a055-dd4d84315477" (UID: "c04f1ab4-85e1-4865-a055-dd4d84315477"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 00:32:15.282176 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:32:15.282146 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c04f1ab4-85e1-4865-a055-dd4d84315477-kube-api-access-w8h8s" (OuterVolumeSpecName: "kube-api-access-w8h8s") pod "c04f1ab4-85e1-4865-a055-dd4d84315477" (UID: "c04f1ab4-85e1-4865-a055-dd4d84315477"). InnerVolumeSpecName "kube-api-access-w8h8s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 00:32:15.282313 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:32:15.282296 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c04f1ab4-85e1-4865-a055-dd4d84315477-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c04f1ab4-85e1-4865-a055-dd4d84315477" (UID: "c04f1ab4-85e1-4865-a055-dd4d84315477"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 00:32:15.282363 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:32:15.282322 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c04f1ab4-85e1-4865-a055-dd4d84315477-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c04f1ab4-85e1-4865-a055-dd4d84315477" (UID: "c04f1ab4-85e1-4865-a055-dd4d84315477"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 00:32:15.381702 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:32:15.381667 2574 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c04f1ab4-85e1-4865-a055-dd4d84315477-console-config\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 00:32:15.381702 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:32:15.381695 2574 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c04f1ab4-85e1-4865-a055-dd4d84315477-service-ca\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 00:32:15.381702 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:32:15.381704 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8h8s\" (UniqueName: \"kubernetes.io/projected/c04f1ab4-85e1-4865-a055-dd4d84315477-kube-api-access-w8h8s\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 00:32:15.381924 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:32:15.381714 2574 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c04f1ab4-85e1-4865-a055-dd4d84315477-oauth-serving-cert\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 00:32:15.381924 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:32:15.381723 2574 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c04f1ab4-85e1-4865-a055-dd4d84315477-console-serving-cert\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 00:32:15.381924 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:32:15.381732 2574 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c04f1ab4-85e1-4865-a055-dd4d84315477-console-oauth-config\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 00:32:16.145471 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:32:16.145439 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69f8ffd6f5-vm74q" event={"ID":"c04f1ab4-85e1-4865-a055-dd4d84315477","Type":"ContainerDied","Data":"fb895e0a26a511d0cd7b89c6002c76981cfd977e2f3f9e47ace1a2f447fb0d6d"} Jul 22 00:32:16.145471 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:32:16.145470 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69f8ffd6f5-vm74q" Jul 22 00:32:16.146026 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:32:16.145499 2574 scope.go:117] "RemoveContainer" containerID="e0fb1c7571d906f16a5c6dace75c07c96fb52c278489a13d1a4894fa91d20631" Jul 22 00:32:16.165245 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:32:16.165219 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-69f8ffd6f5-vm74q"] Jul 22 00:32:16.166944 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:32:16.166924 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-69f8ffd6f5-vm74q"] Jul 22 00:32:17.086992 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:32:17.086960 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c04f1ab4-85e1-4865-a055-dd4d84315477" path="/var/lib/kubelet/pods/c04f1ab4-85e1-4865-a055-dd4d84315477/volumes" Jul 22 00:33:02.205631 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:33:02.205573 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-656b689599-lsmhh"] Jul 22 00:33:02.206299 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:33:02.206020 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c04f1ab4-85e1-4865-a055-dd4d84315477" containerName="console" Jul 22 00:33:02.206299 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:33:02.206042 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c04f1ab4-85e1-4865-a055-dd4d84315477" containerName="console" Jul 22 00:33:02.206299 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:33:02.206108 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c04f1ab4-85e1-4865-a055-dd4d84315477" containerName="console" Jul 22 00:33:02.210515 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:33:02.210489 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-656b689599-lsmhh" Jul 22 00:33:02.213681 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:33:02.213651 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 22 00:33:02.213850 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:33:02.213834 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 22 00:33:02.214072 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:33:02.214053 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 22 00:33:02.214319 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:33:02.214302 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 22 00:33:02.214880 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:33:02.214863 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 22 00:33:02.215043 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:33:02.215016 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 22 00:33:02.215328 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:33:02.214904 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-t9krq\"" Jul 22 00:33:02.215638 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:33:02.215621 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 22 00:33:02.220335 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:33:02.220316 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-656b689599-lsmhh"] Jul 22 00:33:02.220506 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:33:02.220486 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 22 00:33:02.260528 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:33:02.260476 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b928ed55-30da-4e65-aa73-651b5e7db072-trusted-ca-bundle\") pod \"console-656b689599-lsmhh\" (UID: \"b928ed55-30da-4e65-aa73-651b5e7db072\") " pod="openshift-console/console-656b689599-lsmhh" Jul 22 00:33:02.260746 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:33:02.260538 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b928ed55-30da-4e65-aa73-651b5e7db072-oauth-serving-cert\") pod \"console-656b689599-lsmhh\" (UID: \"b928ed55-30da-4e65-aa73-651b5e7db072\") " pod="openshift-console/console-656b689599-lsmhh" Jul 22 00:33:02.260746 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:33:02.260581 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b928ed55-30da-4e65-aa73-651b5e7db072-console-oauth-config\") pod \"console-656b689599-lsmhh\" (UID: \"b928ed55-30da-4e65-aa73-651b5e7db072\") " pod="openshift-console/console-656b689599-lsmhh" Jul 22 00:33:02.260746 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:33:02.260653 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b928ed55-30da-4e65-aa73-651b5e7db072-console-serving-cert\") pod \"console-656b689599-lsmhh\" (UID: \"b928ed55-30da-4e65-aa73-651b5e7db072\") " pod="openshift-console/console-656b689599-lsmhh" Jul 22 00:33:02.260746 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:33:02.260675 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b928ed55-30da-4e65-aa73-651b5e7db072-console-config\") pod \"console-656b689599-lsmhh\" (UID: \"b928ed55-30da-4e65-aa73-651b5e7db072\") " pod="openshift-console/console-656b689599-lsmhh" Jul 22 00:33:02.260957 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:33:02.260755 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b928ed55-30da-4e65-aa73-651b5e7db072-service-ca\") pod \"console-656b689599-lsmhh\" (UID: \"b928ed55-30da-4e65-aa73-651b5e7db072\") " pod="openshift-console/console-656b689599-lsmhh" Jul 22 00:33:02.260957 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:33:02.260799 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twbjk\" (UniqueName: \"kubernetes.io/projected/b928ed55-30da-4e65-aa73-651b5e7db072-kube-api-access-twbjk\") pod \"console-656b689599-lsmhh\" (UID: \"b928ed55-30da-4e65-aa73-651b5e7db072\") " pod="openshift-console/console-656b689599-lsmhh" Jul 22 00:33:02.361992 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:33:02.361957 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twbjk\" (UniqueName: \"kubernetes.io/projected/b928ed55-30da-4e65-aa73-651b5e7db072-kube-api-access-twbjk\") pod \"console-656b689599-lsmhh\" (UID: \"b928ed55-30da-4e65-aa73-651b5e7db072\") " pod="openshift-console/console-656b689599-lsmhh" Jul 22 00:33:02.362189 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:33:02.361998 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b928ed55-30da-4e65-aa73-651b5e7db072-trusted-ca-bundle\") pod \"console-656b689599-lsmhh\" (UID: \"b928ed55-30da-4e65-aa73-651b5e7db072\") " pod="openshift-console/console-656b689599-lsmhh" Jul 22 00:33:02.362189 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:33:02.362023 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b928ed55-30da-4e65-aa73-651b5e7db072-oauth-serving-cert\") pod \"console-656b689599-lsmhh\" (UID: \"b928ed55-30da-4e65-aa73-651b5e7db072\") " pod="openshift-console/console-656b689599-lsmhh" Jul 22 00:33:02.362189 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:33:02.362048 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b928ed55-30da-4e65-aa73-651b5e7db072-console-oauth-config\") pod \"console-656b689599-lsmhh\" (UID: \"b928ed55-30da-4e65-aa73-651b5e7db072\") " pod="openshift-console/console-656b689599-lsmhh" Jul 22 00:33:02.362189 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:33:02.362173 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b928ed55-30da-4e65-aa73-651b5e7db072-console-serving-cert\") pod \"console-656b689599-lsmhh\" (UID: \"b928ed55-30da-4e65-aa73-651b5e7db072\") " pod="openshift-console/console-656b689599-lsmhh" Jul 22 00:33:02.362382 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:33:02.362202 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b928ed55-30da-4e65-aa73-651b5e7db072-console-config\") pod \"console-656b689599-lsmhh\" (UID: \"b928ed55-30da-4e65-aa73-651b5e7db072\") " pod="openshift-console/console-656b689599-lsmhh" Jul 22 00:33:02.362382 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:33:02.362276 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b928ed55-30da-4e65-aa73-651b5e7db072-service-ca\") pod \"console-656b689599-lsmhh\" (UID: \"b928ed55-30da-4e65-aa73-651b5e7db072\") " pod="openshift-console/console-656b689599-lsmhh" Jul 22 00:33:02.362803 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:33:02.362779 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b928ed55-30da-4e65-aa73-651b5e7db072-oauth-serving-cert\") pod \"console-656b689599-lsmhh\" (UID: \"b928ed55-30da-4e65-aa73-651b5e7db072\") " pod="openshift-console/console-656b689599-lsmhh" Jul 22 00:33:02.363048 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:33:02.363026 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b928ed55-30da-4e65-aa73-651b5e7db072-trusted-ca-bundle\") pod \"console-656b689599-lsmhh\" (UID: \"b928ed55-30da-4e65-aa73-651b5e7db072\") " pod="openshift-console/console-656b689599-lsmhh" Jul 22 00:33:02.363048 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:33:02.363038 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b928ed55-30da-4e65-aa73-651b5e7db072-service-ca\") pod \"console-656b689599-lsmhh\" (UID: \"b928ed55-30da-4e65-aa73-651b5e7db072\") " pod="openshift-console/console-656b689599-lsmhh" Jul 22 00:33:02.363193 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:33:02.363078 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b928ed55-30da-4e65-aa73-651b5e7db072-console-config\") pod \"console-656b689599-lsmhh\" (UID: \"b928ed55-30da-4e65-aa73-651b5e7db072\") " pod="openshift-console/console-656b689599-lsmhh" Jul 22 00:33:02.364628 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:33:02.364573 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b928ed55-30da-4e65-aa73-651b5e7db072-console-oauth-config\") pod \"console-656b689599-lsmhh\" (UID: \"b928ed55-30da-4e65-aa73-651b5e7db072\") " pod="openshift-console/console-656b689599-lsmhh" Jul 22 00:33:02.364762 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:33:02.364744 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b928ed55-30da-4e65-aa73-651b5e7db072-console-serving-cert\") pod \"console-656b689599-lsmhh\" (UID: \"b928ed55-30da-4e65-aa73-651b5e7db072\") " pod="openshift-console/console-656b689599-lsmhh" Jul 22 00:33:02.374533 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:33:02.374510 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twbjk\" (UniqueName: \"kubernetes.io/projected/b928ed55-30da-4e65-aa73-651b5e7db072-kube-api-access-twbjk\") pod \"console-656b689599-lsmhh\" (UID: \"b928ed55-30da-4e65-aa73-651b5e7db072\") " pod="openshift-console/console-656b689599-lsmhh" Jul 22 00:33:02.524463 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:33:02.524125 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-656b689599-lsmhh" Jul 22 00:33:02.672620 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:33:02.672571 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-656b689599-lsmhh"] Jul 22 00:33:02.675184 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:33:02.675151 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb928ed55_30da_4e65_aa73_651b5e7db072.slice/crio-06b868520bdd917cdbee1124d3bd446992f800a059071fa3fd48e7b80bb71c5a WatchSource:0}: Error finding container 06b868520bdd917cdbee1124d3bd446992f800a059071fa3fd48e7b80bb71c5a: Status 404 returned error can't find the container with id 06b868520bdd917cdbee1124d3bd446992f800a059071fa3fd48e7b80bb71c5a Jul 22 00:33:03.314032 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:33:03.313992 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-656b689599-lsmhh" event={"ID":"b928ed55-30da-4e65-aa73-651b5e7db072","Type":"ContainerStarted","Data":"98b730e972b75a0a220a8deb6da4db78a4ccebcd00c688752fd61cc4c6d7f79b"} Jul 22 00:33:03.314432 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:33:03.314044 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-656b689599-lsmhh" event={"ID":"b928ed55-30da-4e65-aa73-651b5e7db072","Type":"ContainerStarted","Data":"06b868520bdd917cdbee1124d3bd446992f800a059071fa3fd48e7b80bb71c5a"} Jul 22 00:33:03.331859 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:33:03.331805 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-656b689599-lsmhh" podStartSLOduration=1.331786988 podStartE2EDuration="1.331786988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:33:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:33:03.330787827 +0000 UTC m=+368.687545705" watchObservedRunningTime="2026-07-22 00:33:03.331786988 +0000 UTC m=+368.688544865" Jul 22 00:33:09.013215 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:33:09.013161 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f36a57f8-6e88-43d3-b572-8adf3b0ddb7c-metrics-certs\") pod \"network-metrics-daemon-5xz72\" (UID: \"f36a57f8-6e88-43d3-b572-8adf3b0ddb7c\") " pod="openshift-multus/network-metrics-daemon-5xz72" Jul 22 00:33:09.015441 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:33:09.015419 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f36a57f8-6e88-43d3-b572-8adf3b0ddb7c-metrics-certs\") pod \"network-metrics-daemon-5xz72\" (UID: \"f36a57f8-6e88-43d3-b572-8adf3b0ddb7c\") " pod="openshift-multus/network-metrics-daemon-5xz72" Jul 22 00:33:09.286984 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:33:09.286904 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dh9z6\"" Jul 22 00:33:09.295120 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:33:09.295099 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5xz72" Jul 22 00:33:09.440942 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:33:09.440909 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5xz72"] Jul 22 00:33:09.444074 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:33:09.444047 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf36a57f8_6e88_43d3_b572_8adf3b0ddb7c.slice/crio-9e8b2141bcbc8831b1ebc591d67c8a283bd57161bb34e0aabe816d91387cbd5e WatchSource:0}: Error finding container 9e8b2141bcbc8831b1ebc591d67c8a283bd57161bb34e0aabe816d91387cbd5e: Status 404 returned error can't find the container with id 9e8b2141bcbc8831b1ebc591d67c8a283bd57161bb34e0aabe816d91387cbd5e Jul 22 00:33:09.445965 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:33:09.445949 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 00:33:10.341226 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:33:10.341184 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5xz72" event={"ID":"f36a57f8-6e88-43d3-b572-8adf3b0ddb7c","Type":"ContainerStarted","Data":"9e8b2141bcbc8831b1ebc591d67c8a283bd57161bb34e0aabe816d91387cbd5e"} Jul 22 00:33:11.346040 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:33:11.346009 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5xz72" event={"ID":"f36a57f8-6e88-43d3-b572-8adf3b0ddb7c","Type":"ContainerStarted","Data":"0cef79fd6cd38bf7ab119aa09c200f605b5ee2c571dfe9180e719796d8cc1dc5"} Jul 22 00:33:11.346475 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:33:11.346062 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5xz72" event={"ID":"f36a57f8-6e88-43d3-b572-8adf3b0ddb7c","Type":"ContainerStarted","Data":"429461e9609e5f15ad761d232b9dd1c149f6dbc57dc85c4c37a4fca2837752c9"} Jul 22 00:33:11.361361 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:33:11.361317 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-5xz72" podStartSLOduration=375.364111419 podStartE2EDuration="6m16.361303165s" totalImagesPullingTime="997.191746ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:26:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:33:11.360011776 +0000 UTC m=+376.716769651" watchObservedRunningTime="2026-07-22 00:33:11.361303165 +0000 UTC m=+376.718061041" Jul 22 00:33:12.525471 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:33:12.525428 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-656b689599-lsmhh" Jul 22 00:33:12.525892 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:33:12.525527 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-656b689599-lsmhh" Jul 22 00:33:12.530215 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:33:12.530195 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-656b689599-lsmhh" Jul 22 00:33:13.358276 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:33:13.358249 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-656b689599-lsmhh" Jul 22 00:34:21.839678 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:34:21.839581 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-656b689599-lsmhh"] Jul 22 00:34:46.860553 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:34:46.860502 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-656b689599-lsmhh" podUID="b928ed55-30da-4e65-aa73-651b5e7db072" containerName="console" containerID="cri-o://98b730e972b75a0a220a8deb6da4db78a4ccebcd00c688752fd61cc4c6d7f79b" gracePeriod=15 Jul 22 00:34:47.104091 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:34:47.104067 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-656b689599-lsmhh" Jul 22 00:34:47.172569 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:34:47.172536 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b928ed55-30da-4e65-aa73-651b5e7db072-oauth-serving-cert\") pod \"b928ed55-30da-4e65-aa73-651b5e7db072\" (UID: \"b928ed55-30da-4e65-aa73-651b5e7db072\") " Jul 22 00:34:47.172752 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:34:47.172593 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b928ed55-30da-4e65-aa73-651b5e7db072-service-ca\") pod \"b928ed55-30da-4e65-aa73-651b5e7db072\" (UID: \"b928ed55-30da-4e65-aa73-651b5e7db072\") " Jul 22 00:34:47.172752 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:34:47.172683 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-twbjk\" (UniqueName: \"kubernetes.io/projected/b928ed55-30da-4e65-aa73-651b5e7db072-kube-api-access-twbjk\") pod \"b928ed55-30da-4e65-aa73-651b5e7db072\" (UID: \"b928ed55-30da-4e65-aa73-651b5e7db072\") " Jul 22 00:34:47.172752 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:34:47.172709 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b928ed55-30da-4e65-aa73-651b5e7db072-trusted-ca-bundle\") pod \"b928ed55-30da-4e65-aa73-651b5e7db072\" (UID: \"b928ed55-30da-4e65-aa73-651b5e7db072\") " Jul 22 00:34:47.172752 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:34:47.172730 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b928ed55-30da-4e65-aa73-651b5e7db072-console-serving-cert\") pod \"b928ed55-30da-4e65-aa73-651b5e7db072\" (UID: \"b928ed55-30da-4e65-aa73-651b5e7db072\") " Jul 22 00:34:47.172752 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:34:47.172748 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b928ed55-30da-4e65-aa73-651b5e7db072-console-config\") pod \"b928ed55-30da-4e65-aa73-651b5e7db072\" (UID: \"b928ed55-30da-4e65-aa73-651b5e7db072\") " Jul 22 00:34:47.172999 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:34:47.172769 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b928ed55-30da-4e65-aa73-651b5e7db072-console-oauth-config\") pod \"b928ed55-30da-4e65-aa73-651b5e7db072\" (UID: \"b928ed55-30da-4e65-aa73-651b5e7db072\") " Jul 22 00:34:47.172999 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:34:47.172937 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b928ed55-30da-4e65-aa73-651b5e7db072-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b928ed55-30da-4e65-aa73-651b5e7db072" (UID: "b928ed55-30da-4e65-aa73-651b5e7db072"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 00:34:47.173105 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:34:47.173001 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b928ed55-30da-4e65-aa73-651b5e7db072-service-ca" (OuterVolumeSpecName: "service-ca") pod "b928ed55-30da-4e65-aa73-651b5e7db072" (UID: "b928ed55-30da-4e65-aa73-651b5e7db072"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 00:34:47.173211 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:34:47.173175 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b928ed55-30da-4e65-aa73-651b5e7db072-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b928ed55-30da-4e65-aa73-651b5e7db072" (UID: "b928ed55-30da-4e65-aa73-651b5e7db072"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 00:34:47.173318 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:34:47.173240 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b928ed55-30da-4e65-aa73-651b5e7db072-console-config" (OuterVolumeSpecName: "console-config") pod "b928ed55-30da-4e65-aa73-651b5e7db072" (UID: "b928ed55-30da-4e65-aa73-651b5e7db072"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 00:34:47.174892 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:34:47.174862 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b928ed55-30da-4e65-aa73-651b5e7db072-kube-api-access-twbjk" (OuterVolumeSpecName: "kube-api-access-twbjk") pod "b928ed55-30da-4e65-aa73-651b5e7db072" (UID: "b928ed55-30da-4e65-aa73-651b5e7db072"). InnerVolumeSpecName "kube-api-access-twbjk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 00:34:47.174967 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:34:47.174905 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b928ed55-30da-4e65-aa73-651b5e7db072-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b928ed55-30da-4e65-aa73-651b5e7db072" (UID: "b928ed55-30da-4e65-aa73-651b5e7db072"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 00:34:47.175035 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:34:47.175017 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b928ed55-30da-4e65-aa73-651b5e7db072-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b928ed55-30da-4e65-aa73-651b5e7db072" (UID: "b928ed55-30da-4e65-aa73-651b5e7db072"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 00:34:47.273707 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:34:47.273672 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twbjk\" (UniqueName: \"kubernetes.io/projected/b928ed55-30da-4e65-aa73-651b5e7db072-kube-api-access-twbjk\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 00:34:47.273707 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:34:47.273698 2574 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b928ed55-30da-4e65-aa73-651b5e7db072-trusted-ca-bundle\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 00:34:47.273707 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:34:47.273707 2574 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b928ed55-30da-4e65-aa73-651b5e7db072-console-serving-cert\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 00:34:47.273707 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:34:47.273716 2574 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b928ed55-30da-4e65-aa73-651b5e7db072-console-config\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 00:34:47.273970 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:34:47.273724 2574 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b928ed55-30da-4e65-aa73-651b5e7db072-console-oauth-config\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 00:34:47.273970 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:34:47.273733 2574 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b928ed55-30da-4e65-aa73-651b5e7db072-oauth-serving-cert\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 00:34:47.273970 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:34:47.273741 2574 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b928ed55-30da-4e65-aa73-651b5e7db072-service-ca\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 00:34:47.680025 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:34:47.679987 2574 generic.go:356] "Generic (PLEG): container finished" podID="b928ed55-30da-4e65-aa73-651b5e7db072" containerID="98b730e972b75a0a220a8deb6da4db78a4ccebcd00c688752fd61cc4c6d7f79b" exitCode=0 Jul 22 00:34:47.680170 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:34:47.680030 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-656b689599-lsmhh" Jul 22 00:34:47.680170 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:34:47.680079 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-656b689599-lsmhh" event={"ID":"b928ed55-30da-4e65-aa73-651b5e7db072","Type":"ContainerDied","Data":"98b730e972b75a0a220a8deb6da4db78a4ccebcd00c688752fd61cc4c6d7f79b"} Jul 22 00:34:47.680170 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:34:47.680123 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-656b689599-lsmhh" event={"ID":"b928ed55-30da-4e65-aa73-651b5e7db072","Type":"ContainerDied","Data":"06b868520bdd917cdbee1124d3bd446992f800a059071fa3fd48e7b80bb71c5a"} Jul 22 00:34:47.680170 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:34:47.680142 2574 scope.go:117] "RemoveContainer" containerID="98b730e972b75a0a220a8deb6da4db78a4ccebcd00c688752fd61cc4c6d7f79b" Jul 22 00:34:47.691196 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:34:47.691177 2574 scope.go:117] "RemoveContainer" containerID="98b730e972b75a0a220a8deb6da4db78a4ccebcd00c688752fd61cc4c6d7f79b" Jul 22 00:34:47.691457 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:34:47.691437 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98b730e972b75a0a220a8deb6da4db78a4ccebcd00c688752fd61cc4c6d7f79b\": container with ID starting with 98b730e972b75a0a220a8deb6da4db78a4ccebcd00c688752fd61cc4c6d7f79b not found: ID does not exist" containerID="98b730e972b75a0a220a8deb6da4db78a4ccebcd00c688752fd61cc4c6d7f79b" Jul 22 00:34:47.691523 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:34:47.691464 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98b730e972b75a0a220a8deb6da4db78a4ccebcd00c688752fd61cc4c6d7f79b"} err="failed to get container status \"98b730e972b75a0a220a8deb6da4db78a4ccebcd00c688752fd61cc4c6d7f79b\": rpc error: code = NotFound desc = could not find container \"98b730e972b75a0a220a8deb6da4db78a4ccebcd00c688752fd61cc4c6d7f79b\": container with ID starting with 98b730e972b75a0a220a8deb6da4db78a4ccebcd00c688752fd61cc4c6d7f79b not found: ID does not exist" Jul 22 00:34:47.702780 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:34:47.702755 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-656b689599-lsmhh"] Jul 22 00:34:47.707879 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:34:47.707858 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-656b689599-lsmhh"] Jul 22 00:34:49.087338 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:34:49.087303 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b928ed55-30da-4e65-aa73-651b5e7db072" path="/var/lib/kubelet/pods/b928ed55-30da-4e65-aa73-651b5e7db072/volumes" Jul 22 00:35:15.509592 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:15.509557 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540c7rxs"] Jul 22 00:35:15.510069 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:15.509816 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b928ed55-30da-4e65-aa73-651b5e7db072" containerName="console" Jul 22 00:35:15.510069 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:15.509827 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b928ed55-30da-4e65-aa73-651b5e7db072" containerName="console" Jul 22 00:35:15.510069 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:15.509881 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="b928ed55-30da-4e65-aa73-651b5e7db072" containerName="console" Jul 22 00:35:15.514056 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:15.514038 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540c7rxs" Jul 22 00:35:15.518704 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:15.518659 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 22 00:35:15.519131 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:15.519114 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-rmsjv\"" Jul 22 00:35:15.519227 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:15.519181 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 22 00:35:15.542308 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:15.542279 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540c7rxs"] Jul 22 00:35:15.580992 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:15.580959 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bf9b8cdf-9331-4159-b12b-e4f160d7445f-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540c7rxs\" (UID: \"bf9b8cdf-9331-4159-b12b-e4f160d7445f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540c7rxs" Jul 22 00:35:15.581171 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:15.581010 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bf9b8cdf-9331-4159-b12b-e4f160d7445f-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540c7rxs\" (UID: \"bf9b8cdf-9331-4159-b12b-e4f160d7445f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540c7rxs" Jul 22 00:35:15.581171 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:15.581058 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m56n\" (UniqueName: \"kubernetes.io/projected/bf9b8cdf-9331-4159-b12b-e4f160d7445f-kube-api-access-8m56n\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540c7rxs\" (UID: \"bf9b8cdf-9331-4159-b12b-e4f160d7445f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540c7rxs" Jul 22 00:35:15.682028 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:15.681995 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bf9b8cdf-9331-4159-b12b-e4f160d7445f-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540c7rxs\" (UID: \"bf9b8cdf-9331-4159-b12b-e4f160d7445f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540c7rxs" Jul 22 00:35:15.682197 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:15.682041 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m56n\" (UniqueName: \"kubernetes.io/projected/bf9b8cdf-9331-4159-b12b-e4f160d7445f-kube-api-access-8m56n\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540c7rxs\" (UID: \"bf9b8cdf-9331-4159-b12b-e4f160d7445f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540c7rxs" Jul 22 00:35:15.682197 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:15.682075 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bf9b8cdf-9331-4159-b12b-e4f160d7445f-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540c7rxs\" (UID: \"bf9b8cdf-9331-4159-b12b-e4f160d7445f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540c7rxs" Jul 22 00:35:15.682457 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:15.682441 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bf9b8cdf-9331-4159-b12b-e4f160d7445f-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540c7rxs\" (UID: \"bf9b8cdf-9331-4159-b12b-e4f160d7445f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540c7rxs" Jul 22 00:35:15.682517 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:15.682498 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bf9b8cdf-9331-4159-b12b-e4f160d7445f-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540c7rxs\" (UID: \"bf9b8cdf-9331-4159-b12b-e4f160d7445f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540c7rxs" Jul 22 00:35:15.690712 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:15.690681 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m56n\" (UniqueName: \"kubernetes.io/projected/bf9b8cdf-9331-4159-b12b-e4f160d7445f-kube-api-access-8m56n\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540c7rxs\" (UID: \"bf9b8cdf-9331-4159-b12b-e4f160d7445f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540c7rxs" Jul 22 00:35:15.822580 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:15.822498 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540c7rxs" Jul 22 00:35:15.949237 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:15.949215 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540c7rxs"] Jul 22 00:35:15.951927 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:35:15.951897 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf9b8cdf_9331_4159_b12b_e4f160d7445f.slice/crio-5aade12f2b4bed3224b54d35c28b2d14a95126783bfa94240757cdbf7640fde9 WatchSource:0}: Error finding container 5aade12f2b4bed3224b54d35c28b2d14a95126783bfa94240757cdbf7640fde9: Status 404 returned error can't find the container with id 5aade12f2b4bed3224b54d35c28b2d14a95126783bfa94240757cdbf7640fde9 Jul 22 00:35:16.781512 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:16.781477 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540c7rxs" event={"ID":"bf9b8cdf-9331-4159-b12b-e4f160d7445f","Type":"ContainerStarted","Data":"5aade12f2b4bed3224b54d35c28b2d14a95126783bfa94240757cdbf7640fde9"} Jul 22 00:35:23.807737 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:23.807700 2574 generic.go:356] "Generic (PLEG): container finished" podID="bf9b8cdf-9331-4159-b12b-e4f160d7445f" containerID="7be6e2b7aff66c6948b6c081ef0dbd91680da67b13409b84353dab22317a6dda" exitCode=0 Jul 22 00:35:23.808119 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:23.807770 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540c7rxs" event={"ID":"bf9b8cdf-9331-4159-b12b-e4f160d7445f","Type":"ContainerDied","Data":"7be6e2b7aff66c6948b6c081ef0dbd91680da67b13409b84353dab22317a6dda"} Jul 22 00:35:26.819733 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:26.819701 2574 generic.go:356] "Generic (PLEG): container finished" podID="bf9b8cdf-9331-4159-b12b-e4f160d7445f" containerID="5c596ff09da9f13c8ad62ed268a31c4b010a6f92a1fc7e2136ec1130f8fddea7" exitCode=0 Jul 22 00:35:26.820177 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:26.819789 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540c7rxs" event={"ID":"bf9b8cdf-9331-4159-b12b-e4f160d7445f","Type":"ContainerDied","Data":"5c596ff09da9f13c8ad62ed268a31c4b010a6f92a1fc7e2136ec1130f8fddea7"} Jul 22 00:35:36.859660 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:36.859621 2574 generic.go:356] "Generic (PLEG): container finished" podID="bf9b8cdf-9331-4159-b12b-e4f160d7445f" containerID="ab73844c2af05aa8b7109325bc969a498a8d9645e275d2e37f3ffaf283f91576" exitCode=0 Jul 22 00:35:36.860098 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:36.859679 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540c7rxs" event={"ID":"bf9b8cdf-9331-4159-b12b-e4f160d7445f","Type":"ContainerDied","Data":"ab73844c2af05aa8b7109325bc969a498a8d9645e275d2e37f3ffaf283f91576"} Jul 22 00:35:37.984202 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:37.984170 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540c7rxs" Jul 22 00:35:38.043233 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:38.043197 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bf9b8cdf-9331-4159-b12b-e4f160d7445f-util\") pod \"bf9b8cdf-9331-4159-b12b-e4f160d7445f\" (UID: \"bf9b8cdf-9331-4159-b12b-e4f160d7445f\") " Jul 22 00:35:38.043402 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:38.043245 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bf9b8cdf-9331-4159-b12b-e4f160d7445f-bundle\") pod \"bf9b8cdf-9331-4159-b12b-e4f160d7445f\" (UID: \"bf9b8cdf-9331-4159-b12b-e4f160d7445f\") " Jul 22 00:35:38.043402 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:38.043264 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8m56n\" (UniqueName: \"kubernetes.io/projected/bf9b8cdf-9331-4159-b12b-e4f160d7445f-kube-api-access-8m56n\") pod \"bf9b8cdf-9331-4159-b12b-e4f160d7445f\" (UID: \"bf9b8cdf-9331-4159-b12b-e4f160d7445f\") " Jul 22 00:35:38.044022 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:38.043986 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf9b8cdf-9331-4159-b12b-e4f160d7445f-bundle" (OuterVolumeSpecName: "bundle") pod "bf9b8cdf-9331-4159-b12b-e4f160d7445f" (UID: "bf9b8cdf-9331-4159-b12b-e4f160d7445f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 00:35:38.045465 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:38.045440 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf9b8cdf-9331-4159-b12b-e4f160d7445f-kube-api-access-8m56n" (OuterVolumeSpecName: "kube-api-access-8m56n") pod "bf9b8cdf-9331-4159-b12b-e4f160d7445f" (UID: "bf9b8cdf-9331-4159-b12b-e4f160d7445f"). InnerVolumeSpecName "kube-api-access-8m56n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 00:35:38.048171 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:38.048150 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf9b8cdf-9331-4159-b12b-e4f160d7445f-util" (OuterVolumeSpecName: "util") pod "bf9b8cdf-9331-4159-b12b-e4f160d7445f" (UID: "bf9b8cdf-9331-4159-b12b-e4f160d7445f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 00:35:38.144452 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:38.144425 2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bf9b8cdf-9331-4159-b12b-e4f160d7445f-util\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 00:35:38.144452 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:38.144452 2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bf9b8cdf-9331-4159-b12b-e4f160d7445f-bundle\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 00:35:38.144654 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:38.144463 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8m56n\" (UniqueName: \"kubernetes.io/projected/bf9b8cdf-9331-4159-b12b-e4f160d7445f-kube-api-access-8m56n\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 00:35:38.869287 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:38.869258 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540c7rxs" Jul 22 00:35:38.869469 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:38.869252 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540c7rxs" event={"ID":"bf9b8cdf-9331-4159-b12b-e4f160d7445f","Type":"ContainerDied","Data":"5aade12f2b4bed3224b54d35c28b2d14a95126783bfa94240757cdbf7640fde9"} Jul 22 00:35:38.869469 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:38.869390 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5aade12f2b4bed3224b54d35c28b2d14a95126783bfa94240757cdbf7640fde9" Jul 22 00:35:48.682343 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:48.682256 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fh6cd5"] Jul 22 00:35:48.682760 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:48.682502 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf9b8cdf-9331-4159-b12b-e4f160d7445f" containerName="pull" Jul 22 00:35:48.682760 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:48.682512 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf9b8cdf-9331-4159-b12b-e4f160d7445f" containerName="pull" Jul 22 00:35:48.682760 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:48.682526 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf9b8cdf-9331-4159-b12b-e4f160d7445f" containerName="extract" Jul 22 00:35:48.682760 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:48.682533 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf9b8cdf-9331-4159-b12b-e4f160d7445f" containerName="extract" Jul 22 00:35:48.682760 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:48.682541 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf9b8cdf-9331-4159-b12b-e4f160d7445f" containerName="util" Jul 22 00:35:48.682760 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:48.682546 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf9b8cdf-9331-4159-b12b-e4f160d7445f" containerName="util" Jul 22 00:35:48.682760 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:48.682610 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf9b8cdf-9331-4159-b12b-e4f160d7445f" containerName="extract" Jul 22 00:35:48.685514 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:48.685499 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fh6cd5" Jul 22 00:35:48.688158 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:48.688136 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 22 00:35:48.688279 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:48.688163 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 22 00:35:48.689306 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:48.689290 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-rmsjv\"" Jul 22 00:35:48.700259 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:48.700237 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fh6cd5"] Jul 22 00:35:48.722734 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:48.722704 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e0711d39-7c33-41c4-b108-bd7e02c3ff3d-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fh6cd5\" (UID: \"e0711d39-7c33-41c4-b108-bd7e02c3ff3d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fh6cd5" Jul 22 00:35:48.722847 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:48.722748 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gzp7\" (UniqueName: \"kubernetes.io/projected/e0711d39-7c33-41c4-b108-bd7e02c3ff3d-kube-api-access-5gzp7\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fh6cd5\" (UID: \"e0711d39-7c33-41c4-b108-bd7e02c3ff3d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fh6cd5" Jul 22 00:35:48.722896 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:48.722846 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e0711d39-7c33-41c4-b108-bd7e02c3ff3d-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fh6cd5\" (UID: \"e0711d39-7c33-41c4-b108-bd7e02c3ff3d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fh6cd5" Jul 22 00:35:48.823755 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:48.823719 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gzp7\" (UniqueName: \"kubernetes.io/projected/e0711d39-7c33-41c4-b108-bd7e02c3ff3d-kube-api-access-5gzp7\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fh6cd5\" (UID: \"e0711d39-7c33-41c4-b108-bd7e02c3ff3d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fh6cd5" Jul 22 00:35:48.823903 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:48.823781 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e0711d39-7c33-41c4-b108-bd7e02c3ff3d-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fh6cd5\" (UID: \"e0711d39-7c33-41c4-b108-bd7e02c3ff3d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fh6cd5" Jul 22 00:35:48.823903 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:48.823808 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e0711d39-7c33-41c4-b108-bd7e02c3ff3d-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fh6cd5\" (UID: \"e0711d39-7c33-41c4-b108-bd7e02c3ff3d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fh6cd5" Jul 22 00:35:48.824126 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:48.824111 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e0711d39-7c33-41c4-b108-bd7e02c3ff3d-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fh6cd5\" (UID: \"e0711d39-7c33-41c4-b108-bd7e02c3ff3d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fh6cd5" Jul 22 00:35:48.824162 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:48.824146 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e0711d39-7c33-41c4-b108-bd7e02c3ff3d-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fh6cd5\" (UID: \"e0711d39-7c33-41c4-b108-bd7e02c3ff3d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fh6cd5" Jul 22 00:35:48.835418 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:48.835385 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gzp7\" (UniqueName: \"kubernetes.io/projected/e0711d39-7c33-41c4-b108-bd7e02c3ff3d-kube-api-access-5gzp7\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fh6cd5\" (UID: \"e0711d39-7c33-41c4-b108-bd7e02c3ff3d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fh6cd5" Jul 22 00:35:48.994322 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:48.994236 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fh6cd5" Jul 22 00:35:49.123459 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:49.123433 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fh6cd5"] Jul 22 00:35:49.126037 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:35:49.126011 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0711d39_7c33_41c4_b108_bd7e02c3ff3d.slice/crio-4d13e5bed76b9007b4c255442a0e5b23e574d87eb3184a9762b7d4a1378a21ff WatchSource:0}: Error finding container 4d13e5bed76b9007b4c255442a0e5b23e574d87eb3184a9762b7d4a1378a21ff: Status 404 returned error can't find the container with id 4d13e5bed76b9007b4c255442a0e5b23e574d87eb3184a9762b7d4a1378a21ff Jul 22 00:35:49.911284 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:49.911252 2574 generic.go:356] "Generic (PLEG): container finished" podID="e0711d39-7c33-41c4-b108-bd7e02c3ff3d" containerID="b233f94ab7d71ef8f8db47d0cff82937993dc587e3f690a69b9f212630b41a82" exitCode=0 Jul 22 00:35:49.911777 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:49.911321 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fh6cd5" event={"ID":"e0711d39-7c33-41c4-b108-bd7e02c3ff3d","Type":"ContainerDied","Data":"b233f94ab7d71ef8f8db47d0cff82937993dc587e3f690a69b9f212630b41a82"} Jul 22 00:35:49.911777 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:49.911355 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fh6cd5" event={"ID":"e0711d39-7c33-41c4-b108-bd7e02c3ff3d","Type":"ContainerStarted","Data":"4d13e5bed76b9007b4c255442a0e5b23e574d87eb3184a9762b7d4a1378a21ff"} Jul 22 00:35:52.923176 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:52.923140 2574 generic.go:356] "Generic (PLEG): container finished" podID="e0711d39-7c33-41c4-b108-bd7e02c3ff3d" containerID="d0dd4d7ccc620f0c11e4ae7391a4ad8fe998f0bcc08b83274a604da7a1f1b534" exitCode=0 Jul 22 00:35:52.923664 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:52.923189 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fh6cd5" event={"ID":"e0711d39-7c33-41c4-b108-bd7e02c3ff3d","Type":"ContainerDied","Data":"d0dd4d7ccc620f0c11e4ae7391a4ad8fe998f0bcc08b83274a604da7a1f1b534"} Jul 22 00:35:53.929625 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:53.929573 2574 generic.go:356] "Generic (PLEG): container finished" podID="e0711d39-7c33-41c4-b108-bd7e02c3ff3d" containerID="74984d10f9d1997102c66beb96595e2fbed63537aab91f41bc642aee4ecd4611" exitCode=0 Jul 22 00:35:53.930019 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:53.929630 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fh6cd5" event={"ID":"e0711d39-7c33-41c4-b108-bd7e02c3ff3d","Type":"ContainerDied","Data":"74984d10f9d1997102c66beb96595e2fbed63537aab91f41bc642aee4ecd4611"} Jul 22 00:35:55.054282 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:55.054258 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fh6cd5" Jul 22 00:35:55.170740 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:55.170702 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e0711d39-7c33-41c4-b108-bd7e02c3ff3d-util\") pod \"e0711d39-7c33-41c4-b108-bd7e02c3ff3d\" (UID: \"e0711d39-7c33-41c4-b108-bd7e02c3ff3d\") " Jul 22 00:35:55.170740 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:55.170736 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e0711d39-7c33-41c4-b108-bd7e02c3ff3d-bundle\") pod \"e0711d39-7c33-41c4-b108-bd7e02c3ff3d\" (UID: \"e0711d39-7c33-41c4-b108-bd7e02c3ff3d\") " Jul 22 00:35:55.170958 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:55.170768 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5gzp7\" (UniqueName: \"kubernetes.io/projected/e0711d39-7c33-41c4-b108-bd7e02c3ff3d-kube-api-access-5gzp7\") pod \"e0711d39-7c33-41c4-b108-bd7e02c3ff3d\" (UID: \"e0711d39-7c33-41c4-b108-bd7e02c3ff3d\") " Jul 22 00:35:55.171124 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:55.171097 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0711d39-7c33-41c4-b108-bd7e02c3ff3d-bundle" (OuterVolumeSpecName: "bundle") pod "e0711d39-7c33-41c4-b108-bd7e02c3ff3d" (UID: "e0711d39-7c33-41c4-b108-bd7e02c3ff3d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 00:35:55.172849 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:55.172824 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0711d39-7c33-41c4-b108-bd7e02c3ff3d-kube-api-access-5gzp7" (OuterVolumeSpecName: "kube-api-access-5gzp7") pod "e0711d39-7c33-41c4-b108-bd7e02c3ff3d" (UID: "e0711d39-7c33-41c4-b108-bd7e02c3ff3d"). InnerVolumeSpecName "kube-api-access-5gzp7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 00:35:55.175709 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:55.175678 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0711d39-7c33-41c4-b108-bd7e02c3ff3d-util" (OuterVolumeSpecName: "util") pod "e0711d39-7c33-41c4-b108-bd7e02c3ff3d" (UID: "e0711d39-7c33-41c4-b108-bd7e02c3ff3d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 00:35:55.271392 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:55.271282 2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e0711d39-7c33-41c4-b108-bd7e02c3ff3d-bundle\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 00:35:55.271392 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:55.271325 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gzp7\" (UniqueName: \"kubernetes.io/projected/e0711d39-7c33-41c4-b108-bd7e02c3ff3d-kube-api-access-5gzp7\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 00:35:55.271392 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:55.271336 2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e0711d39-7c33-41c4-b108-bd7e02c3ff3d-util\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 00:35:55.939791 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:55.939760 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fh6cd5" Jul 22 00:35:55.939791 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:55.939768 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fh6cd5" event={"ID":"e0711d39-7c33-41c4-b108-bd7e02c3ff3d","Type":"ContainerDied","Data":"4d13e5bed76b9007b4c255442a0e5b23e574d87eb3184a9762b7d4a1378a21ff"} Jul 22 00:35:55.939984 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:55.939808 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d13e5bed76b9007b4c255442a0e5b23e574d87eb3184a9762b7d4a1378a21ff" Jul 22 00:35:59.250736 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:59.250703 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-9pbtm"] Jul 22 00:35:59.251151 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:59.250952 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0711d39-7c33-41c4-b108-bd7e02c3ff3d" containerName="pull" Jul 22 00:35:59.251151 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:59.250963 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0711d39-7c33-41c4-b108-bd7e02c3ff3d" containerName="pull" Jul 22 00:35:59.251151 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:59.250972 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0711d39-7c33-41c4-b108-bd7e02c3ff3d" containerName="extract" Jul 22 00:35:59.251151 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:59.250979 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0711d39-7c33-41c4-b108-bd7e02c3ff3d" containerName="extract" Jul 22 00:35:59.251151 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:59.250994 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0711d39-7c33-41c4-b108-bd7e02c3ff3d" containerName="util" Jul 22 00:35:59.251151 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:59.250999 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0711d39-7c33-41c4-b108-bd7e02c3ff3d" containerName="util" Jul 22 00:35:59.251151 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:59.251042 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0711d39-7c33-41c4-b108-bd7e02c3ff3d" containerName="extract" Jul 22 00:35:59.257532 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:59.257513 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-9pbtm" Jul 22 00:35:59.260259 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:59.260238 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 22 00:35:59.260352 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:59.260246 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 22 00:35:59.261139 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:59.261126 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-bhpn6\"" Jul 22 00:35:59.263412 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:59.263394 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-9pbtm"] Jul 22 00:35:59.299615 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:59.299570 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/82ff1258-28f7-4e3f-8c6c-da377877cb42-bound-sa-token\") pod \"cert-manager-55b4d954d5-9pbtm\" (UID: \"82ff1258-28f7-4e3f-8c6c-da377877cb42\") " pod="cert-manager/cert-manager-55b4d954d5-9pbtm" Jul 22 00:35:59.299744 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:59.299620 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjlfs\" (UniqueName: \"kubernetes.io/projected/82ff1258-28f7-4e3f-8c6c-da377877cb42-kube-api-access-tjlfs\") pod \"cert-manager-55b4d954d5-9pbtm\" (UID: \"82ff1258-28f7-4e3f-8c6c-da377877cb42\") " pod="cert-manager/cert-manager-55b4d954d5-9pbtm" Jul 22 00:35:59.400828 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:59.400795 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjlfs\" (UniqueName: \"kubernetes.io/projected/82ff1258-28f7-4e3f-8c6c-da377877cb42-kube-api-access-tjlfs\") pod \"cert-manager-55b4d954d5-9pbtm\" (UID: \"82ff1258-28f7-4e3f-8c6c-da377877cb42\") " pod="cert-manager/cert-manager-55b4d954d5-9pbtm" Jul 22 00:35:59.400959 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:59.400882 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/82ff1258-28f7-4e3f-8c6c-da377877cb42-bound-sa-token\") pod \"cert-manager-55b4d954d5-9pbtm\" (UID: \"82ff1258-28f7-4e3f-8c6c-da377877cb42\") " pod="cert-manager/cert-manager-55b4d954d5-9pbtm" Jul 22 00:35:59.409058 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:59.409027 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/82ff1258-28f7-4e3f-8c6c-da377877cb42-bound-sa-token\") pod \"cert-manager-55b4d954d5-9pbtm\" (UID: \"82ff1258-28f7-4e3f-8c6c-da377877cb42\") " pod="cert-manager/cert-manager-55b4d954d5-9pbtm" Jul 22 00:35:59.409185 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:59.409157 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjlfs\" (UniqueName: \"kubernetes.io/projected/82ff1258-28f7-4e3f-8c6c-da377877cb42-kube-api-access-tjlfs\") pod \"cert-manager-55b4d954d5-9pbtm\" (UID: \"82ff1258-28f7-4e3f-8c6c-da377877cb42\") " pod="cert-manager/cert-manager-55b4d954d5-9pbtm" Jul 22 00:35:59.567555 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:59.567469 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-9pbtm" Jul 22 00:35:59.694064 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:59.694039 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-9pbtm"] Jul 22 00:35:59.696373 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:35:59.696344 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod82ff1258_28f7_4e3f_8c6c_da377877cb42.slice/crio-6dd44b6f7b228472e0c996cd602e9df27578b387b565dc461dd95f0b8ae93fcb WatchSource:0}: Error finding container 6dd44b6f7b228472e0c996cd602e9df27578b387b565dc461dd95f0b8ae93fcb: Status 404 returned error can't find the container with id 6dd44b6f7b228472e0c996cd602e9df27578b387b565dc461dd95f0b8ae93fcb Jul 22 00:35:59.955364 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:35:59.955325 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-9pbtm" event={"ID":"82ff1258-28f7-4e3f-8c6c-da377877cb42","Type":"ContainerStarted","Data":"6dd44b6f7b228472e0c996cd602e9df27578b387b565dc461dd95f0b8ae93fcb"} Jul 22 00:36:02.968730 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:02.968689 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-9pbtm" event={"ID":"82ff1258-28f7-4e3f-8c6c-da377877cb42","Type":"ContainerStarted","Data":"8e95970d922c49c8db579872678dc8ed9ee0a879ef533693f6b0f4fa44a4b01a"} Jul 22 00:36:02.984551 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:02.984502 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-9pbtm" podStartSLOduration=1.15360145 podStartE2EDuration="3.984482118s" totalImagesPullingTime="2.830880668s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:35:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:36:02.983049408 +0000 UTC m=+548.339807285" watchObservedRunningTime="2026-07-22 00:36:02.984482118 +0000 UTC m=+548.341239995" Jul 22 00:36:12.282588 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:12.282548 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6jqcr"] Jul 22 00:36:12.284631 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:12.284596 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6jqcr" Jul 22 00:36:12.287839 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:12.287817 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 22 00:36:12.287981 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:12.287934 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 22 00:36:12.288038 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:12.288029 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-z2xcc\"" Jul 22 00:36:12.308882 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:12.308858 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6jqcr"] Jul 22 00:36:12.402863 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:12.402827 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p89zb\" (UniqueName: \"kubernetes.io/projected/f61ea847-05db-4dfb-86d3-3e84726bcc91-kube-api-access-p89zb\") pod \"openshift-lws-operator-fd8ccff4c-6jqcr\" (UID: \"f61ea847-05db-4dfb-86d3-3e84726bcc91\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6jqcr" Jul 22 00:36:12.403015 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:12.402870 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f61ea847-05db-4dfb-86d3-3e84726bcc91-tmp\") pod \"openshift-lws-operator-fd8ccff4c-6jqcr\" (UID: \"f61ea847-05db-4dfb-86d3-3e84726bcc91\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6jqcr" Jul 22 00:36:12.503818 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:12.503786 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p89zb\" (UniqueName: \"kubernetes.io/projected/f61ea847-05db-4dfb-86d3-3e84726bcc91-kube-api-access-p89zb\") pod \"openshift-lws-operator-fd8ccff4c-6jqcr\" (UID: \"f61ea847-05db-4dfb-86d3-3e84726bcc91\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6jqcr" Jul 22 00:36:12.503933 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:12.503825 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f61ea847-05db-4dfb-86d3-3e84726bcc91-tmp\") pod \"openshift-lws-operator-fd8ccff4c-6jqcr\" (UID: \"f61ea847-05db-4dfb-86d3-3e84726bcc91\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6jqcr" Jul 22 00:36:12.504183 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:12.504166 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f61ea847-05db-4dfb-86d3-3e84726bcc91-tmp\") pod \"openshift-lws-operator-fd8ccff4c-6jqcr\" (UID: \"f61ea847-05db-4dfb-86d3-3e84726bcc91\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6jqcr" Jul 22 00:36:12.511902 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:12.511879 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p89zb\" (UniqueName: \"kubernetes.io/projected/f61ea847-05db-4dfb-86d3-3e84726bcc91-kube-api-access-p89zb\") pod \"openshift-lws-operator-fd8ccff4c-6jqcr\" (UID: \"f61ea847-05db-4dfb-86d3-3e84726bcc91\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6jqcr" Jul 22 00:36:12.594317 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:12.594253 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6jqcr" Jul 22 00:36:12.719518 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:12.719486 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6jqcr"] Jul 22 00:36:12.723047 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:36:12.723025 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf61ea847_05db_4dfb_86d3_3e84726bcc91.slice/crio-8172bddb4e2766ed6a8238f7963c3b7e273d1b35890dcfa6ba53a10560a7e177 WatchSource:0}: Error finding container 8172bddb4e2766ed6a8238f7963c3b7e273d1b35890dcfa6ba53a10560a7e177: Status 404 returned error can't find the container with id 8172bddb4e2766ed6a8238f7963c3b7e273d1b35890dcfa6ba53a10560a7e177 Jul 22 00:36:13.007426 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:13.007392 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6jqcr" event={"ID":"f61ea847-05db-4dfb-86d3-3e84726bcc91","Type":"ContainerStarted","Data":"8172bddb4e2766ed6a8238f7963c3b7e273d1b35890dcfa6ba53a10560a7e177"} Jul 22 00:36:15.016064 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:15.016029 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6jqcr" event={"ID":"f61ea847-05db-4dfb-86d3-3e84726bcc91","Type":"ContainerStarted","Data":"b1ba4f0bb5bed53cd7fc85899686bdd275f896193d5aa3cd521e6a746db647ef"} Jul 22 00:36:15.033462 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:15.033414 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-6jqcr" podStartSLOduration=0.870231588 podStartE2EDuration="3.033399787s" totalImagesPullingTime="2.163168199s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:36:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:36:15.031685811 +0000 UTC m=+560.388443688" watchObservedRunningTime="2026-07-22 00:36:15.033399787 +0000 UTC m=+560.390157664" Jul 22 00:36:17.799093 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:17.799053 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affllfg"] Jul 22 00:36:17.801298 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:17.801281 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affllfg" Jul 22 00:36:17.803831 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:17.803808 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 22 00:36:17.803941 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:17.803822 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-rmsjv\"" Jul 22 00:36:17.803941 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:17.803878 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 22 00:36:17.809886 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:17.809861 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affllfg"] Jul 22 00:36:17.951927 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:17.951889 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/949e7b2b-b35d-40e7-8373-24e3771b80b3-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affllfg\" (UID: \"949e7b2b-b35d-40e7-8373-24e3771b80b3\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affllfg" Jul 22 00:36:17.952106 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:17.951936 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/949e7b2b-b35d-40e7-8373-24e3771b80b3-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affllfg\" (UID: \"949e7b2b-b35d-40e7-8373-24e3771b80b3\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affllfg" Jul 22 00:36:17.952106 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:17.952036 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99np6\" (UniqueName: \"kubernetes.io/projected/949e7b2b-b35d-40e7-8373-24e3771b80b3-kube-api-access-99np6\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affllfg\" (UID: \"949e7b2b-b35d-40e7-8373-24e3771b80b3\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affllfg" Jul 22 00:36:18.052810 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:18.052731 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/949e7b2b-b35d-40e7-8373-24e3771b80b3-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affllfg\" (UID: \"949e7b2b-b35d-40e7-8373-24e3771b80b3\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affllfg" Jul 22 00:36:18.052948 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:18.052804 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99np6\" (UniqueName: \"kubernetes.io/projected/949e7b2b-b35d-40e7-8373-24e3771b80b3-kube-api-access-99np6\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affllfg\" (UID: \"949e7b2b-b35d-40e7-8373-24e3771b80b3\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affllfg" Jul 22 00:36:18.052948 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:18.052844 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/949e7b2b-b35d-40e7-8373-24e3771b80b3-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affllfg\" (UID: \"949e7b2b-b35d-40e7-8373-24e3771b80b3\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affllfg" Jul 22 00:36:18.053191 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:18.053169 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/949e7b2b-b35d-40e7-8373-24e3771b80b3-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affllfg\" (UID: \"949e7b2b-b35d-40e7-8373-24e3771b80b3\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affllfg" Jul 22 00:36:18.053237 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:18.053199 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/949e7b2b-b35d-40e7-8373-24e3771b80b3-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affllfg\" (UID: \"949e7b2b-b35d-40e7-8373-24e3771b80b3\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affllfg" Jul 22 00:36:18.060115 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:18.060090 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99np6\" (UniqueName: \"kubernetes.io/projected/949e7b2b-b35d-40e7-8373-24e3771b80b3-kube-api-access-99np6\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affllfg\" (UID: \"949e7b2b-b35d-40e7-8373-24e3771b80b3\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affllfg" Jul 22 00:36:18.110779 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:18.110743 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affllfg" Jul 22 00:36:18.236296 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:18.236258 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affllfg"] Jul 22 00:36:18.239252 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:36:18.239219 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod949e7b2b_b35d_40e7_8373_24e3771b80b3.slice/crio-d70461ce6209be56eebf6fa36d6a9fcd11ab28a0d2e48de657db5258f4eabbe9 WatchSource:0}: Error finding container d70461ce6209be56eebf6fa36d6a9fcd11ab28a0d2e48de657db5258f4eabbe9: Status 404 returned error can't find the container with id d70461ce6209be56eebf6fa36d6a9fcd11ab28a0d2e48de657db5258f4eabbe9 Jul 22 00:36:19.032194 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:19.032152 2574 generic.go:356] "Generic (PLEG): container finished" podID="949e7b2b-b35d-40e7-8373-24e3771b80b3" containerID="0c91406f952aca584b65cdb201468de702c4045c87ea69c7685bffd95412ace6" exitCode=0 Jul 22 00:36:19.032642 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:19.032243 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affllfg" event={"ID":"949e7b2b-b35d-40e7-8373-24e3771b80b3","Type":"ContainerDied","Data":"0c91406f952aca584b65cdb201468de702c4045c87ea69c7685bffd95412ace6"} Jul 22 00:36:19.032642 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:19.032297 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affllfg" event={"ID":"949e7b2b-b35d-40e7-8373-24e3771b80b3","Type":"ContainerStarted","Data":"d70461ce6209be56eebf6fa36d6a9fcd11ab28a0d2e48de657db5258f4eabbe9"} Jul 22 00:36:21.046869 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:21.046839 2574 generic.go:356] "Generic (PLEG): container finished" podID="949e7b2b-b35d-40e7-8373-24e3771b80b3" containerID="ff506b651feb7c76ed35da0330b48ca641382f3c08f143ab9d8cc844472d5f8c" exitCode=0 Jul 22 00:36:21.046869 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:21.046873 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affllfg" event={"ID":"949e7b2b-b35d-40e7-8373-24e3771b80b3","Type":"ContainerDied","Data":"ff506b651feb7c76ed35da0330b48ca641382f3c08f143ab9d8cc844472d5f8c"} Jul 22 00:36:22.053089 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:22.053054 2574 generic.go:356] "Generic (PLEG): container finished" podID="949e7b2b-b35d-40e7-8373-24e3771b80b3" containerID="eea1314fdbdb6b9a637b0b94e8591ca3be3cd2a539eb0c2575b7780c2f725dca" exitCode=0 Jul 22 00:36:22.053503 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:22.053141 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affllfg" event={"ID":"949e7b2b-b35d-40e7-8373-24e3771b80b3","Type":"ContainerDied","Data":"eea1314fdbdb6b9a637b0b94e8591ca3be3cd2a539eb0c2575b7780c2f725dca"} Jul 22 00:36:23.176224 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:23.176198 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affllfg" Jul 22 00:36:23.291862 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:23.291835 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/949e7b2b-b35d-40e7-8373-24e3771b80b3-util\") pod \"949e7b2b-b35d-40e7-8373-24e3771b80b3\" (UID: \"949e7b2b-b35d-40e7-8373-24e3771b80b3\") " Jul 22 00:36:23.292026 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:23.291901 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-99np6\" (UniqueName: \"kubernetes.io/projected/949e7b2b-b35d-40e7-8373-24e3771b80b3-kube-api-access-99np6\") pod \"949e7b2b-b35d-40e7-8373-24e3771b80b3\" (UID: \"949e7b2b-b35d-40e7-8373-24e3771b80b3\") " Jul 22 00:36:23.292026 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:23.291943 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/949e7b2b-b35d-40e7-8373-24e3771b80b3-bundle\") pod \"949e7b2b-b35d-40e7-8373-24e3771b80b3\" (UID: \"949e7b2b-b35d-40e7-8373-24e3771b80b3\") " Jul 22 00:36:23.292673 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:23.292648 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/949e7b2b-b35d-40e7-8373-24e3771b80b3-bundle" (OuterVolumeSpecName: "bundle") pod "949e7b2b-b35d-40e7-8373-24e3771b80b3" (UID: "949e7b2b-b35d-40e7-8373-24e3771b80b3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 00:36:23.293832 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:23.293808 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/949e7b2b-b35d-40e7-8373-24e3771b80b3-kube-api-access-99np6" (OuterVolumeSpecName: "kube-api-access-99np6") pod "949e7b2b-b35d-40e7-8373-24e3771b80b3" (UID: "949e7b2b-b35d-40e7-8373-24e3771b80b3"). InnerVolumeSpecName "kube-api-access-99np6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 00:36:23.297199 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:23.297181 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/949e7b2b-b35d-40e7-8373-24e3771b80b3-util" (OuterVolumeSpecName: "util") pod "949e7b2b-b35d-40e7-8373-24e3771b80b3" (UID: "949e7b2b-b35d-40e7-8373-24e3771b80b3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 00:36:23.392807 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:23.392781 2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/949e7b2b-b35d-40e7-8373-24e3771b80b3-util\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 00:36:23.392807 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:23.392806 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99np6\" (UniqueName: \"kubernetes.io/projected/949e7b2b-b35d-40e7-8373-24e3771b80b3-kube-api-access-99np6\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 00:36:23.392987 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:23.392816 2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/949e7b2b-b35d-40e7-8373-24e3771b80b3-bundle\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 00:36:24.063586 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:24.063557 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affllfg" Jul 22 00:36:24.063833 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:24.063551 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affllfg" event={"ID":"949e7b2b-b35d-40e7-8373-24e3771b80b3","Type":"ContainerDied","Data":"d70461ce6209be56eebf6fa36d6a9fcd11ab28a0d2e48de657db5258f4eabbe9"} Jul 22 00:36:24.063833 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:24.063687 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d70461ce6209be56eebf6fa36d6a9fcd11ab28a0d2e48de657db5258f4eabbe9" Jul 22 00:36:35.307639 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:35.307558 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6bb84b476f-t469v"] Jul 22 00:36:35.308324 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:35.307883 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="949e7b2b-b35d-40e7-8373-24e3771b80b3" containerName="util" Jul 22 00:36:35.308324 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:35.307896 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="949e7b2b-b35d-40e7-8373-24e3771b80b3" containerName="util" Jul 22 00:36:35.308324 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:35.307904 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="949e7b2b-b35d-40e7-8373-24e3771b80b3" containerName="pull" Jul 22 00:36:35.308324 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:35.307911 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="949e7b2b-b35d-40e7-8373-24e3771b80b3" containerName="pull" Jul 22 00:36:35.308324 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:35.307920 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="949e7b2b-b35d-40e7-8373-24e3771b80b3" containerName="extract" Jul 22 00:36:35.308324 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:35.307927 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="949e7b2b-b35d-40e7-8373-24e3771b80b3" containerName="extract" Jul 22 00:36:35.308324 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:35.307992 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="949e7b2b-b35d-40e7-8373-24e3771b80b3" containerName="extract" Jul 22 00:36:35.312458 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:35.312428 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6bb84b476f-t469v" Jul 22 00:36:35.315160 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:35.315131 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 22 00:36:35.315359 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:35.315239 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 22 00:36:35.315492 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:35.315428 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 22 00:36:35.315571 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:35.315494 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 22 00:36:35.315571 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:35.315566 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-8wt49\"" Jul 22 00:36:35.329898 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:35.329867 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6bb84b476f-t469v"] Jul 22 00:36:35.381531 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:35.381493 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/62a8d5be-9c0f-4ace-a6f3-2be1c2090e86-manifests\") pod \"opendatahub-operator-controller-manager-6bb84b476f-t469v\" (UID: \"62a8d5be-9c0f-4ace-a6f3-2be1c2090e86\") " pod="opendatahub/opendatahub-operator-controller-manager-6bb84b476f-t469v" Jul 22 00:36:35.381531 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:35.381530 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/62a8d5be-9c0f-4ace-a6f3-2be1c2090e86-webhook-cert\") pod \"opendatahub-operator-controller-manager-6bb84b476f-t469v\" (UID: \"62a8d5be-9c0f-4ace-a6f3-2be1c2090e86\") " pod="opendatahub/opendatahub-operator-controller-manager-6bb84b476f-t469v" Jul 22 00:36:35.381792 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:35.381561 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/62a8d5be-9c0f-4ace-a6f3-2be1c2090e86-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6bb84b476f-t469v\" (UID: \"62a8d5be-9c0f-4ace-a6f3-2be1c2090e86\") " pod="opendatahub/opendatahub-operator-controller-manager-6bb84b476f-t469v" Jul 22 00:36:35.381792 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:35.381704 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/62a8d5be-9c0f-4ace-a6f3-2be1c2090e86-tmp\") pod \"opendatahub-operator-controller-manager-6bb84b476f-t469v\" (UID: \"62a8d5be-9c0f-4ace-a6f3-2be1c2090e86\") " pod="opendatahub/opendatahub-operator-controller-manager-6bb84b476f-t469v" Jul 22 00:36:35.381792 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:35.381744 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2cjf\" (UniqueName: \"kubernetes.io/projected/62a8d5be-9c0f-4ace-a6f3-2be1c2090e86-kube-api-access-g2cjf\") pod \"opendatahub-operator-controller-manager-6bb84b476f-t469v\" (UID: \"62a8d5be-9c0f-4ace-a6f3-2be1c2090e86\") " pod="opendatahub/opendatahub-operator-controller-manager-6bb84b476f-t469v" Jul 22 00:36:35.482285 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:35.482249 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/62a8d5be-9c0f-4ace-a6f3-2be1c2090e86-tmp\") pod \"opendatahub-operator-controller-manager-6bb84b476f-t469v\" (UID: \"62a8d5be-9c0f-4ace-a6f3-2be1c2090e86\") " pod="opendatahub/opendatahub-operator-controller-manager-6bb84b476f-t469v" Jul 22 00:36:35.482486 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:35.482292 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2cjf\" (UniqueName: \"kubernetes.io/projected/62a8d5be-9c0f-4ace-a6f3-2be1c2090e86-kube-api-access-g2cjf\") pod \"opendatahub-operator-controller-manager-6bb84b476f-t469v\" (UID: \"62a8d5be-9c0f-4ace-a6f3-2be1c2090e86\") " pod="opendatahub/opendatahub-operator-controller-manager-6bb84b476f-t469v" Jul 22 00:36:35.482486 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:35.482345 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/62a8d5be-9c0f-4ace-a6f3-2be1c2090e86-manifests\") pod \"opendatahub-operator-controller-manager-6bb84b476f-t469v\" (UID: \"62a8d5be-9c0f-4ace-a6f3-2be1c2090e86\") " pod="opendatahub/opendatahub-operator-controller-manager-6bb84b476f-t469v" Jul 22 00:36:35.482486 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:35.482366 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/62a8d5be-9c0f-4ace-a6f3-2be1c2090e86-webhook-cert\") pod \"opendatahub-operator-controller-manager-6bb84b476f-t469v\" (UID: \"62a8d5be-9c0f-4ace-a6f3-2be1c2090e86\") " pod="opendatahub/opendatahub-operator-controller-manager-6bb84b476f-t469v" Jul 22 00:36:35.482486 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:35.482404 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/62a8d5be-9c0f-4ace-a6f3-2be1c2090e86-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6bb84b476f-t469v\" (UID: \"62a8d5be-9c0f-4ace-a6f3-2be1c2090e86\") " pod="opendatahub/opendatahub-operator-controller-manager-6bb84b476f-t469v" Jul 22 00:36:35.482769 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:35.482746 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/62a8d5be-9c0f-4ace-a6f3-2be1c2090e86-manifests\") pod \"opendatahub-operator-controller-manager-6bb84b476f-t469v\" (UID: \"62a8d5be-9c0f-4ace-a6f3-2be1c2090e86\") " pod="opendatahub/opendatahub-operator-controller-manager-6bb84b476f-t469v" Jul 22 00:36:35.484481 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:35.484459 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/62a8d5be-9c0f-4ace-a6f3-2be1c2090e86-tmp\") pod \"opendatahub-operator-controller-manager-6bb84b476f-t469v\" (UID: \"62a8d5be-9c0f-4ace-a6f3-2be1c2090e86\") " pod="opendatahub/opendatahub-operator-controller-manager-6bb84b476f-t469v" Jul 22 00:36:35.484749 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:35.484731 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/62a8d5be-9c0f-4ace-a6f3-2be1c2090e86-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6bb84b476f-t469v\" (UID: \"62a8d5be-9c0f-4ace-a6f3-2be1c2090e86\") " pod="opendatahub/opendatahub-operator-controller-manager-6bb84b476f-t469v" Jul 22 00:36:35.484820 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:35.484785 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/62a8d5be-9c0f-4ace-a6f3-2be1c2090e86-webhook-cert\") pod \"opendatahub-operator-controller-manager-6bb84b476f-t469v\" (UID: \"62a8d5be-9c0f-4ace-a6f3-2be1c2090e86\") " pod="opendatahub/opendatahub-operator-controller-manager-6bb84b476f-t469v" Jul 22 00:36:35.490175 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:35.490153 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2cjf\" (UniqueName: \"kubernetes.io/projected/62a8d5be-9c0f-4ace-a6f3-2be1c2090e86-kube-api-access-g2cjf\") pod \"opendatahub-operator-controller-manager-6bb84b476f-t469v\" (UID: \"62a8d5be-9c0f-4ace-a6f3-2be1c2090e86\") " pod="opendatahub/opendatahub-operator-controller-manager-6bb84b476f-t469v" Jul 22 00:36:35.628738 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:35.628708 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6bb84b476f-t469v" Jul 22 00:36:35.762932 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:35.762904 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6bb84b476f-t469v"] Jul 22 00:36:35.764886 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:36:35.764860 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62a8d5be_9c0f_4ace_a6f3_2be1c2090e86.slice/crio-a8b26aa775aa9ba7854711e5839a7f89548caa7964e59346ba71e5b0e725cbc4 WatchSource:0}: Error finding container a8b26aa775aa9ba7854711e5839a7f89548caa7964e59346ba71e5b0e725cbc4: Status 404 returned error can't find the container with id a8b26aa775aa9ba7854711e5839a7f89548caa7964e59346ba71e5b0e725cbc4 Jul 22 00:36:36.111650 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:36.111555 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6bb84b476f-t469v" event={"ID":"62a8d5be-9c0f-4ace-a6f3-2be1c2090e86","Type":"ContainerStarted","Data":"a8b26aa775aa9ba7854711e5839a7f89548caa7964e59346ba71e5b0e725cbc4"} Jul 22 00:36:40.131030 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:40.130996 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6bb84b476f-t469v" event={"ID":"62a8d5be-9c0f-4ace-a6f3-2be1c2090e86","Type":"ContainerStarted","Data":"5a4239ed9c30ac9f8c3c5d7880d0edc5fa70886368c822bfb7cc1957b580afba"} Jul 22 00:36:41.136352 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:41.136318 2574 generic.go:356] "Generic (PLEG): container finished" podID="62a8d5be-9c0f-4ace-a6f3-2be1c2090e86" containerID="5a4239ed9c30ac9f8c3c5d7880d0edc5fa70886368c822bfb7cc1957b580afba" exitCode=0 Jul 22 00:36:41.136795 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:41.136407 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6bb84b476f-t469v" event={"ID":"62a8d5be-9c0f-4ace-a6f3-2be1c2090e86","Type":"ContainerDied","Data":"5a4239ed9c30ac9f8c3c5d7880d0edc5fa70886368c822bfb7cc1957b580afba"} Jul 22 00:36:42.142487 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:42.142454 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6bb84b476f-t469v" event={"ID":"62a8d5be-9c0f-4ace-a6f3-2be1c2090e86","Type":"ContainerStarted","Data":"3c22dacbe9683a46f49215b98991c3faf309efb8ef28cc082c5e440e40ca565a"} Jul 22 00:36:42.142939 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:42.142573 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6bb84b476f-t469v" Jul 22 00:36:42.164701 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:42.164647 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6bb84b476f-t469v" podStartSLOduration=2.019812408 podStartE2EDuration="7.16463247s" totalImagesPullingTime="4.1414473s" totalInitContainerRuntime="1.003372762s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:36:35 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:36:42.16408319 +0000 UTC m=+587.520841068" watchObservedRunningTime="2026-07-22 00:36:42.16463247 +0000 UTC m=+587.521390347" Jul 22 00:36:53.149094 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:53.149058 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6bb84b476f-t469v" Jul 22 00:36:54.973051 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:54.973024 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-hb69v_bf7b370a-c67c-4c3c-b3ba-7533131eace2/console-operator/1.log" Jul 22 00:36:54.973488 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:54.973253 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-hb69v_bf7b370a-c67c-4c3c-b3ba-7533131eace2/console-operator/1.log" Jul 22 00:36:54.981967 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:54.981937 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hr5fn_947d311a-ec4b-42e7-a0fb-ee5edc0caf8d/ovn-acl-logging/0.log" Jul 22 00:36:54.982515 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:54.982494 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hr5fn_947d311a-ec4b-42e7-a0fb-ee5edc0caf8d/ovn-acl-logging/0.log" Jul 22 00:36:55.499535 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:55.499504 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-zzw9z"] Jul 22 00:36:55.505138 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:55.505121 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-zzw9z" Jul 22 00:36:55.505782 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:55.505762 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvv9f\" (UniqueName: \"kubernetes.io/projected/fcc3095a-033a-43f1-8ed0-1f42ed66dcd7-kube-api-access-pvv9f\") pod \"odh-model-controller-7998c67dc4-zzw9z\" (UID: \"fcc3095a-033a-43f1-8ed0-1f42ed66dcd7\") " pod="opendatahub/odh-model-controller-7998c67dc4-zzw9z" Jul 22 00:36:55.505877 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:55.505801 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fcc3095a-033a-43f1-8ed0-1f42ed66dcd7-cert\") pod \"odh-model-controller-7998c67dc4-zzw9z\" (UID: \"fcc3095a-033a-43f1-8ed0-1f42ed66dcd7\") " pod="opendatahub/odh-model-controller-7998c67dc4-zzw9z" Jul 22 00:36:55.508283 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:55.508262 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-np8l6\"" Jul 22 00:36:55.508462 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:55.508447 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 22 00:36:55.513291 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:55.513269 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-zzw9z"] Jul 22 00:36:55.606712 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:55.606642 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvv9f\" (UniqueName: \"kubernetes.io/projected/fcc3095a-033a-43f1-8ed0-1f42ed66dcd7-kube-api-access-pvv9f\") pod \"odh-model-controller-7998c67dc4-zzw9z\" (UID: \"fcc3095a-033a-43f1-8ed0-1f42ed66dcd7\") " pod="opendatahub/odh-model-controller-7998c67dc4-zzw9z" Jul 22 00:36:55.606905 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:55.606726 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fcc3095a-033a-43f1-8ed0-1f42ed66dcd7-cert\") pod \"odh-model-controller-7998c67dc4-zzw9z\" (UID: \"fcc3095a-033a-43f1-8ed0-1f42ed66dcd7\") " pod="opendatahub/odh-model-controller-7998c67dc4-zzw9z" Jul 22 00:36:55.606975 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:36:55.606901 2574 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 22 00:36:55.607030 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:36:55.606978 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcc3095a-033a-43f1-8ed0-1f42ed66dcd7-cert podName:fcc3095a-033a-43f1-8ed0-1f42ed66dcd7 nodeName:}" failed. No retries permitted until 2026-07-22 00:36:56.106956148 +0000 UTC m=+601.463714016 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/fcc3095a-033a-43f1-8ed0-1f42ed66dcd7-cert") pod "odh-model-controller-7998c67dc4-zzw9z" (UID: "fcc3095a-033a-43f1-8ed0-1f42ed66dcd7") : secret "odh-model-controller-webhook-cert" not found Jul 22 00:36:55.615535 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:55.615510 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvv9f\" (UniqueName: \"kubernetes.io/projected/fcc3095a-033a-43f1-8ed0-1f42ed66dcd7-kube-api-access-pvv9f\") pod \"odh-model-controller-7998c67dc4-zzw9z\" (UID: \"fcc3095a-033a-43f1-8ed0-1f42ed66dcd7\") " pod="opendatahub/odh-model-controller-7998c67dc4-zzw9z" Jul 22 00:36:56.109676 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:56.109638 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fcc3095a-033a-43f1-8ed0-1f42ed66dcd7-cert\") pod \"odh-model-controller-7998c67dc4-zzw9z\" (UID: \"fcc3095a-033a-43f1-8ed0-1f42ed66dcd7\") " pod="opendatahub/odh-model-controller-7998c67dc4-zzw9z" Jul 22 00:36:56.112036 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:56.112012 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fcc3095a-033a-43f1-8ed0-1f42ed66dcd7-cert\") pod \"odh-model-controller-7998c67dc4-zzw9z\" (UID: \"fcc3095a-033a-43f1-8ed0-1f42ed66dcd7\") " pod="opendatahub/odh-model-controller-7998c67dc4-zzw9z" Jul 22 00:36:56.116249 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:56.116226 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-zzw9z" Jul 22 00:36:56.139628 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:56.139589 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-hqj4z"] Jul 22 00:36:56.143714 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:56.143694 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-hqj4z" Jul 22 00:36:56.146268 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:56.146246 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-r6kh5\"" Jul 22 00:36:56.146407 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:56.146392 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 22 00:36:56.156405 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:56.156374 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-hqj4z"] Jul 22 00:36:56.211091 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:56.210527 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swfbv\" (UniqueName: \"kubernetes.io/projected/02566740-8d72-417f-be20-b16b2dc78276-kube-api-access-swfbv\") pod \"model-serving-api-54d89bd5fc-hqj4z\" (UID: \"02566740-8d72-417f-be20-b16b2dc78276\") " pod="opendatahub/model-serving-api-54d89bd5fc-hqj4z" Jul 22 00:36:56.211091 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:56.210650 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02566740-8d72-417f-be20-b16b2dc78276-tls-certs\") pod \"model-serving-api-54d89bd5fc-hqj4z\" (UID: \"02566740-8d72-417f-be20-b16b2dc78276\") " pod="opendatahub/model-serving-api-54d89bd5fc-hqj4z" Jul 22 00:36:56.257628 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:56.257584 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-zzw9z"] Jul 22 00:36:56.260019 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:36:56.259991 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcc3095a_033a_43f1_8ed0_1f42ed66dcd7.slice/crio-687ed9c793b11daf617792abe134387a473ab75d170bac58e0754dbb4be37642 WatchSource:0}: Error finding container 687ed9c793b11daf617792abe134387a473ab75d170bac58e0754dbb4be37642: Status 404 returned error can't find the container with id 687ed9c793b11daf617792abe134387a473ab75d170bac58e0754dbb4be37642 Jul 22 00:36:56.311090 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:56.311057 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swfbv\" (UniqueName: \"kubernetes.io/projected/02566740-8d72-417f-be20-b16b2dc78276-kube-api-access-swfbv\") pod \"model-serving-api-54d89bd5fc-hqj4z\" (UID: \"02566740-8d72-417f-be20-b16b2dc78276\") " pod="opendatahub/model-serving-api-54d89bd5fc-hqj4z" Jul 22 00:36:56.311320 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:56.311157 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02566740-8d72-417f-be20-b16b2dc78276-tls-certs\") pod \"model-serving-api-54d89bd5fc-hqj4z\" (UID: \"02566740-8d72-417f-be20-b16b2dc78276\") " pod="opendatahub/model-serving-api-54d89bd5fc-hqj4z" Jul 22 00:36:56.311387 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:36:56.311315 2574 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Jul 22 00:36:56.311387 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:36:56.311385 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02566740-8d72-417f-be20-b16b2dc78276-tls-certs podName:02566740-8d72-417f-be20-b16b2dc78276 nodeName:}" failed. No retries permitted until 2026-07-22 00:36:56.811363212 +0000 UTC m=+602.168121083 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/02566740-8d72-417f-be20-b16b2dc78276-tls-certs") pod "model-serving-api-54d89bd5fc-hqj4z" (UID: "02566740-8d72-417f-be20-b16b2dc78276") : secret "model-serving-api-tls" not found Jul 22 00:36:56.322950 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:56.322906 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swfbv\" (UniqueName: \"kubernetes.io/projected/02566740-8d72-417f-be20-b16b2dc78276-kube-api-access-swfbv\") pod \"model-serving-api-54d89bd5fc-hqj4z\" (UID: \"02566740-8d72-417f-be20-b16b2dc78276\") " pod="opendatahub/model-serving-api-54d89bd5fc-hqj4z" Jul 22 00:36:56.815196 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:56.815161 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02566740-8d72-417f-be20-b16b2dc78276-tls-certs\") pod \"model-serving-api-54d89bd5fc-hqj4z\" (UID: \"02566740-8d72-417f-be20-b16b2dc78276\") " pod="opendatahub/model-serving-api-54d89bd5fc-hqj4z" Jul 22 00:36:56.817680 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:56.817652 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02566740-8d72-417f-be20-b16b2dc78276-tls-certs\") pod \"model-serving-api-54d89bd5fc-hqj4z\" (UID: \"02566740-8d72-417f-be20-b16b2dc78276\") " pod="opendatahub/model-serving-api-54d89bd5fc-hqj4z" Jul 22 00:36:57.055563 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:57.055520 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-hqj4z" Jul 22 00:36:57.215713 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:57.215650 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-zzw9z" event={"ID":"fcc3095a-033a-43f1-8ed0-1f42ed66dcd7","Type":"ContainerStarted","Data":"687ed9c793b11daf617792abe134387a473ab75d170bac58e0754dbb4be37642"} Jul 22 00:36:57.227172 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:57.227140 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-hqj4z"] Jul 22 00:36:57.231854 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:36:57.231825 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02566740_8d72_417f_be20_b16b2dc78276.slice/crio-3b4db08cb6f1a3b0783f7e27e7aada6a2d53d9ba10c4a09273a04d4b2ffe1668 WatchSource:0}: Error finding container 3b4db08cb6f1a3b0783f7e27e7aada6a2d53d9ba10c4a09273a04d4b2ffe1668: Status 404 returned error can't find the container with id 3b4db08cb6f1a3b0783f7e27e7aada6a2d53d9ba10c4a09273a04d4b2ffe1668 Jul 22 00:36:58.223627 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:36:58.223542 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-hqj4z" event={"ID":"02566740-8d72-417f-be20-b16b2dc78276","Type":"ContainerStarted","Data":"3b4db08cb6f1a3b0783f7e27e7aada6a2d53d9ba10c4a09273a04d4b2ffe1668"} Jul 22 00:37:00.235610 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:00.235570 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-hqj4z" event={"ID":"02566740-8d72-417f-be20-b16b2dc78276","Type":"ContainerStarted","Data":"282c63c756c583656315667789076b4e1436b9dbab0fac7d7aa2ce2c56fa02c2"} Jul 22 00:37:00.236111 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:00.235654 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-hqj4z" Jul 22 00:37:00.237169 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:00.237141 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-zzw9z" event={"ID":"fcc3095a-033a-43f1-8ed0-1f42ed66dcd7","Type":"ContainerStarted","Data":"5f7cf851c80b9ac853dd1862de62c81ae6d60383725dd2bf9a56a2b0026f9569"} Jul 22 00:37:00.237285 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:00.237212 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-zzw9z" Jul 22 00:37:00.252738 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:00.252698 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-hqj4z" podStartSLOduration=1.7582223030000002 podStartE2EDuration="4.252686272s" totalImagesPullingTime="2.494463969s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:36:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:37:00.251862845 +0000 UTC m=+605.608620742" watchObservedRunningTime="2026-07-22 00:37:00.252686272 +0000 UTC m=+605.609444149" Jul 22 00:37:00.268119 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:00.268074 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-zzw9z" podStartSLOduration=1.7976345550000001 podStartE2EDuration="5.268058092s" totalImagesPullingTime="3.470423537s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:36:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:37:00.266829538 +0000 UTC m=+605.623587412" watchObservedRunningTime="2026-07-22 00:37:00.268058092 +0000 UTC m=+605.624815968" Jul 22 00:37:00.694447 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:00.694416 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6467ff7b5c-lj8jm"] Jul 22 00:37:00.697731 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:00.697712 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6467ff7b5c-lj8jm" Jul 22 00:37:00.700091 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:00.700069 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-95fht\"" Jul 22 00:37:00.700198 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:00.700125 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 22 00:37:00.700293 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:00.700275 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 22 00:37:00.700399 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:00.700382 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 22 00:37:00.700461 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:00.700437 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 22 00:37:00.709037 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:00.709010 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6467ff7b5c-lj8jm"] Jul 22 00:37:00.747789 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:00.747766 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/017333c1-6450-4714-8b46-cb84b6a4b65b-tls-certs\") pod \"kube-auth-proxy-6467ff7b5c-lj8jm\" (UID: \"017333c1-6450-4714-8b46-cb84b6a4b65b\") " pod="openshift-ingress/kube-auth-proxy-6467ff7b5c-lj8jm" Jul 22 00:37:00.747916 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:00.747804 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/017333c1-6450-4714-8b46-cb84b6a4b65b-tmp\") pod \"kube-auth-proxy-6467ff7b5c-lj8jm\" (UID: \"017333c1-6450-4714-8b46-cb84b6a4b65b\") " pod="openshift-ingress/kube-auth-proxy-6467ff7b5c-lj8jm" Jul 22 00:37:00.747916 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:00.747821 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f58px\" (UniqueName: \"kubernetes.io/projected/017333c1-6450-4714-8b46-cb84b6a4b65b-kube-api-access-f58px\") pod \"kube-auth-proxy-6467ff7b5c-lj8jm\" (UID: \"017333c1-6450-4714-8b46-cb84b6a4b65b\") " pod="openshift-ingress/kube-auth-proxy-6467ff7b5c-lj8jm" Jul 22 00:37:00.848864 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:00.848837 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/017333c1-6450-4714-8b46-cb84b6a4b65b-tls-certs\") pod \"kube-auth-proxy-6467ff7b5c-lj8jm\" (UID: \"017333c1-6450-4714-8b46-cb84b6a4b65b\") " pod="openshift-ingress/kube-auth-proxy-6467ff7b5c-lj8jm" Jul 22 00:37:00.849042 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:00.848890 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/017333c1-6450-4714-8b46-cb84b6a4b65b-tmp\") pod \"kube-auth-proxy-6467ff7b5c-lj8jm\" (UID: \"017333c1-6450-4714-8b46-cb84b6a4b65b\") " pod="openshift-ingress/kube-auth-proxy-6467ff7b5c-lj8jm" Jul 22 00:37:00.849042 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:00.848919 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f58px\" (UniqueName: \"kubernetes.io/projected/017333c1-6450-4714-8b46-cb84b6a4b65b-kube-api-access-f58px\") pod \"kube-auth-proxy-6467ff7b5c-lj8jm\" (UID: \"017333c1-6450-4714-8b46-cb84b6a4b65b\") " pod="openshift-ingress/kube-auth-proxy-6467ff7b5c-lj8jm" Jul 22 00:37:00.851096 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:00.851067 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/017333c1-6450-4714-8b46-cb84b6a4b65b-tmp\") pod \"kube-auth-proxy-6467ff7b5c-lj8jm\" (UID: \"017333c1-6450-4714-8b46-cb84b6a4b65b\") " pod="openshift-ingress/kube-auth-proxy-6467ff7b5c-lj8jm" Jul 22 00:37:00.851223 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:00.851213 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/017333c1-6450-4714-8b46-cb84b6a4b65b-tls-certs\") pod \"kube-auth-proxy-6467ff7b5c-lj8jm\" (UID: \"017333c1-6450-4714-8b46-cb84b6a4b65b\") " pod="openshift-ingress/kube-auth-proxy-6467ff7b5c-lj8jm" Jul 22 00:37:00.857428 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:00.857396 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f58px\" (UniqueName: \"kubernetes.io/projected/017333c1-6450-4714-8b46-cb84b6a4b65b-kube-api-access-f58px\") pod \"kube-auth-proxy-6467ff7b5c-lj8jm\" (UID: \"017333c1-6450-4714-8b46-cb84b6a4b65b\") " pod="openshift-ingress/kube-auth-proxy-6467ff7b5c-lj8jm" Jul 22 00:37:01.007507 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:01.007422 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6467ff7b5c-lj8jm" Jul 22 00:37:01.140177 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:01.140153 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6467ff7b5c-lj8jm"] Jul 22 00:37:01.142483 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:37:01.142449 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod017333c1_6450_4714_8b46_cb84b6a4b65b.slice/crio-c174322bd02fa55bfb088b1848da7e68ee61e7acfa7e82bc5b45e15eaf43d3fc WatchSource:0}: Error finding container c174322bd02fa55bfb088b1848da7e68ee61e7acfa7e82bc5b45e15eaf43d3fc: Status 404 returned error can't find the container with id c174322bd02fa55bfb088b1848da7e68ee61e7acfa7e82bc5b45e15eaf43d3fc Jul 22 00:37:01.242025 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:01.241992 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6467ff7b5c-lj8jm" event={"ID":"017333c1-6450-4714-8b46-cb84b6a4b65b","Type":"ContainerStarted","Data":"c174322bd02fa55bfb088b1848da7e68ee61e7acfa7e82bc5b45e15eaf43d3fc"} Jul 22 00:37:01.966332 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:01.966294 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jfssq"] Jul 22 00:37:01.969613 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:01.969580 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jfssq" Jul 22 00:37:01.973654 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:01.973592 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 22 00:37:01.973895 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:01.973879 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-slfnc\"" Jul 22 00:37:01.974002 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:01.973959 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 22 00:37:01.975003 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:01.974988 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 22 00:37:01.976487 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:01.976353 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 22 00:37:02.008235 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:02.008202 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jfssq"] Jul 22 00:37:02.060910 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:02.060873 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/78bbc2cc-2bf2-44e5-a0ca-910e31f6ed20-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-jfssq\" (UID: \"78bbc2cc-2bf2-44e5-a0ca-910e31f6ed20\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jfssq" Jul 22 00:37:02.061077 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:02.060926 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78bbc2cc-2bf2-44e5-a0ca-910e31f6ed20-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-jfssq\" (UID: \"78bbc2cc-2bf2-44e5-a0ca-910e31f6ed20\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jfssq" Jul 22 00:37:02.061077 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:02.060972 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/78bbc2cc-2bf2-44e5-a0ca-910e31f6ed20-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-jfssq\" (UID: \"78bbc2cc-2bf2-44e5-a0ca-910e31f6ed20\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jfssq" Jul 22 00:37:02.061077 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:02.061019 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/78bbc2cc-2bf2-44e5-a0ca-910e31f6ed20-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-jfssq\" (UID: \"78bbc2cc-2bf2-44e5-a0ca-910e31f6ed20\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jfssq" Jul 22 00:37:02.061077 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:02.061051 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcbjz\" (UniqueName: \"kubernetes.io/projected/78bbc2cc-2bf2-44e5-a0ca-910e31f6ed20-kube-api-access-wcbjz\") pod \"istiod-openshift-gateway-94bb8fbfd-jfssq\" (UID: \"78bbc2cc-2bf2-44e5-a0ca-910e31f6ed20\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jfssq" Jul 22 00:37:02.061265 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:02.061078 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/78bbc2cc-2bf2-44e5-a0ca-910e31f6ed20-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-jfssq\" (UID: \"78bbc2cc-2bf2-44e5-a0ca-910e31f6ed20\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jfssq" Jul 22 00:37:02.061265 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:02.061133 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/78bbc2cc-2bf2-44e5-a0ca-910e31f6ed20-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-jfssq\" (UID: \"78bbc2cc-2bf2-44e5-a0ca-910e31f6ed20\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jfssq" Jul 22 00:37:02.162256 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:02.162222 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/78bbc2cc-2bf2-44e5-a0ca-910e31f6ed20-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-jfssq\" (UID: \"78bbc2cc-2bf2-44e5-a0ca-910e31f6ed20\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jfssq" Jul 22 00:37:02.162256 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:02.162262 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78bbc2cc-2bf2-44e5-a0ca-910e31f6ed20-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-jfssq\" (UID: \"78bbc2cc-2bf2-44e5-a0ca-910e31f6ed20\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jfssq" Jul 22 00:37:02.162460 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:02.162280 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/78bbc2cc-2bf2-44e5-a0ca-910e31f6ed20-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-jfssq\" (UID: \"78bbc2cc-2bf2-44e5-a0ca-910e31f6ed20\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jfssq" Jul 22 00:37:02.162460 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:02.162395 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/78bbc2cc-2bf2-44e5-a0ca-910e31f6ed20-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-jfssq\" (UID: \"78bbc2cc-2bf2-44e5-a0ca-910e31f6ed20\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jfssq" Jul 22 00:37:02.162460 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:02.162437 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcbjz\" (UniqueName: \"kubernetes.io/projected/78bbc2cc-2bf2-44e5-a0ca-910e31f6ed20-kube-api-access-wcbjz\") pod \"istiod-openshift-gateway-94bb8fbfd-jfssq\" (UID: \"78bbc2cc-2bf2-44e5-a0ca-910e31f6ed20\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jfssq" Jul 22 00:37:02.162631 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:02.162465 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/78bbc2cc-2bf2-44e5-a0ca-910e31f6ed20-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-jfssq\" (UID: \"78bbc2cc-2bf2-44e5-a0ca-910e31f6ed20\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jfssq" Jul 22 00:37:02.162631 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:02.162520 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/78bbc2cc-2bf2-44e5-a0ca-910e31f6ed20-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-jfssq\" (UID: \"78bbc2cc-2bf2-44e5-a0ca-910e31f6ed20\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jfssq" Jul 22 00:37:02.163077 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:02.163051 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/78bbc2cc-2bf2-44e5-a0ca-910e31f6ed20-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-jfssq\" (UID: \"78bbc2cc-2bf2-44e5-a0ca-910e31f6ed20\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jfssq" Jul 22 00:37:02.164913 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:02.164871 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/78bbc2cc-2bf2-44e5-a0ca-910e31f6ed20-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-jfssq\" (UID: \"78bbc2cc-2bf2-44e5-a0ca-910e31f6ed20\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jfssq" Jul 22 00:37:02.165019 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:02.165004 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/78bbc2cc-2bf2-44e5-a0ca-910e31f6ed20-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-jfssq\" (UID: \"78bbc2cc-2bf2-44e5-a0ca-910e31f6ed20\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jfssq" Jul 22 00:37:02.165091 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:02.165069 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/78bbc2cc-2bf2-44e5-a0ca-910e31f6ed20-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-jfssq\" (UID: \"78bbc2cc-2bf2-44e5-a0ca-910e31f6ed20\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jfssq" Jul 22 00:37:02.165148 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:02.165124 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/78bbc2cc-2bf2-44e5-a0ca-910e31f6ed20-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-jfssq\" (UID: \"78bbc2cc-2bf2-44e5-a0ca-910e31f6ed20\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jfssq" Jul 22 00:37:02.171085 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:02.171038 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78bbc2cc-2bf2-44e5-a0ca-910e31f6ed20-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-jfssq\" (UID: \"78bbc2cc-2bf2-44e5-a0ca-910e31f6ed20\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jfssq" Jul 22 00:37:02.171326 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:02.171306 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcbjz\" (UniqueName: \"kubernetes.io/projected/78bbc2cc-2bf2-44e5-a0ca-910e31f6ed20-kube-api-access-wcbjz\") pod \"istiod-openshift-gateway-94bb8fbfd-jfssq\" (UID: \"78bbc2cc-2bf2-44e5-a0ca-910e31f6ed20\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jfssq" Jul 22 00:37:02.284637 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:02.284530 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jfssq" Jul 22 00:37:02.507510 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:02.507478 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jfssq"] Jul 22 00:37:03.253096 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:03.253060 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jfssq" event={"ID":"78bbc2cc-2bf2-44e5-a0ca-910e31f6ed20","Type":"ContainerStarted","Data":"a6973929e61b0729ca6c0b6d4b5ab228cd1db88818a5d2db93257449b8dbf00d"} Jul 22 00:37:05.265589 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:05.265550 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6467ff7b5c-lj8jm" event={"ID":"017333c1-6450-4714-8b46-cb84b6a4b65b","Type":"ContainerStarted","Data":"7a9a8d3c553c3a93a88419f32c37a8863459a44fe4056bfb109ae15874b640f8"} Jul 22 00:37:06.307893 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:06.307849 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 00:37:06.308205 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:06.307929 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 00:37:07.285860 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:07.285821 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jfssq" event={"ID":"78bbc2cc-2bf2-44e5-a0ca-910e31f6ed20","Type":"ContainerStarted","Data":"fa138f965edfdcde52a097c8470038aca8860431943a9d864f93ed3961764c1e"} Jul 22 00:37:07.286054 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:07.286032 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jfssq" Jul 22 00:37:07.287829 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:07.287806 2574 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-jfssq container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 22 00:37:07.287930 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:07.287852 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jfssq" podUID="78bbc2cc-2bf2-44e5-a0ca-910e31f6ed20" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 22 00:37:07.319669 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:07.319622 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6467ff7b5c-lj8jm" podStartSLOduration=4.083720722 podStartE2EDuration="7.319590823s" totalImagesPullingTime="3.235870101s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:37:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:37:05.285341681 +0000 UTC m=+610.642099558" watchObservedRunningTime="2026-07-22 00:37:07.319590823 +0000 UTC m=+612.676348700" Jul 22 00:37:07.320319 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:07.320291 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jfssq" podStartSLOduration=2.535697659 podStartE2EDuration="6.320283581s" totalImagesPullingTime="3.784585922s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:37:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:37:07.318075472 +0000 UTC m=+612.674833349" watchObservedRunningTime="2026-07-22 00:37:07.320283581 +0000 UTC m=+612.677041458" Jul 22 00:37:08.290820 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:08.290794 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jfssq" Jul 22 00:37:11.244360 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:11.244323 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-zzw9z" Jul 22 00:37:11.246536 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:11.246513 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-hqj4z" Jul 22 00:37:56.360935 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:56.360901 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-pfp79"] Jul 22 00:37:56.364415 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:56.364394 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-pfp79" Jul 22 00:37:56.367094 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:56.367067 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 22 00:37:56.368283 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:56.368259 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-zm875\"" Jul 22 00:37:56.368395 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:56.368264 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 22 00:37:56.372815 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:56.372788 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-pfp79"] Jul 22 00:37:56.419398 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:56.419367 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9gsr\" (UniqueName: \"kubernetes.io/projected/3a6ecab4-4610-4924-96c0-6f4f18eb74a0-kube-api-access-w9gsr\") pod \"kuadrant-operator-catalog-pfp79\" (UID: \"3a6ecab4-4610-4924-96c0-6f4f18eb74a0\") " pod="kuadrant-system/kuadrant-operator-catalog-pfp79" Jul 22 00:37:56.520707 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:56.520676 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9gsr\" (UniqueName: \"kubernetes.io/projected/3a6ecab4-4610-4924-96c0-6f4f18eb74a0-kube-api-access-w9gsr\") pod \"kuadrant-operator-catalog-pfp79\" (UID: \"3a6ecab4-4610-4924-96c0-6f4f18eb74a0\") " pod="kuadrant-system/kuadrant-operator-catalog-pfp79" Jul 22 00:37:56.528280 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:56.528258 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9gsr\" (UniqueName: \"kubernetes.io/projected/3a6ecab4-4610-4924-96c0-6f4f18eb74a0-kube-api-access-w9gsr\") pod \"kuadrant-operator-catalog-pfp79\" (UID: \"3a6ecab4-4610-4924-96c0-6f4f18eb74a0\") " pod="kuadrant-system/kuadrant-operator-catalog-pfp79" Jul 22 00:37:56.675200 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:56.675156 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-pfp79" Jul 22 00:37:56.804050 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:56.803989 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-pfp79"] Jul 22 00:37:56.893916 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:56.893886 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-pfp79"] Jul 22 00:37:56.947488 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:56.947419 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-kgcvm"] Jul 22 00:37:56.951503 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:56.951487 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-kgcvm" Jul 22 00:37:56.959941 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:56.959920 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-kgcvm"] Jul 22 00:37:57.024054 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:57.024024 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lp4p\" (UniqueName: \"kubernetes.io/projected/e05a821e-c5ae-46ba-a304-2704b9c0bb4c-kube-api-access-2lp4p\") pod \"kuadrant-operator-catalog-kgcvm\" (UID: \"e05a821e-c5ae-46ba-a304-2704b9c0bb4c\") " pod="kuadrant-system/kuadrant-operator-catalog-kgcvm" Jul 22 00:37:57.125041 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:57.125006 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lp4p\" (UniqueName: \"kubernetes.io/projected/e05a821e-c5ae-46ba-a304-2704b9c0bb4c-kube-api-access-2lp4p\") pod \"kuadrant-operator-catalog-kgcvm\" (UID: \"e05a821e-c5ae-46ba-a304-2704b9c0bb4c\") " pod="kuadrant-system/kuadrant-operator-catalog-kgcvm" Jul 22 00:37:57.134164 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:57.134140 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lp4p\" (UniqueName: \"kubernetes.io/projected/e05a821e-c5ae-46ba-a304-2704b9c0bb4c-kube-api-access-2lp4p\") pod \"kuadrant-operator-catalog-kgcvm\" (UID: \"e05a821e-c5ae-46ba-a304-2704b9c0bb4c\") " pod="kuadrant-system/kuadrant-operator-catalog-kgcvm" Jul 22 00:37:57.261222 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:57.261146 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-kgcvm" Jul 22 00:37:57.400692 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:57.400663 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-kgcvm"] Jul 22 00:37:57.401445 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:37:57.401421 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode05a821e_c5ae_46ba_a304_2704b9c0bb4c.slice/crio-3ca68ee58946009e33e0970a7a4d92329021f59ba726a0f5426a4b7c83ce9a47 WatchSource:0}: Error finding container 3ca68ee58946009e33e0970a7a4d92329021f59ba726a0f5426a4b7c83ce9a47: Status 404 returned error can't find the container with id 3ca68ee58946009e33e0970a7a4d92329021f59ba726a0f5426a4b7c83ce9a47 Jul 22 00:37:57.504017 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:57.503982 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-pfp79" event={"ID":"3a6ecab4-4610-4924-96c0-6f4f18eb74a0","Type":"ContainerStarted","Data":"5160e055993b1ccd7c27c2718f8a4ac09127d2b83bee5a2de0caa56b90308b48"} Jul 22 00:37:57.505082 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:37:57.505057 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-kgcvm" event={"ID":"e05a821e-c5ae-46ba-a304-2704b9c0bb4c","Type":"ContainerStarted","Data":"3ca68ee58946009e33e0970a7a4d92329021f59ba726a0f5426a4b7c83ce9a47"} Jul 22 00:38:00.521077 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:00.521040 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-kgcvm" event={"ID":"e05a821e-c5ae-46ba-a304-2704b9c0bb4c","Type":"ContainerStarted","Data":"5d3382e777cd79b43499a620fa63ede2abbb4ff63f6025e21ad48a6f74ed2bba"} Jul 22 00:38:00.522578 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:00.522550 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-pfp79" event={"ID":"3a6ecab4-4610-4924-96c0-6f4f18eb74a0","Type":"ContainerStarted","Data":"51b16aba6e6264eebb652118084ec92e9387225c7ef4c22c0ffd095bdc8f3197"} Jul 22 00:38:00.522713 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:00.522630 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-pfp79" podUID="3a6ecab4-4610-4924-96c0-6f4f18eb74a0" containerName="registry-server" containerID="cri-o://51b16aba6e6264eebb652118084ec92e9387225c7ef4c22c0ffd095bdc8f3197" gracePeriod=2 Jul 22 00:38:00.537022 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:00.536981 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-kgcvm" podStartSLOduration=2.40189049 podStartE2EDuration="4.536967347s" totalImagesPullingTime="2.135076857s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:37:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:38:00.536436333 +0000 UTC m=+665.893194231" watchObservedRunningTime="2026-07-22 00:38:00.536967347 +0000 UTC m=+665.893725268" Jul 22 00:38:00.551374 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:00.551335 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-pfp79" podStartSLOduration=1.81754282 podStartE2EDuration="4.551319855s" totalImagesPullingTime="2.733777035s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:37:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:38:00.550234819 +0000 UTC m=+665.906992717" watchObservedRunningTime="2026-07-22 00:38:00.551319855 +0000 UTC m=+665.908077734" Jul 22 00:38:00.762816 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:00.762793 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-pfp79" Jul 22 00:38:00.855097 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:00.855008 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w9gsr\" (UniqueName: \"kubernetes.io/projected/3a6ecab4-4610-4924-96c0-6f4f18eb74a0-kube-api-access-w9gsr\") pod \"3a6ecab4-4610-4924-96c0-6f4f18eb74a0\" (UID: \"3a6ecab4-4610-4924-96c0-6f4f18eb74a0\") " Jul 22 00:38:00.857131 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:00.857108 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a6ecab4-4610-4924-96c0-6f4f18eb74a0-kube-api-access-w9gsr" (OuterVolumeSpecName: "kube-api-access-w9gsr") pod "3a6ecab4-4610-4924-96c0-6f4f18eb74a0" (UID: "3a6ecab4-4610-4924-96c0-6f4f18eb74a0"). InnerVolumeSpecName "kube-api-access-w9gsr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 00:38:00.955748 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:00.955714 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9gsr\" (UniqueName: \"kubernetes.io/projected/3a6ecab4-4610-4924-96c0-6f4f18eb74a0-kube-api-access-w9gsr\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 00:38:01.527683 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:01.527651 2574 generic.go:356] "Generic (PLEG): container finished" podID="3a6ecab4-4610-4924-96c0-6f4f18eb74a0" containerID="51b16aba6e6264eebb652118084ec92e9387225c7ef4c22c0ffd095bdc8f3197" exitCode=0 Jul 22 00:38:01.528156 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:01.527693 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-pfp79" Jul 22 00:38:01.528156 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:01.527741 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-pfp79" event={"ID":"3a6ecab4-4610-4924-96c0-6f4f18eb74a0","Type":"ContainerDied","Data":"51b16aba6e6264eebb652118084ec92e9387225c7ef4c22c0ffd095bdc8f3197"} Jul 22 00:38:01.528156 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:01.527799 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-pfp79" event={"ID":"3a6ecab4-4610-4924-96c0-6f4f18eb74a0","Type":"ContainerDied","Data":"5160e055993b1ccd7c27c2718f8a4ac09127d2b83bee5a2de0caa56b90308b48"} Jul 22 00:38:01.528156 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:01.527831 2574 scope.go:117] "RemoveContainer" containerID="51b16aba6e6264eebb652118084ec92e9387225c7ef4c22c0ffd095bdc8f3197" Jul 22 00:38:01.537276 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:01.537258 2574 scope.go:117] "RemoveContainer" containerID="51b16aba6e6264eebb652118084ec92e9387225c7ef4c22c0ffd095bdc8f3197" Jul 22 00:38:01.537550 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:38:01.537529 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51b16aba6e6264eebb652118084ec92e9387225c7ef4c22c0ffd095bdc8f3197\": container with ID starting with 51b16aba6e6264eebb652118084ec92e9387225c7ef4c22c0ffd095bdc8f3197 not found: ID does not exist" containerID="51b16aba6e6264eebb652118084ec92e9387225c7ef4c22c0ffd095bdc8f3197" Jul 22 00:38:01.537619 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:01.537563 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51b16aba6e6264eebb652118084ec92e9387225c7ef4c22c0ffd095bdc8f3197"} err="failed to get container status \"51b16aba6e6264eebb652118084ec92e9387225c7ef4c22c0ffd095bdc8f3197\": rpc error: code = NotFound desc = could not find container \"51b16aba6e6264eebb652118084ec92e9387225c7ef4c22c0ffd095bdc8f3197\": container with ID starting with 51b16aba6e6264eebb652118084ec92e9387225c7ef4c22c0ffd095bdc8f3197 not found: ID does not exist" Jul 22 00:38:01.543332 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:01.543309 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-pfp79"] Jul 22 00:38:01.546899 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:01.546876 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-pfp79"] Jul 22 00:38:03.088505 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:03.088473 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a6ecab4-4610-4924-96c0-6f4f18eb74a0" path="/var/lib/kubelet/pods/3a6ecab4-4610-4924-96c0-6f4f18eb74a0/volumes" Jul 22 00:38:07.261935 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:07.261897 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-kgcvm" Jul 22 00:38:07.262336 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:07.261954 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-kgcvm" Jul 22 00:38:07.284185 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:07.284155 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-kgcvm" Jul 22 00:38:07.576490 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:07.576419 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-kgcvm" Jul 22 00:38:09.724254 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:09.724220 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-59878fc9f5-pwpl6"] Jul 22 00:38:09.724843 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:09.724778 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a6ecab4-4610-4924-96c0-6f4f18eb74a0" containerName="registry-server" Jul 22 00:38:09.725039 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:09.725022 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a6ecab4-4610-4924-96c0-6f4f18eb74a0" containerName="registry-server" Jul 22 00:38:09.725244 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:09.725223 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a6ecab4-4610-4924-96c0-6f4f18eb74a0" containerName="registry-server" Jul 22 00:38:09.729806 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:09.729788 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59878fc9f5-pwpl6" Jul 22 00:38:09.732563 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:09.732542 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-t9krq\"" Jul 22 00:38:09.733842 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:09.733821 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 22 00:38:09.733937 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:09.733821 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 22 00:38:09.733937 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:09.733894 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 22 00:38:09.733937 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:09.733823 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 22 00:38:09.734101 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:09.734021 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 22 00:38:09.734101 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:09.734064 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 22 00:38:09.735272 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:09.735250 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 22 00:38:09.739252 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:09.739231 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 22 00:38:09.740001 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:09.739985 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59878fc9f5-pwpl6"] Jul 22 00:38:09.832214 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:09.832183 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3b3d22fe-fe28-4de2-93f3-fab1548afd1b-console-serving-cert\") pod \"console-59878fc9f5-pwpl6\" (UID: \"3b3d22fe-fe28-4de2-93f3-fab1548afd1b\") " pod="openshift-console/console-59878fc9f5-pwpl6" Jul 22 00:38:09.832214 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:09.832217 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3b3d22fe-fe28-4de2-93f3-fab1548afd1b-service-ca\") pod \"console-59878fc9f5-pwpl6\" (UID: \"3b3d22fe-fe28-4de2-93f3-fab1548afd1b\") " pod="openshift-console/console-59878fc9f5-pwpl6" Jul 22 00:38:09.832495 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:09.832233 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b3d22fe-fe28-4de2-93f3-fab1548afd1b-trusted-ca-bundle\") pod \"console-59878fc9f5-pwpl6\" (UID: \"3b3d22fe-fe28-4de2-93f3-fab1548afd1b\") " pod="openshift-console/console-59878fc9f5-pwpl6" Jul 22 00:38:09.832495 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:09.832332 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3b3d22fe-fe28-4de2-93f3-fab1548afd1b-console-config\") pod \"console-59878fc9f5-pwpl6\" (UID: \"3b3d22fe-fe28-4de2-93f3-fab1548afd1b\") " pod="openshift-console/console-59878fc9f5-pwpl6" Jul 22 00:38:09.832495 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:09.832360 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3b3d22fe-fe28-4de2-93f3-fab1548afd1b-console-oauth-config\") pod \"console-59878fc9f5-pwpl6\" (UID: \"3b3d22fe-fe28-4de2-93f3-fab1548afd1b\") " pod="openshift-console/console-59878fc9f5-pwpl6" Jul 22 00:38:09.832495 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:09.832383 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3b3d22fe-fe28-4de2-93f3-fab1548afd1b-oauth-serving-cert\") pod \"console-59878fc9f5-pwpl6\" (UID: \"3b3d22fe-fe28-4de2-93f3-fab1548afd1b\") " pod="openshift-console/console-59878fc9f5-pwpl6" Jul 22 00:38:09.832495 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:09.832472 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bb6rv\" (UniqueName: \"kubernetes.io/projected/3b3d22fe-fe28-4de2-93f3-fab1548afd1b-kube-api-access-bb6rv\") pod \"console-59878fc9f5-pwpl6\" (UID: \"3b3d22fe-fe28-4de2-93f3-fab1548afd1b\") " pod="openshift-console/console-59878fc9f5-pwpl6" Jul 22 00:38:09.933416 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:09.933382 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3b3d22fe-fe28-4de2-93f3-fab1548afd1b-console-config\") pod \"console-59878fc9f5-pwpl6\" (UID: \"3b3d22fe-fe28-4de2-93f3-fab1548afd1b\") " pod="openshift-console/console-59878fc9f5-pwpl6" Jul 22 00:38:09.933416 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:09.933418 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3b3d22fe-fe28-4de2-93f3-fab1548afd1b-console-oauth-config\") pod \"console-59878fc9f5-pwpl6\" (UID: \"3b3d22fe-fe28-4de2-93f3-fab1548afd1b\") " pod="openshift-console/console-59878fc9f5-pwpl6" Jul 22 00:38:09.933682 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:09.933439 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3b3d22fe-fe28-4de2-93f3-fab1548afd1b-oauth-serving-cert\") pod \"console-59878fc9f5-pwpl6\" (UID: \"3b3d22fe-fe28-4de2-93f3-fab1548afd1b\") " pod="openshift-console/console-59878fc9f5-pwpl6" Jul 22 00:38:09.933682 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:09.933493 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb6rv\" (UniqueName: \"kubernetes.io/projected/3b3d22fe-fe28-4de2-93f3-fab1548afd1b-kube-api-access-bb6rv\") pod \"console-59878fc9f5-pwpl6\" (UID: \"3b3d22fe-fe28-4de2-93f3-fab1548afd1b\") " pod="openshift-console/console-59878fc9f5-pwpl6" Jul 22 00:38:09.933682 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:09.933528 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3b3d22fe-fe28-4de2-93f3-fab1548afd1b-console-serving-cert\") pod \"console-59878fc9f5-pwpl6\" (UID: \"3b3d22fe-fe28-4de2-93f3-fab1548afd1b\") " pod="openshift-console/console-59878fc9f5-pwpl6" Jul 22 00:38:09.933682 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:09.933553 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3b3d22fe-fe28-4de2-93f3-fab1548afd1b-service-ca\") pod \"console-59878fc9f5-pwpl6\" (UID: \"3b3d22fe-fe28-4de2-93f3-fab1548afd1b\") " pod="openshift-console/console-59878fc9f5-pwpl6" Jul 22 00:38:09.933682 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:09.933626 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b3d22fe-fe28-4de2-93f3-fab1548afd1b-trusted-ca-bundle\") pod \"console-59878fc9f5-pwpl6\" (UID: \"3b3d22fe-fe28-4de2-93f3-fab1548afd1b\") " pod="openshift-console/console-59878fc9f5-pwpl6" Jul 22 00:38:09.934198 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:09.934176 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3b3d22fe-fe28-4de2-93f3-fab1548afd1b-console-config\") pod \"console-59878fc9f5-pwpl6\" (UID: \"3b3d22fe-fe28-4de2-93f3-fab1548afd1b\") " pod="openshift-console/console-59878fc9f5-pwpl6" Jul 22 00:38:09.934310 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:09.934289 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3b3d22fe-fe28-4de2-93f3-fab1548afd1b-service-ca\") pod \"console-59878fc9f5-pwpl6\" (UID: \"3b3d22fe-fe28-4de2-93f3-fab1548afd1b\") " pod="openshift-console/console-59878fc9f5-pwpl6" Jul 22 00:38:09.934376 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:09.934357 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3b3d22fe-fe28-4de2-93f3-fab1548afd1b-oauth-serving-cert\") pod \"console-59878fc9f5-pwpl6\" (UID: \"3b3d22fe-fe28-4de2-93f3-fab1548afd1b\") " pod="openshift-console/console-59878fc9f5-pwpl6" Jul 22 00:38:09.934479 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:09.934465 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b3d22fe-fe28-4de2-93f3-fab1548afd1b-trusted-ca-bundle\") pod \"console-59878fc9f5-pwpl6\" (UID: \"3b3d22fe-fe28-4de2-93f3-fab1548afd1b\") " pod="openshift-console/console-59878fc9f5-pwpl6" Jul 22 00:38:09.935867 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:09.935843 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3b3d22fe-fe28-4de2-93f3-fab1548afd1b-console-oauth-config\") pod \"console-59878fc9f5-pwpl6\" (UID: \"3b3d22fe-fe28-4de2-93f3-fab1548afd1b\") " pod="openshift-console/console-59878fc9f5-pwpl6" Jul 22 00:38:09.936002 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:09.935986 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3b3d22fe-fe28-4de2-93f3-fab1548afd1b-console-serving-cert\") pod \"console-59878fc9f5-pwpl6\" (UID: \"3b3d22fe-fe28-4de2-93f3-fab1548afd1b\") " pod="openshift-console/console-59878fc9f5-pwpl6" Jul 22 00:38:09.941103 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:09.941084 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb6rv\" (UniqueName: \"kubernetes.io/projected/3b3d22fe-fe28-4de2-93f3-fab1548afd1b-kube-api-access-bb6rv\") pod \"console-59878fc9f5-pwpl6\" (UID: \"3b3d22fe-fe28-4de2-93f3-fab1548afd1b\") " pod="openshift-console/console-59878fc9f5-pwpl6" Jul 22 00:38:10.041337 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:10.041255 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59878fc9f5-pwpl6" Jul 22 00:38:10.175257 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:10.175236 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59878fc9f5-pwpl6"] Jul 22 00:38:10.177773 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:38:10.177742 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b3d22fe_fe28_4de2_93f3_fab1548afd1b.slice/crio-8951864910a05e8b0767e77eb87a56791c1ca395c2b4a1fe55872d2c36c532cf WatchSource:0}: Error finding container 8951864910a05e8b0767e77eb87a56791c1ca395c2b4a1fe55872d2c36c532cf: Status 404 returned error can't find the container with id 8951864910a05e8b0767e77eb87a56791c1ca395c2b4a1fe55872d2c36c532cf Jul 22 00:38:10.569025 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:10.568990 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59878fc9f5-pwpl6" event={"ID":"3b3d22fe-fe28-4de2-93f3-fab1548afd1b","Type":"ContainerStarted","Data":"835ae3d60f788392c5b4308530c4c75d9b4c69058257ecda857496eba13b22e5"} Jul 22 00:38:10.569180 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:10.569041 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59878fc9f5-pwpl6" event={"ID":"3b3d22fe-fe28-4de2-93f3-fab1548afd1b","Type":"ContainerStarted","Data":"8951864910a05e8b0767e77eb87a56791c1ca395c2b4a1fe55872d2c36c532cf"} Jul 22 00:38:10.589344 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:10.589294 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-59878fc9f5-pwpl6" podStartSLOduration=1.5892800980000001 podStartE2EDuration="1.589280098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:38:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:38:10.588236499 +0000 UTC m=+675.944994396" watchObservedRunningTime="2026-07-22 00:38:10.589280098 +0000 UTC m=+675.946037975" Jul 22 00:38:12.172246 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:12.172207 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef148wqw"] Jul 22 00:38:12.175845 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:12.175823 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef148wqw" Jul 22 00:38:12.178286 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:12.178268 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-mphx6\"" Jul 22 00:38:12.183336 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:12.183022 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef148wqw"] Jul 22 00:38:12.254010 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:12.253982 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1e1e65cd-f7b5-439f-ab31-cad60691ce9a-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef148wqw\" (UID: \"1e1e65cd-f7b5-439f-ab31-cad60691ce9a\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef148wqw" Jul 22 00:38:12.254168 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:12.254014 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g8h6\" (UniqueName: \"kubernetes.io/projected/1e1e65cd-f7b5-439f-ab31-cad60691ce9a-kube-api-access-4g8h6\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef148wqw\" (UID: \"1e1e65cd-f7b5-439f-ab31-cad60691ce9a\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef148wqw" Jul 22 00:38:12.254168 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:12.254040 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1e1e65cd-f7b5-439f-ab31-cad60691ce9a-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef148wqw\" (UID: \"1e1e65cd-f7b5-439f-ab31-cad60691ce9a\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef148wqw" Jul 22 00:38:12.355336 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:12.355300 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1e1e65cd-f7b5-439f-ab31-cad60691ce9a-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef148wqw\" (UID: \"1e1e65cd-f7b5-439f-ab31-cad60691ce9a\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef148wqw" Jul 22 00:38:12.355504 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:12.355341 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g8h6\" (UniqueName: \"kubernetes.io/projected/1e1e65cd-f7b5-439f-ab31-cad60691ce9a-kube-api-access-4g8h6\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef148wqw\" (UID: \"1e1e65cd-f7b5-439f-ab31-cad60691ce9a\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef148wqw" Jul 22 00:38:12.355504 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:12.355374 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1e1e65cd-f7b5-439f-ab31-cad60691ce9a-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef148wqw\" (UID: \"1e1e65cd-f7b5-439f-ab31-cad60691ce9a\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef148wqw" Jul 22 00:38:12.355801 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:12.355780 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1e1e65cd-f7b5-439f-ab31-cad60691ce9a-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef148wqw\" (UID: \"1e1e65cd-f7b5-439f-ab31-cad60691ce9a\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef148wqw" Jul 22 00:38:12.355853 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:12.355802 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1e1e65cd-f7b5-439f-ab31-cad60691ce9a-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef148wqw\" (UID: \"1e1e65cd-f7b5-439f-ab31-cad60691ce9a\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef148wqw" Jul 22 00:38:12.364064 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:12.364043 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g8h6\" (UniqueName: \"kubernetes.io/projected/1e1e65cd-f7b5-439f-ab31-cad60691ce9a-kube-api-access-4g8h6\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef148wqw\" (UID: \"1e1e65cd-f7b5-439f-ab31-cad60691ce9a\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef148wqw" Jul 22 00:38:12.486375 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:12.486292 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef148wqw" Jul 22 00:38:12.573115 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:12.573085 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mfn4p"] Jul 22 00:38:12.578259 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:12.578236 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mfn4p" Jul 22 00:38:12.583401 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:12.583337 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mfn4p"] Jul 22 00:38:12.616789 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:12.616764 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef148wqw"] Jul 22 00:38:12.618794 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:38:12.618768 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e1e65cd_f7b5_439f_ab31_cad60691ce9a.slice/crio-adffdebd4c1e2170ba2379b6868a281bee9f4fbc0b8452beba4bb930cd6bf339 WatchSource:0}: Error finding container adffdebd4c1e2170ba2379b6868a281bee9f4fbc0b8452beba4bb930cd6bf339: Status 404 returned error can't find the container with id adffdebd4c1e2170ba2379b6868a281bee9f4fbc0b8452beba4bb930cd6bf339 Jul 22 00:38:12.658393 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:12.658368 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6ce6a5b6-b585-45b1-99ee-ee0c9826c982-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mfn4p\" (UID: \"6ce6a5b6-b585-45b1-99ee-ee0c9826c982\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mfn4p" Jul 22 00:38:12.658496 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:12.658410 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6ce6a5b6-b585-45b1-99ee-ee0c9826c982-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mfn4p\" (UID: \"6ce6a5b6-b585-45b1-99ee-ee0c9826c982\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mfn4p" Jul 22 00:38:12.658496 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:12.658478 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk4zl\" (UniqueName: \"kubernetes.io/projected/6ce6a5b6-b585-45b1-99ee-ee0c9826c982-kube-api-access-hk4zl\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mfn4p\" (UID: \"6ce6a5b6-b585-45b1-99ee-ee0c9826c982\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mfn4p" Jul 22 00:38:12.759623 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:12.759536 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6ce6a5b6-b585-45b1-99ee-ee0c9826c982-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mfn4p\" (UID: \"6ce6a5b6-b585-45b1-99ee-ee0c9826c982\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mfn4p" Jul 22 00:38:12.759623 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:12.759567 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6ce6a5b6-b585-45b1-99ee-ee0c9826c982-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mfn4p\" (UID: \"6ce6a5b6-b585-45b1-99ee-ee0c9826c982\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mfn4p" Jul 22 00:38:12.759623 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:12.759595 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk4zl\" (UniqueName: \"kubernetes.io/projected/6ce6a5b6-b585-45b1-99ee-ee0c9826c982-kube-api-access-hk4zl\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mfn4p\" (UID: \"6ce6a5b6-b585-45b1-99ee-ee0c9826c982\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mfn4p" Jul 22 00:38:12.759909 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:12.759888 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6ce6a5b6-b585-45b1-99ee-ee0c9826c982-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mfn4p\" (UID: \"6ce6a5b6-b585-45b1-99ee-ee0c9826c982\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mfn4p" Jul 22 00:38:12.759983 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:12.759960 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6ce6a5b6-b585-45b1-99ee-ee0c9826c982-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mfn4p\" (UID: \"6ce6a5b6-b585-45b1-99ee-ee0c9826c982\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mfn4p" Jul 22 00:38:12.766993 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:12.766971 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk4zl\" (UniqueName: \"kubernetes.io/projected/6ce6a5b6-b585-45b1-99ee-ee0c9826c982-kube-api-access-hk4zl\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mfn4p\" (UID: \"6ce6a5b6-b585-45b1-99ee-ee0c9826c982\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mfn4p" Jul 22 00:38:12.891351 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:12.891315 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mfn4p" Jul 22 00:38:12.971124 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:12.971092 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738frn4"] Jul 22 00:38:12.976284 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:12.976256 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738frn4" Jul 22 00:38:12.983272 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:12.983234 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738frn4"] Jul 22 00:38:13.022556 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:13.022505 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mfn4p"] Jul 22 00:38:13.024669 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:38:13.024648 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ce6a5b6_b585_45b1_99ee_ee0c9826c982.slice/crio-59bbbaf0fe9d8d5fed0ce9e606c023e1097051903657f9cc22065035940fc53d WatchSource:0}: Error finding container 59bbbaf0fe9d8d5fed0ce9e606c023e1097051903657f9cc22065035940fc53d: Status 404 returned error can't find the container with id 59bbbaf0fe9d8d5fed0ce9e606c023e1097051903657f9cc22065035940fc53d Jul 22 00:38:13.061208 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:13.061172 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55zzk\" (UniqueName: \"kubernetes.io/projected/8bb8e779-075b-4123-a24a-be5ecae1a32f-kube-api-access-55zzk\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738frn4\" (UID: \"8bb8e779-075b-4123-a24a-be5ecae1a32f\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738frn4" Jul 22 00:38:13.061322 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:13.061208 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8bb8e779-075b-4123-a24a-be5ecae1a32f-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738frn4\" (UID: \"8bb8e779-075b-4123-a24a-be5ecae1a32f\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738frn4" Jul 22 00:38:13.061491 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:13.061320 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8bb8e779-075b-4123-a24a-be5ecae1a32f-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738frn4\" (UID: \"8bb8e779-075b-4123-a24a-be5ecae1a32f\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738frn4" Jul 22 00:38:13.161904 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:13.161878 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55zzk\" (UniqueName: \"kubernetes.io/projected/8bb8e779-075b-4123-a24a-be5ecae1a32f-kube-api-access-55zzk\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738frn4\" (UID: \"8bb8e779-075b-4123-a24a-be5ecae1a32f\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738frn4" Jul 22 00:38:13.162030 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:13.161909 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8bb8e779-075b-4123-a24a-be5ecae1a32f-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738frn4\" (UID: \"8bb8e779-075b-4123-a24a-be5ecae1a32f\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738frn4" Jul 22 00:38:13.162072 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:13.162036 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8bb8e779-075b-4123-a24a-be5ecae1a32f-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738frn4\" (UID: \"8bb8e779-075b-4123-a24a-be5ecae1a32f\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738frn4" Jul 22 00:38:13.162220 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:13.162205 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8bb8e779-075b-4123-a24a-be5ecae1a32f-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738frn4\" (UID: \"8bb8e779-075b-4123-a24a-be5ecae1a32f\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738frn4" Jul 22 00:38:13.162331 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:13.162315 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8bb8e779-075b-4123-a24a-be5ecae1a32f-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738frn4\" (UID: \"8bb8e779-075b-4123-a24a-be5ecae1a32f\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738frn4" Jul 22 00:38:13.169668 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:13.169645 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55zzk\" (UniqueName: \"kubernetes.io/projected/8bb8e779-075b-4123-a24a-be5ecae1a32f-kube-api-access-55zzk\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738frn4\" (UID: \"8bb8e779-075b-4123-a24a-be5ecae1a32f\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738frn4" Jul 22 00:38:13.171158 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:13.171135 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q4vzc"] Jul 22 00:38:13.174812 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:13.174789 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q4vzc" Jul 22 00:38:13.182594 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:13.182572 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q4vzc"] Jul 22 00:38:13.263289 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:13.263262 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8phz\" (UniqueName: \"kubernetes.io/projected/4f0c0663-abac-4777-a42e-00bb0cf90c9f-kube-api-access-x8phz\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q4vzc\" (UID: \"4f0c0663-abac-4777-a42e-00bb0cf90c9f\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q4vzc" Jul 22 00:38:13.263447 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:13.263297 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4f0c0663-abac-4777-a42e-00bb0cf90c9f-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q4vzc\" (UID: \"4f0c0663-abac-4777-a42e-00bb0cf90c9f\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q4vzc" Jul 22 00:38:13.263447 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:13.263328 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4f0c0663-abac-4777-a42e-00bb0cf90c9f-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q4vzc\" (UID: \"4f0c0663-abac-4777-a42e-00bb0cf90c9f\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q4vzc" Jul 22 00:38:13.290210 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:13.290150 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738frn4" Jul 22 00:38:13.364128 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:13.364100 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8phz\" (UniqueName: \"kubernetes.io/projected/4f0c0663-abac-4777-a42e-00bb0cf90c9f-kube-api-access-x8phz\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q4vzc\" (UID: \"4f0c0663-abac-4777-a42e-00bb0cf90c9f\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q4vzc" Jul 22 00:38:13.364280 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:13.364140 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4f0c0663-abac-4777-a42e-00bb0cf90c9f-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q4vzc\" (UID: \"4f0c0663-abac-4777-a42e-00bb0cf90c9f\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q4vzc" Jul 22 00:38:13.364280 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:13.364185 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4f0c0663-abac-4777-a42e-00bb0cf90c9f-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q4vzc\" (UID: \"4f0c0663-abac-4777-a42e-00bb0cf90c9f\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q4vzc" Jul 22 00:38:13.364653 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:13.364632 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4f0c0663-abac-4777-a42e-00bb0cf90c9f-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q4vzc\" (UID: \"4f0c0663-abac-4777-a42e-00bb0cf90c9f\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q4vzc" Jul 22 00:38:13.364741 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:13.364663 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4f0c0663-abac-4777-a42e-00bb0cf90c9f-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q4vzc\" (UID: \"4f0c0663-abac-4777-a42e-00bb0cf90c9f\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q4vzc" Jul 22 00:38:13.372106 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:13.372079 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8phz\" (UniqueName: \"kubernetes.io/projected/4f0c0663-abac-4777-a42e-00bb0cf90c9f-kube-api-access-x8phz\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q4vzc\" (UID: \"4f0c0663-abac-4777-a42e-00bb0cf90c9f\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q4vzc" Jul 22 00:38:13.417524 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:13.417497 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738frn4"] Jul 22 00:38:13.419027 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:38:13.419000 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bb8e779_075b_4123_a24a_be5ecae1a32f.slice/crio-87ff736d01cb7938c81340c1dbb724b75e78b6609716e62e0f795767b936b1a3 WatchSource:0}: Error finding container 87ff736d01cb7938c81340c1dbb724b75e78b6609716e62e0f795767b936b1a3: Status 404 returned error can't find the container with id 87ff736d01cb7938c81340c1dbb724b75e78b6609716e62e0f795767b936b1a3 Jul 22 00:38:13.492078 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:13.492057 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q4vzc" Jul 22 00:38:13.587912 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:13.587879 2574 generic.go:356] "Generic (PLEG): container finished" podID="1e1e65cd-f7b5-439f-ab31-cad60691ce9a" containerID="f61991a7de685b9ff973fcdd1a66c871649cf867730711ceb446d33a78540522" exitCode=0 Jul 22 00:38:13.588074 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:13.587970 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef148wqw" event={"ID":"1e1e65cd-f7b5-439f-ab31-cad60691ce9a","Type":"ContainerDied","Data":"f61991a7de685b9ff973fcdd1a66c871649cf867730711ceb446d33a78540522"} Jul 22 00:38:13.588074 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:13.588029 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef148wqw" event={"ID":"1e1e65cd-f7b5-439f-ab31-cad60691ce9a","Type":"ContainerStarted","Data":"adffdebd4c1e2170ba2379b6868a281bee9f4fbc0b8452beba4bb930cd6bf339"} Jul 22 00:38:13.588565 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:13.588517 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 00:38:13.590640 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:13.590199 2574 generic.go:356] "Generic (PLEG): container finished" podID="8bb8e779-075b-4123-a24a-be5ecae1a32f" containerID="47e86a203bea76f9337bb5e8f0f9e43ddae60d725820d7a1a5f21e11209c4a1b" exitCode=0 Jul 22 00:38:13.590640 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:13.590298 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738frn4" event={"ID":"8bb8e779-075b-4123-a24a-be5ecae1a32f","Type":"ContainerDied","Data":"47e86a203bea76f9337bb5e8f0f9e43ddae60d725820d7a1a5f21e11209c4a1b"} Jul 22 00:38:13.590640 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:13.590335 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738frn4" event={"ID":"8bb8e779-075b-4123-a24a-be5ecae1a32f","Type":"ContainerStarted","Data":"87ff736d01cb7938c81340c1dbb724b75e78b6609716e62e0f795767b936b1a3"} Jul 22 00:38:13.593281 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:13.593256 2574 generic.go:356] "Generic (PLEG): container finished" podID="6ce6a5b6-b585-45b1-99ee-ee0c9826c982" containerID="327470f0cf7201f3051728cc0117d7e601566a1c27bb5c4b1f4bd3c15a95f0d1" exitCode=0 Jul 22 00:38:13.593383 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:13.593297 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mfn4p" event={"ID":"6ce6a5b6-b585-45b1-99ee-ee0c9826c982","Type":"ContainerDied","Data":"327470f0cf7201f3051728cc0117d7e601566a1c27bb5c4b1f4bd3c15a95f0d1"} Jul 22 00:38:13.593383 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:13.593347 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mfn4p" event={"ID":"6ce6a5b6-b585-45b1-99ee-ee0c9826c982","Type":"ContainerStarted","Data":"59bbbaf0fe9d8d5fed0ce9e606c023e1097051903657f9cc22065035940fc53d"} Jul 22 00:38:13.618573 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:13.618547 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q4vzc"] Jul 22 00:38:13.621498 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:38:13.621471 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f0c0663_abac_4777_a42e_00bb0cf90c9f.slice/crio-e17f61bce19d36d06ef4f382baa4983df784d08ba8c4a393f2b0ecd772ccda98 WatchSource:0}: Error finding container e17f61bce19d36d06ef4f382baa4983df784d08ba8c4a393f2b0ecd772ccda98: Status 404 returned error can't find the container with id e17f61bce19d36d06ef4f382baa4983df784d08ba8c4a393f2b0ecd772ccda98 Jul 22 00:38:14.606203 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:14.606122 2574 generic.go:356] "Generic (PLEG): container finished" podID="4f0c0663-abac-4777-a42e-00bb0cf90c9f" containerID="a5363bb2e478c98006f3ac56fa219f3a1af6398514b33a392b67ad34ed26b651" exitCode=0 Jul 22 00:38:14.606622 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:14.606211 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q4vzc" event={"ID":"4f0c0663-abac-4777-a42e-00bb0cf90c9f","Type":"ContainerDied","Data":"a5363bb2e478c98006f3ac56fa219f3a1af6398514b33a392b67ad34ed26b651"} Jul 22 00:38:14.606622 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:14.606298 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q4vzc" event={"ID":"4f0c0663-abac-4777-a42e-00bb0cf90c9f","Type":"ContainerStarted","Data":"e17f61bce19d36d06ef4f382baa4983df784d08ba8c4a393f2b0ecd772ccda98"} Jul 22 00:38:14.608377 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:14.608356 2574 generic.go:356] "Generic (PLEG): container finished" podID="8bb8e779-075b-4123-a24a-be5ecae1a32f" containerID="32668b04dca82f143e5dc610e3bb1f66695bb5b03bb4eb9fd6c32b2ae4386712" exitCode=0 Jul 22 00:38:14.608482 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:14.608410 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738frn4" event={"ID":"8bb8e779-075b-4123-a24a-be5ecae1a32f","Type":"ContainerDied","Data":"32668b04dca82f143e5dc610e3bb1f66695bb5b03bb4eb9fd6c32b2ae4386712"} Jul 22 00:38:15.615336 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:15.615301 2574 generic.go:356] "Generic (PLEG): container finished" podID="1e1e65cd-f7b5-439f-ab31-cad60691ce9a" containerID="307dba04d3b53819a624e0b802c0ba8abe6f847bcff18992865d24ed888fc6d2" exitCode=0 Jul 22 00:38:15.615672 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:15.615371 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef148wqw" event={"ID":"1e1e65cd-f7b5-439f-ab31-cad60691ce9a","Type":"ContainerDied","Data":"307dba04d3b53819a624e0b802c0ba8abe6f847bcff18992865d24ed888fc6d2"} Jul 22 00:38:15.617524 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:15.617500 2574 generic.go:356] "Generic (PLEG): container finished" podID="8bb8e779-075b-4123-a24a-be5ecae1a32f" containerID="177243a6ab0ea207ef806efab094e3c87a9151ea31ee3e6f527da373612722f0" exitCode=0 Jul 22 00:38:15.617648 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:15.617580 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738frn4" event={"ID":"8bb8e779-075b-4123-a24a-be5ecae1a32f","Type":"ContainerDied","Data":"177243a6ab0ea207ef806efab094e3c87a9151ea31ee3e6f527da373612722f0"} Jul 22 00:38:15.619650 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:15.619536 2574 generic.go:356] "Generic (PLEG): container finished" podID="6ce6a5b6-b585-45b1-99ee-ee0c9826c982" containerID="e2659ea5e521a102528f894d79ee169228d35c080ac746db6b349aeaeb955011" exitCode=0 Jul 22 00:38:15.619650 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:15.619565 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mfn4p" event={"ID":"6ce6a5b6-b585-45b1-99ee-ee0c9826c982","Type":"ContainerDied","Data":"e2659ea5e521a102528f894d79ee169228d35c080ac746db6b349aeaeb955011"} Jul 22 00:38:15.622052 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:15.622033 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q4vzc" event={"ID":"4f0c0663-abac-4777-a42e-00bb0cf90c9f","Type":"ContainerStarted","Data":"eb71fbc80720e38d8b114fef47e3c88b3298d1592173ab1e5501cb6f5aed5abe"} Jul 22 00:38:16.628271 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:16.628236 2574 generic.go:356] "Generic (PLEG): container finished" podID="1e1e65cd-f7b5-439f-ab31-cad60691ce9a" containerID="fb5fc87b98990e536fca8f5df3864f39e4ac2645050e331f2f4a3625857bcf73" exitCode=0 Jul 22 00:38:16.628707 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:16.628317 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef148wqw" event={"ID":"1e1e65cd-f7b5-439f-ab31-cad60691ce9a","Type":"ContainerDied","Data":"fb5fc87b98990e536fca8f5df3864f39e4ac2645050e331f2f4a3625857bcf73"} Jul 22 00:38:16.630251 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:16.630229 2574 generic.go:356] "Generic (PLEG): container finished" podID="6ce6a5b6-b585-45b1-99ee-ee0c9826c982" containerID="55775c86e2b57895fc1c46d87e5a379921d7890080f3ffe7d1a411bd6b3a619b" exitCode=0 Jul 22 00:38:16.630341 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:16.630280 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mfn4p" event={"ID":"6ce6a5b6-b585-45b1-99ee-ee0c9826c982","Type":"ContainerDied","Data":"55775c86e2b57895fc1c46d87e5a379921d7890080f3ffe7d1a411bd6b3a619b"} Jul 22 00:38:16.631818 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:16.631799 2574 generic.go:356] "Generic (PLEG): container finished" podID="4f0c0663-abac-4777-a42e-00bb0cf90c9f" containerID="eb71fbc80720e38d8b114fef47e3c88b3298d1592173ab1e5501cb6f5aed5abe" exitCode=0 Jul 22 00:38:16.631920 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:16.631861 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q4vzc" event={"ID":"4f0c0663-abac-4777-a42e-00bb0cf90c9f","Type":"ContainerDied","Data":"eb71fbc80720e38d8b114fef47e3c88b3298d1592173ab1e5501cb6f5aed5abe"} Jul 22 00:38:16.779506 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:16.779479 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738frn4" Jul 22 00:38:16.898075 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:16.898045 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8bb8e779-075b-4123-a24a-be5ecae1a32f-util\") pod \"8bb8e779-075b-4123-a24a-be5ecae1a32f\" (UID: \"8bb8e779-075b-4123-a24a-be5ecae1a32f\") " Jul 22 00:38:16.898223 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:16.898126 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-55zzk\" (UniqueName: \"kubernetes.io/projected/8bb8e779-075b-4123-a24a-be5ecae1a32f-kube-api-access-55zzk\") pod \"8bb8e779-075b-4123-a24a-be5ecae1a32f\" (UID: \"8bb8e779-075b-4123-a24a-be5ecae1a32f\") " Jul 22 00:38:16.898223 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:16.898146 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8bb8e779-075b-4123-a24a-be5ecae1a32f-bundle\") pod \"8bb8e779-075b-4123-a24a-be5ecae1a32f\" (UID: \"8bb8e779-075b-4123-a24a-be5ecae1a32f\") " Jul 22 00:38:16.898976 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:16.898947 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bb8e779-075b-4123-a24a-be5ecae1a32f-bundle" (OuterVolumeSpecName: "bundle") pod "8bb8e779-075b-4123-a24a-be5ecae1a32f" (UID: "8bb8e779-075b-4123-a24a-be5ecae1a32f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 00:38:16.900169 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:16.900147 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bb8e779-075b-4123-a24a-be5ecae1a32f-kube-api-access-55zzk" (OuterVolumeSpecName: "kube-api-access-55zzk") pod "8bb8e779-075b-4123-a24a-be5ecae1a32f" (UID: "8bb8e779-075b-4123-a24a-be5ecae1a32f"). InnerVolumeSpecName "kube-api-access-55zzk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 00:38:16.903404 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:16.903384 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bb8e779-075b-4123-a24a-be5ecae1a32f-util" (OuterVolumeSpecName: "util") pod "8bb8e779-075b-4123-a24a-be5ecae1a32f" (UID: "8bb8e779-075b-4123-a24a-be5ecae1a32f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 00:38:16.999729 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:16.999651 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55zzk\" (UniqueName: \"kubernetes.io/projected/8bb8e779-075b-4123-a24a-be5ecae1a32f-kube-api-access-55zzk\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 00:38:16.999729 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:16.999677 2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8bb8e779-075b-4123-a24a-be5ecae1a32f-bundle\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 00:38:16.999729 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:16.999689 2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8bb8e779-075b-4123-a24a-be5ecae1a32f-util\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 00:38:17.638289 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:17.638255 2574 generic.go:356] "Generic (PLEG): container finished" podID="4f0c0663-abac-4777-a42e-00bb0cf90c9f" containerID="2e5513d09268ca8c6d4d2e52196489af8331fe90743e41cca6010dc88e79c459" exitCode=0 Jul 22 00:38:17.638788 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:17.638337 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q4vzc" event={"ID":"4f0c0663-abac-4777-a42e-00bb0cf90c9f","Type":"ContainerDied","Data":"2e5513d09268ca8c6d4d2e52196489af8331fe90743e41cca6010dc88e79c459"} Jul 22 00:38:17.640198 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:17.640179 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738frn4" Jul 22 00:38:17.640324 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:17.640211 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738frn4" event={"ID":"8bb8e779-075b-4123-a24a-be5ecae1a32f","Type":"ContainerDied","Data":"87ff736d01cb7938c81340c1dbb724b75e78b6609716e62e0f795767b936b1a3"} Jul 22 00:38:17.640324 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:17.640253 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87ff736d01cb7938c81340c1dbb724b75e78b6609716e62e0f795767b936b1a3" Jul 22 00:38:17.792003 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:17.791976 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef148wqw" Jul 22 00:38:17.793883 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:17.793867 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mfn4p" Jul 22 00:38:17.908054 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:17.908026 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6ce6a5b6-b585-45b1-99ee-ee0c9826c982-util\") pod \"6ce6a5b6-b585-45b1-99ee-ee0c9826c982\" (UID: \"6ce6a5b6-b585-45b1-99ee-ee0c9826c982\") " Jul 22 00:38:17.908228 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:17.908068 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1e1e65cd-f7b5-439f-ab31-cad60691ce9a-util\") pod \"1e1e65cd-f7b5-439f-ab31-cad60691ce9a\" (UID: \"1e1e65cd-f7b5-439f-ab31-cad60691ce9a\") " Jul 22 00:38:17.908228 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:17.908094 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6ce6a5b6-b585-45b1-99ee-ee0c9826c982-bundle\") pod \"6ce6a5b6-b585-45b1-99ee-ee0c9826c982\" (UID: \"6ce6a5b6-b585-45b1-99ee-ee0c9826c982\") " Jul 22 00:38:17.908228 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:17.908150 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1e1e65cd-f7b5-439f-ab31-cad60691ce9a-bundle\") pod \"1e1e65cd-f7b5-439f-ab31-cad60691ce9a\" (UID: \"1e1e65cd-f7b5-439f-ab31-cad60691ce9a\") " Jul 22 00:38:17.908228 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:17.908177 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4g8h6\" (UniqueName: \"kubernetes.io/projected/1e1e65cd-f7b5-439f-ab31-cad60691ce9a-kube-api-access-4g8h6\") pod \"1e1e65cd-f7b5-439f-ab31-cad60691ce9a\" (UID: \"1e1e65cd-f7b5-439f-ab31-cad60691ce9a\") " Jul 22 00:38:17.908228 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:17.908210 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hk4zl\" (UniqueName: \"kubernetes.io/projected/6ce6a5b6-b585-45b1-99ee-ee0c9826c982-kube-api-access-hk4zl\") pod \"6ce6a5b6-b585-45b1-99ee-ee0c9826c982\" (UID: \"6ce6a5b6-b585-45b1-99ee-ee0c9826c982\") " Jul 22 00:38:17.908749 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:17.908706 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ce6a5b6-b585-45b1-99ee-ee0c9826c982-bundle" (OuterVolumeSpecName: "bundle") pod "6ce6a5b6-b585-45b1-99ee-ee0c9826c982" (UID: "6ce6a5b6-b585-45b1-99ee-ee0c9826c982"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 00:38:17.908749 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:17.908730 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e1e65cd-f7b5-439f-ab31-cad60691ce9a-bundle" (OuterVolumeSpecName: "bundle") pod "1e1e65cd-f7b5-439f-ab31-cad60691ce9a" (UID: "1e1e65cd-f7b5-439f-ab31-cad60691ce9a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 00:38:17.910402 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:17.910368 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e1e65cd-f7b5-439f-ab31-cad60691ce9a-kube-api-access-4g8h6" (OuterVolumeSpecName: "kube-api-access-4g8h6") pod "1e1e65cd-f7b5-439f-ab31-cad60691ce9a" (UID: "1e1e65cd-f7b5-439f-ab31-cad60691ce9a"). InnerVolumeSpecName "kube-api-access-4g8h6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 00:38:17.910642 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:17.910616 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ce6a5b6-b585-45b1-99ee-ee0c9826c982-kube-api-access-hk4zl" (OuterVolumeSpecName: "kube-api-access-hk4zl") pod "6ce6a5b6-b585-45b1-99ee-ee0c9826c982" (UID: "6ce6a5b6-b585-45b1-99ee-ee0c9826c982"). InnerVolumeSpecName "kube-api-access-hk4zl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 00:38:17.914179 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:17.914158 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e1e65cd-f7b5-439f-ab31-cad60691ce9a-util" (OuterVolumeSpecName: "util") pod "1e1e65cd-f7b5-439f-ab31-cad60691ce9a" (UID: "1e1e65cd-f7b5-439f-ab31-cad60691ce9a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 00:38:17.914571 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:17.914552 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ce6a5b6-b585-45b1-99ee-ee0c9826c982-util" (OuterVolumeSpecName: "util") pod "6ce6a5b6-b585-45b1-99ee-ee0c9826c982" (UID: "6ce6a5b6-b585-45b1-99ee-ee0c9826c982"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 00:38:18.008854 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:18.008789 2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1e1e65cd-f7b5-439f-ab31-cad60691ce9a-bundle\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 00:38:18.008854 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:18.008812 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4g8h6\" (UniqueName: \"kubernetes.io/projected/1e1e65cd-f7b5-439f-ab31-cad60691ce9a-kube-api-access-4g8h6\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 00:38:18.008854 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:18.008822 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hk4zl\" (UniqueName: \"kubernetes.io/projected/6ce6a5b6-b585-45b1-99ee-ee0c9826c982-kube-api-access-hk4zl\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 00:38:18.008854 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:18.008830 2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6ce6a5b6-b585-45b1-99ee-ee0c9826c982-util\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 00:38:18.008854 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:18.008838 2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1e1e65cd-f7b5-439f-ab31-cad60691ce9a-util\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 00:38:18.008854 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:18.008847 2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6ce6a5b6-b585-45b1-99ee-ee0c9826c982-bundle\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 00:38:18.646872 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:18.646846 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef148wqw" Jul 22 00:38:18.647261 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:18.646872 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef148wqw" event={"ID":"1e1e65cd-f7b5-439f-ab31-cad60691ce9a","Type":"ContainerDied","Data":"adffdebd4c1e2170ba2379b6868a281bee9f4fbc0b8452beba4bb930cd6bf339"} Jul 22 00:38:18.647261 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:18.646921 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="adffdebd4c1e2170ba2379b6868a281bee9f4fbc0b8452beba4bb930cd6bf339" Jul 22 00:38:18.648660 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:18.648635 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mfn4p" event={"ID":"6ce6a5b6-b585-45b1-99ee-ee0c9826c982","Type":"ContainerDied","Data":"59bbbaf0fe9d8d5fed0ce9e606c023e1097051903657f9cc22065035940fc53d"} Jul 22 00:38:18.648660 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:18.648660 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mfn4p" Jul 22 00:38:18.648809 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:18.648678 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59bbbaf0fe9d8d5fed0ce9e606c023e1097051903657f9cc22065035940fc53d" Jul 22 00:38:18.777853 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:18.777829 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q4vzc" Jul 22 00:38:18.917302 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:18.917226 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4f0c0663-abac-4777-a42e-00bb0cf90c9f-util\") pod \"4f0c0663-abac-4777-a42e-00bb0cf90c9f\" (UID: \"4f0c0663-abac-4777-a42e-00bb0cf90c9f\") " Jul 22 00:38:18.917302 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:18.917255 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4f0c0663-abac-4777-a42e-00bb0cf90c9f-bundle\") pod \"4f0c0663-abac-4777-a42e-00bb0cf90c9f\" (UID: \"4f0c0663-abac-4777-a42e-00bb0cf90c9f\") " Jul 22 00:38:18.917302 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:18.917277 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x8phz\" (UniqueName: \"kubernetes.io/projected/4f0c0663-abac-4777-a42e-00bb0cf90c9f-kube-api-access-x8phz\") pod \"4f0c0663-abac-4777-a42e-00bb0cf90c9f\" (UID: \"4f0c0663-abac-4777-a42e-00bb0cf90c9f\") " Jul 22 00:38:18.917697 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:18.917666 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f0c0663-abac-4777-a42e-00bb0cf90c9f-bundle" (OuterVolumeSpecName: "bundle") pod "4f0c0663-abac-4777-a42e-00bb0cf90c9f" (UID: "4f0c0663-abac-4777-a42e-00bb0cf90c9f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 00:38:18.919500 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:18.919485 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f0c0663-abac-4777-a42e-00bb0cf90c9f-kube-api-access-x8phz" (OuterVolumeSpecName: "kube-api-access-x8phz") pod "4f0c0663-abac-4777-a42e-00bb0cf90c9f" (UID: "4f0c0663-abac-4777-a42e-00bb0cf90c9f"). InnerVolumeSpecName "kube-api-access-x8phz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 00:38:18.922417 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:18.922383 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f0c0663-abac-4777-a42e-00bb0cf90c9f-util" (OuterVolumeSpecName: "util") pod "4f0c0663-abac-4777-a42e-00bb0cf90c9f" (UID: "4f0c0663-abac-4777-a42e-00bb0cf90c9f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 00:38:19.018061 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:19.018016 2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4f0c0663-abac-4777-a42e-00bb0cf90c9f-util\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 00:38:19.018061 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:19.018057 2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4f0c0663-abac-4777-a42e-00bb0cf90c9f-bundle\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 00:38:19.018061 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:19.018066 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8phz\" (UniqueName: \"kubernetes.io/projected/4f0c0663-abac-4777-a42e-00bb0cf90c9f-kube-api-access-x8phz\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 00:38:19.656254 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:19.656217 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q4vzc" event={"ID":"4f0c0663-abac-4777-a42e-00bb0cf90c9f","Type":"ContainerDied","Data":"e17f61bce19d36d06ef4f382baa4983df784d08ba8c4a393f2b0ecd772ccda98"} Jul 22 00:38:19.656254 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:19.656235 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q4vzc" Jul 22 00:38:19.656696 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:19.656268 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e17f61bce19d36d06ef4f382baa4983df784d08ba8c4a393f2b0ecd772ccda98" Jul 22 00:38:20.041971 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:20.041884 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-59878fc9f5-pwpl6" Jul 22 00:38:20.041971 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:20.041947 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-59878fc9f5-pwpl6" Jul 22 00:38:20.046706 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:20.046685 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-59878fc9f5-pwpl6" Jul 22 00:38:20.665220 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:20.665192 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-59878fc9f5-pwpl6" Jul 22 00:38:34.026911 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:34.026875 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fkc72"] Jul 22 00:38:34.027384 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:34.027233 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bb8e779-075b-4123-a24a-be5ecae1a32f" containerName="extract" Jul 22 00:38:34.027384 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:34.027245 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bb8e779-075b-4123-a24a-be5ecae1a32f" containerName="extract" Jul 22 00:38:34.027384 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:34.027259 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f0c0663-abac-4777-a42e-00bb0cf90c9f" containerName="util" Jul 22 00:38:34.027384 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:34.027267 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f0c0663-abac-4777-a42e-00bb0cf90c9f" containerName="util" Jul 22 00:38:34.027384 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:34.027273 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bb8e779-075b-4123-a24a-be5ecae1a32f" containerName="util" Jul 22 00:38:34.027384 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:34.027281 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bb8e779-075b-4123-a24a-be5ecae1a32f" containerName="util" Jul 22 00:38:34.027384 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:34.027287 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e1e65cd-f7b5-439f-ab31-cad60691ce9a" containerName="extract" Jul 22 00:38:34.027384 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:34.027292 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e1e65cd-f7b5-439f-ab31-cad60691ce9a" containerName="extract" Jul 22 00:38:34.027384 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:34.027303 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ce6a5b6-b585-45b1-99ee-ee0c9826c982" containerName="util" Jul 22 00:38:34.027384 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:34.027308 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ce6a5b6-b585-45b1-99ee-ee0c9826c982" containerName="util" Jul 22 00:38:34.027384 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:34.027314 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f0c0663-abac-4777-a42e-00bb0cf90c9f" containerName="pull" Jul 22 00:38:34.027384 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:34.027319 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f0c0663-abac-4777-a42e-00bb0cf90c9f" containerName="pull" Jul 22 00:38:34.027384 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:34.027328 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f0c0663-abac-4777-a42e-00bb0cf90c9f" containerName="extract" Jul 22 00:38:34.027384 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:34.027337 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f0c0663-abac-4777-a42e-00bb0cf90c9f" containerName="extract" Jul 22 00:38:34.027384 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:34.027348 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bb8e779-075b-4123-a24a-be5ecae1a32f" containerName="pull" Jul 22 00:38:34.027384 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:34.027353 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bb8e779-075b-4123-a24a-be5ecae1a32f" containerName="pull" Jul 22 00:38:34.027384 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:34.027358 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ce6a5b6-b585-45b1-99ee-ee0c9826c982" containerName="pull" Jul 22 00:38:34.027384 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:34.027363 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ce6a5b6-b585-45b1-99ee-ee0c9826c982" containerName="pull" Jul 22 00:38:34.027384 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:34.027368 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e1e65cd-f7b5-439f-ab31-cad60691ce9a" containerName="util" Jul 22 00:38:34.027384 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:34.027372 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e1e65cd-f7b5-439f-ab31-cad60691ce9a" containerName="util" Jul 22 00:38:34.027384 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:34.027379 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ce6a5b6-b585-45b1-99ee-ee0c9826c982" containerName="extract" Jul 22 00:38:34.027384 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:34.027384 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ce6a5b6-b585-45b1-99ee-ee0c9826c982" containerName="extract" Jul 22 00:38:34.027384 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:34.027390 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e1e65cd-f7b5-439f-ab31-cad60691ce9a" containerName="pull" Jul 22 00:38:34.027384 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:34.027395 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e1e65cd-f7b5-439f-ab31-cad60691ce9a" containerName="pull" Jul 22 00:38:34.028097 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:34.027450 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f0c0663-abac-4777-a42e-00bb0cf90c9f" containerName="extract" Jul 22 00:38:34.028097 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:34.027459 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ce6a5b6-b585-45b1-99ee-ee0c9826c982" containerName="extract" Jul 22 00:38:34.028097 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:34.027465 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bb8e779-075b-4123-a24a-be5ecae1a32f" containerName="extract" Jul 22 00:38:34.028097 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:34.027472 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e1e65cd-f7b5-439f-ab31-cad60691ce9a" containerName="extract" Jul 22 00:38:34.033520 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:34.033501 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fkc72" Jul 22 00:38:34.035910 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:34.035893 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-pr929\"" Jul 22 00:38:34.044489 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:34.044467 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fkc72"] Jul 22 00:38:34.154259 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:34.154225 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d32cf1fc-fc00-48aa-90bd-55e8730603ff-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-fkc72\" (UID: \"d32cf1fc-fc00-48aa-90bd-55e8730603ff\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fkc72" Jul 22 00:38:34.154456 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:34.154380 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8j7m\" (UniqueName: \"kubernetes.io/projected/d32cf1fc-fc00-48aa-90bd-55e8730603ff-kube-api-access-d8j7m\") pod \"kuadrant-operator-controller-manager-8684c9ff49-fkc72\" (UID: \"d32cf1fc-fc00-48aa-90bd-55e8730603ff\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fkc72" Jul 22 00:38:34.255146 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:34.255107 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8j7m\" (UniqueName: \"kubernetes.io/projected/d32cf1fc-fc00-48aa-90bd-55e8730603ff-kube-api-access-d8j7m\") pod \"kuadrant-operator-controller-manager-8684c9ff49-fkc72\" (UID: \"d32cf1fc-fc00-48aa-90bd-55e8730603ff\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fkc72" Jul 22 00:38:34.255332 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:34.255153 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d32cf1fc-fc00-48aa-90bd-55e8730603ff-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-fkc72\" (UID: \"d32cf1fc-fc00-48aa-90bd-55e8730603ff\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fkc72" Jul 22 00:38:34.255558 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:34.255538 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d32cf1fc-fc00-48aa-90bd-55e8730603ff-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-fkc72\" (UID: \"d32cf1fc-fc00-48aa-90bd-55e8730603ff\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fkc72" Jul 22 00:38:34.269113 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:34.269081 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8j7m\" (UniqueName: \"kubernetes.io/projected/d32cf1fc-fc00-48aa-90bd-55e8730603ff-kube-api-access-d8j7m\") pod \"kuadrant-operator-controller-manager-8684c9ff49-fkc72\" (UID: \"d32cf1fc-fc00-48aa-90bd-55e8730603ff\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fkc72" Jul 22 00:38:34.344709 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:34.344634 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fkc72" Jul 22 00:38:34.485144 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:34.485117 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fkc72"] Jul 22 00:38:34.487290 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:38:34.487263 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd32cf1fc_fc00_48aa_90bd_55e8730603ff.slice/crio-8c489f055c7073c50e5bb36c4c1b6da48db9e778e48f496f831b5b5a40271fc9 WatchSource:0}: Error finding container 8c489f055c7073c50e5bb36c4c1b6da48db9e778e48f496f831b5b5a40271fc9: Status 404 returned error can't find the container with id 8c489f055c7073c50e5bb36c4c1b6da48db9e778e48f496f831b5b5a40271fc9 Jul 22 00:38:34.728929 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:34.728883 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fkc72" event={"ID":"d32cf1fc-fc00-48aa-90bd-55e8730603ff","Type":"ContainerStarted","Data":"8c489f055c7073c50e5bb36c4c1b6da48db9e778e48f496f831b5b5a40271fc9"} Jul 22 00:38:40.377407 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:40.377373 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9j7sw"] Jul 22 00:38:40.380786 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:40.380754 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9j7sw" Jul 22 00:38:40.383900 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:40.383713 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-4hdbg\"" Jul 22 00:38:40.396403 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:40.396378 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9j7sw"] Jul 22 00:38:40.407881 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:40.407858 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs5qv\" (UniqueName: \"kubernetes.io/projected/8ff59e62-7fa8-4e20-ae28-9d575e5bb6da-kube-api-access-hs5qv\") pod \"limitador-operator-controller-manager-859d7b78bd-9j7sw\" (UID: \"8ff59e62-7fa8-4e20-ae28-9d575e5bb6da\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9j7sw" Jul 22 00:38:40.509105 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:40.509078 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs5qv\" (UniqueName: \"kubernetes.io/projected/8ff59e62-7fa8-4e20-ae28-9d575e5bb6da-kube-api-access-hs5qv\") pod \"limitador-operator-controller-manager-859d7b78bd-9j7sw\" (UID: \"8ff59e62-7fa8-4e20-ae28-9d575e5bb6da\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9j7sw" Jul 22 00:38:40.516485 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:40.516453 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs5qv\" (UniqueName: \"kubernetes.io/projected/8ff59e62-7fa8-4e20-ae28-9d575e5bb6da-kube-api-access-hs5qv\") pod \"limitador-operator-controller-manager-859d7b78bd-9j7sw\" (UID: \"8ff59e62-7fa8-4e20-ae28-9d575e5bb6da\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9j7sw" Jul 22 00:38:40.698127 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:40.698044 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9j7sw" Jul 22 00:38:40.765484 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:40.765444 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fkc72" event={"ID":"d32cf1fc-fc00-48aa-90bd-55e8730603ff","Type":"ContainerStarted","Data":"d46d005944e606f9809ee7240e5a41ec37dc6fc8e6d0761087629d002e12387b"} Jul 22 00:38:40.765996 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:40.765976 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fkc72" Jul 22 00:38:40.784545 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:40.784503 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fkc72" podStartSLOduration=0.841126578 podStartE2EDuration="6.784487787s" totalImagesPullingTime="5.943361209s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:38:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:38:40.782685424 +0000 UTC m=+706.139443313" watchObservedRunningTime="2026-07-22 00:38:40.784487787 +0000 UTC m=+706.141245675" Jul 22 00:38:40.836093 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:40.836062 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9j7sw"] Jul 22 00:38:40.838127 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:38:40.838091 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ff59e62_7fa8_4e20_ae28_9d575e5bb6da.slice/crio-164ec792296b4ebee309b27f8d0b0e7493431f83f37bbac482c470b85c729a29 WatchSource:0}: Error finding container 164ec792296b4ebee309b27f8d0b0e7493431f83f37bbac482c470b85c729a29: Status 404 returned error can't find the container with id 164ec792296b4ebee309b27f8d0b0e7493431f83f37bbac482c470b85c729a29 Jul 22 00:38:41.772925 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:41.772887 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9j7sw" event={"ID":"8ff59e62-7fa8-4e20-ae28-9d575e5bb6da","Type":"ContainerStarted","Data":"164ec792296b4ebee309b27f8d0b0e7493431f83f37bbac482c470b85c729a29"} Jul 22 00:38:42.781047 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:42.781013 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9j7sw" event={"ID":"8ff59e62-7fa8-4e20-ae28-9d575e5bb6da","Type":"ContainerStarted","Data":"9d19f94669190acc3a99c74c54cd414fbb40c990d2b038e8a5116c860c7ddbcb"} Jul 22 00:38:42.781461 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:42.781093 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9j7sw" Jul 22 00:38:42.798629 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:42.798572 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9j7sw" podStartSLOduration=1.077462887 podStartE2EDuration="2.798558207s" totalImagesPullingTime="1.72109532s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:38:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:38:42.79602622 +0000 UTC m=+708.152784096" watchObservedRunningTime="2026-07-22 00:38:42.798558207 +0000 UTC m=+708.155316084" Jul 22 00:38:51.774796 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:51.774720 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fkc72" Jul 22 00:38:53.543510 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:53.543472 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fkc72"] Jul 22 00:38:53.543918 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:53.543660 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fkc72" podUID="d32cf1fc-fc00-48aa-90bd-55e8730603ff" containerName="manager" containerID="cri-o://d46d005944e606f9809ee7240e5a41ec37dc6fc8e6d0761087629d002e12387b" gracePeriod=2 Jul 22 00:38:53.557811 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:53.557771 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fkc72"] Jul 22 00:38:53.568952 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:53.568926 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jkvgc"] Jul 22 00:38:53.569138 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:38:53.569104 2574 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"d32cf1fc-fc00-48aa-90bd-55e8730603ff\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-fkc72\": pods \"kuadrant-operator-controller-manager-8684c9ff49-fkc72\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fkc72" Jul 22 00:38:53.569287 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:53.569274 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d32cf1fc-fc00-48aa-90bd-55e8730603ff" containerName="manager" Jul 22 00:38:53.569334 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:53.569289 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d32cf1fc-fc00-48aa-90bd-55e8730603ff" containerName="manager" Jul 22 00:38:53.569370 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:53.569358 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="d32cf1fc-fc00-48aa-90bd-55e8730603ff" containerName="manager" Jul 22 00:38:53.572226 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:53.572209 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jkvgc" Jul 22 00:38:53.575349 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:53.575327 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9j7sw"] Jul 22 00:38:53.575561 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:53.575533 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9j7sw" podUID="8ff59e62-7fa8-4e20-ae28-9d575e5bb6da" containerName="manager" containerID="cri-o://9d19f94669190acc3a99c74c54cd414fbb40c990d2b038e8a5116c860c7ddbcb" gracePeriod=2 Jul 22 00:38:53.575859 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:38:53.575769 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-fkc72\" is forbidden: User \"system:node:ip-10-0-139-192.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-192.ec2.internal' and this object" podUID="d32cf1fc-fc00-48aa-90bd-55e8730603ff" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fkc72" Jul 22 00:38:53.577228 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:53.577207 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9j7sw" Jul 22 00:38:53.581030 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:53.581006 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9j7sw"] Jul 22 00:38:53.586638 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:53.586590 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jkvgc"] Jul 22 00:38:53.600204 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:53.600176 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nrvbn"] Jul 22 00:38:53.600623 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:53.600590 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ff59e62-7fa8-4e20-ae28-9d575e5bb6da" containerName="manager" Jul 22 00:38:53.600687 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:53.600626 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ff59e62-7fa8-4e20-ae28-9d575e5bb6da" containerName="manager" Jul 22 00:38:53.600751 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:53.600714 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ff59e62-7fa8-4e20-ae28-9d575e5bb6da" containerName="manager" Jul 22 00:38:53.604734 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:38:53.604707 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-9j7sw\" is forbidden: User \"system:node:ip-10-0-139-192.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-192.ec2.internal' and this object" podUID="8ff59e62-7fa8-4e20-ae28-9d575e5bb6da" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9j7sw" Jul 22 00:38:53.606841 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:38:53.606808 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-fkc72\" is forbidden: User \"system:node:ip-10-0-139-192.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-192.ec2.internal' and this object" podUID="d32cf1fc-fc00-48aa-90bd-55e8730603ff" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fkc72" Jul 22 00:38:53.610920 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:53.610902 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nrvbn" Jul 22 00:38:53.614647 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:38:53.614476 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-9j7sw\" is forbidden: User \"system:node:ip-10-0-139-192.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-192.ec2.internal' and this object" podUID="8ff59e62-7fa8-4e20-ae28-9d575e5bb6da" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9j7sw" Jul 22 00:38:53.617060 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:53.617041 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nrvbn"] Jul 22 00:38:53.621092 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:53.621074 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29vg6\" (UniqueName: \"kubernetes.io/projected/70ce4a2f-6c5a-49cd-8286-679a6f4bff1a-kube-api-access-29vg6\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jkvgc\" (UID: \"70ce4a2f-6c5a-49cd-8286-679a6f4bff1a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jkvgc" Jul 22 00:38:53.621178 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:53.621113 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/70ce4a2f-6c5a-49cd-8286-679a6f4bff1a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jkvgc\" (UID: \"70ce4a2f-6c5a-49cd-8286-679a6f4bff1a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jkvgc" Jul 22 00:38:53.632523 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:38:53.632495 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-fkc72\" is forbidden: User \"system:node:ip-10-0-139-192.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-192.ec2.internal' and this object" podUID="d32cf1fc-fc00-48aa-90bd-55e8730603ff" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fkc72" Jul 22 00:38:53.721753 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:53.721710 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/70ce4a2f-6c5a-49cd-8286-679a6f4bff1a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jkvgc\" (UID: \"70ce4a2f-6c5a-49cd-8286-679a6f4bff1a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jkvgc" Jul 22 00:38:53.721916 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:53.721792 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt97c\" (UniqueName: \"kubernetes.io/projected/0b5eeae9-2497-4dc4-b0ef-25a654135223-kube-api-access-zt97c\") pod \"limitador-operator-controller-manager-859d7b78bd-nrvbn\" (UID: \"0b5eeae9-2497-4dc4-b0ef-25a654135223\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nrvbn" Jul 22 00:38:53.721916 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:53.721830 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29vg6\" (UniqueName: \"kubernetes.io/projected/70ce4a2f-6c5a-49cd-8286-679a6f4bff1a-kube-api-access-29vg6\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jkvgc\" (UID: \"70ce4a2f-6c5a-49cd-8286-679a6f4bff1a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jkvgc" Jul 22 00:38:53.722147 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:53.722125 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/70ce4a2f-6c5a-49cd-8286-679a6f4bff1a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jkvgc\" (UID: \"70ce4a2f-6c5a-49cd-8286-679a6f4bff1a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jkvgc" Jul 22 00:38:53.730164 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:53.730135 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29vg6\" (UniqueName: \"kubernetes.io/projected/70ce4a2f-6c5a-49cd-8286-679a6f4bff1a-kube-api-access-29vg6\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jkvgc\" (UID: \"70ce4a2f-6c5a-49cd-8286-679a6f4bff1a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jkvgc" Jul 22 00:38:53.805809 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:53.805787 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fkc72" Jul 22 00:38:53.808423 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:53.808405 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9j7sw" Jul 22 00:38:53.808509 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:38:53.808402 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-fkc72\" is forbidden: User \"system:node:ip-10-0-139-192.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-192.ec2.internal' and this object" podUID="d32cf1fc-fc00-48aa-90bd-55e8730603ff" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fkc72" Jul 22 00:38:53.810511 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:38:53.810488 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-9j7sw\" is forbidden: User \"system:node:ip-10-0-139-192.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-192.ec2.internal' and this object" podUID="8ff59e62-7fa8-4e20-ae28-9d575e5bb6da" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9j7sw" Jul 22 00:38:53.812666 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:38:53.812646 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-fkc72\" is forbidden: User \"system:node:ip-10-0-139-192.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-192.ec2.internal' and this object" podUID="d32cf1fc-fc00-48aa-90bd-55e8730603ff" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fkc72" Jul 22 00:38:53.814954 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:38:53.814936 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-9j7sw\" is forbidden: User \"system:node:ip-10-0-139-192.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-192.ec2.internal' and this object" podUID="8ff59e62-7fa8-4e20-ae28-9d575e5bb6da" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9j7sw" Jul 22 00:38:53.822340 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:53.822321 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt97c\" (UniqueName: \"kubernetes.io/projected/0b5eeae9-2497-4dc4-b0ef-25a654135223-kube-api-access-zt97c\") pod \"limitador-operator-controller-manager-859d7b78bd-nrvbn\" (UID: \"0b5eeae9-2497-4dc4-b0ef-25a654135223\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nrvbn" Jul 22 00:38:53.833215 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:53.833195 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt97c\" (UniqueName: \"kubernetes.io/projected/0b5eeae9-2497-4dc4-b0ef-25a654135223-kube-api-access-zt97c\") pod \"limitador-operator-controller-manager-859d7b78bd-nrvbn\" (UID: \"0b5eeae9-2497-4dc4-b0ef-25a654135223\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nrvbn" Jul 22 00:38:53.841482 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:53.841463 2574 generic.go:356] "Generic (PLEG): container finished" podID="8ff59e62-7fa8-4e20-ae28-9d575e5bb6da" containerID="9d19f94669190acc3a99c74c54cd414fbb40c990d2b038e8a5116c860c7ddbcb" exitCode=0 Jul 22 00:38:53.841567 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:53.841494 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9j7sw" Jul 22 00:38:53.841567 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:53.841554 2574 scope.go:117] "RemoveContainer" containerID="9d19f94669190acc3a99c74c54cd414fbb40c990d2b038e8a5116c860c7ddbcb" Jul 22 00:38:53.842898 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:53.842871 2574 generic.go:356] "Generic (PLEG): container finished" podID="d32cf1fc-fc00-48aa-90bd-55e8730603ff" containerID="d46d005944e606f9809ee7240e5a41ec37dc6fc8e6d0761087629d002e12387b" exitCode=0 Jul 22 00:38:53.842969 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:53.842906 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fkc72" Jul 22 00:38:53.844075 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:38:53.844057 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-9j7sw\" is forbidden: User \"system:node:ip-10-0-139-192.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-192.ec2.internal' and this object" podUID="8ff59e62-7fa8-4e20-ae28-9d575e5bb6da" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9j7sw" Jul 22 00:38:53.846258 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:38:53.846235 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-fkc72\" is forbidden: User \"system:node:ip-10-0-139-192.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-192.ec2.internal' and this object" podUID="d32cf1fc-fc00-48aa-90bd-55e8730603ff" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fkc72" Jul 22 00:38:53.848072 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:38:53.848053 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-fkc72\" is forbidden: User \"system:node:ip-10-0-139-192.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-192.ec2.internal' and this object" podUID="d32cf1fc-fc00-48aa-90bd-55e8730603ff" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fkc72" Jul 22 00:38:53.849730 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:53.849716 2574 scope.go:117] "RemoveContainer" containerID="9d19f94669190acc3a99c74c54cd414fbb40c990d2b038e8a5116c860c7ddbcb" Jul 22 00:38:53.849984 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:38:53.849965 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d19f94669190acc3a99c74c54cd414fbb40c990d2b038e8a5116c860c7ddbcb\": container with ID starting with 9d19f94669190acc3a99c74c54cd414fbb40c990d2b038e8a5116c860c7ddbcb not found: ID does not exist" containerID="9d19f94669190acc3a99c74c54cd414fbb40c990d2b038e8a5116c860c7ddbcb" Jul 22 00:38:53.850050 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:53.849994 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d19f94669190acc3a99c74c54cd414fbb40c990d2b038e8a5116c860c7ddbcb"} err="failed to get container status \"9d19f94669190acc3a99c74c54cd414fbb40c990d2b038e8a5116c860c7ddbcb\": rpc error: code = NotFound desc = could not find container \"9d19f94669190acc3a99c74c54cd414fbb40c990d2b038e8a5116c860c7ddbcb\": container with ID starting with 9d19f94669190acc3a99c74c54cd414fbb40c990d2b038e8a5116c860c7ddbcb not found: ID does not exist" Jul 22 00:38:53.850050 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:53.850015 2574 scope.go:117] "RemoveContainer" containerID="d46d005944e606f9809ee7240e5a41ec37dc6fc8e6d0761087629d002e12387b" Jul 22 00:38:53.850148 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:38:53.850095 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-9j7sw\" is forbidden: User \"system:node:ip-10-0-139-192.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-192.ec2.internal' and this object" podUID="8ff59e62-7fa8-4e20-ae28-9d575e5bb6da" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9j7sw" Jul 22 00:38:53.857374 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:53.857359 2574 scope.go:117] "RemoveContainer" containerID="d46d005944e606f9809ee7240e5a41ec37dc6fc8e6d0761087629d002e12387b" Jul 22 00:38:53.857642 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:38:53.857624 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d46d005944e606f9809ee7240e5a41ec37dc6fc8e6d0761087629d002e12387b\": container with ID starting with d46d005944e606f9809ee7240e5a41ec37dc6fc8e6d0761087629d002e12387b not found: ID does not exist" containerID="d46d005944e606f9809ee7240e5a41ec37dc6fc8e6d0761087629d002e12387b" Jul 22 00:38:53.857690 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:53.857648 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d46d005944e606f9809ee7240e5a41ec37dc6fc8e6d0761087629d002e12387b"} err="failed to get container status \"d46d005944e606f9809ee7240e5a41ec37dc6fc8e6d0761087629d002e12387b\": rpc error: code = NotFound desc = could not find container \"d46d005944e606f9809ee7240e5a41ec37dc6fc8e6d0761087629d002e12387b\": container with ID starting with d46d005944e606f9809ee7240e5a41ec37dc6fc8e6d0761087629d002e12387b not found: ID does not exist" Jul 22 00:38:53.923372 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:53.923347 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d32cf1fc-fc00-48aa-90bd-55e8730603ff-extensions-socket-volume\") pod \"d32cf1fc-fc00-48aa-90bd-55e8730603ff\" (UID: \"d32cf1fc-fc00-48aa-90bd-55e8730603ff\") " Jul 22 00:38:53.923487 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:53.923403 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hs5qv\" (UniqueName: \"kubernetes.io/projected/8ff59e62-7fa8-4e20-ae28-9d575e5bb6da-kube-api-access-hs5qv\") pod \"8ff59e62-7fa8-4e20-ae28-9d575e5bb6da\" (UID: \"8ff59e62-7fa8-4e20-ae28-9d575e5bb6da\") " Jul 22 00:38:53.923487 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:53.923459 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d8j7m\" (UniqueName: \"kubernetes.io/projected/d32cf1fc-fc00-48aa-90bd-55e8730603ff-kube-api-access-d8j7m\") pod \"d32cf1fc-fc00-48aa-90bd-55e8730603ff\" (UID: \"d32cf1fc-fc00-48aa-90bd-55e8730603ff\") " Jul 22 00:38:53.923927 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:53.923898 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d32cf1fc-fc00-48aa-90bd-55e8730603ff-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "d32cf1fc-fc00-48aa-90bd-55e8730603ff" (UID: "d32cf1fc-fc00-48aa-90bd-55e8730603ff"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 00:38:53.925376 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:53.925354 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ff59e62-7fa8-4e20-ae28-9d575e5bb6da-kube-api-access-hs5qv" (OuterVolumeSpecName: "kube-api-access-hs5qv") pod "8ff59e62-7fa8-4e20-ae28-9d575e5bb6da" (UID: "8ff59e62-7fa8-4e20-ae28-9d575e5bb6da"). InnerVolumeSpecName "kube-api-access-hs5qv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 00:38:53.925435 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:53.925384 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d32cf1fc-fc00-48aa-90bd-55e8730603ff-kube-api-access-d8j7m" (OuterVolumeSpecName: "kube-api-access-d8j7m") pod "d32cf1fc-fc00-48aa-90bd-55e8730603ff" (UID: "d32cf1fc-fc00-48aa-90bd-55e8730603ff"). InnerVolumeSpecName "kube-api-access-d8j7m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 00:38:53.977692 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:53.977665 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jkvgc" Jul 22 00:38:53.983948 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:53.983928 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nrvbn" Jul 22 00:38:54.024824 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:54.024786 2574 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d32cf1fc-fc00-48aa-90bd-55e8730603ff-extensions-socket-volume\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 00:38:54.024824 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:54.024821 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hs5qv\" (UniqueName: \"kubernetes.io/projected/8ff59e62-7fa8-4e20-ae28-9d575e5bb6da-kube-api-access-hs5qv\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 00:38:54.025015 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:54.024838 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8j7m\" (UniqueName: \"kubernetes.io/projected/d32cf1fc-fc00-48aa-90bd-55e8730603ff-kube-api-access-d8j7m\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 00:38:54.117580 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:54.117552 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jkvgc"] Jul 22 00:38:54.119343 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:38:54.119319 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70ce4a2f_6c5a_49cd_8286_679a6f4bff1a.slice/crio-6b1de0da7e2832b374118b01e7b8625b63ea739dac9f4b90b5f82a98614661f4 WatchSource:0}: Error finding container 6b1de0da7e2832b374118b01e7b8625b63ea739dac9f4b90b5f82a98614661f4: Status 404 returned error can't find the container with id 6b1de0da7e2832b374118b01e7b8625b63ea739dac9f4b90b5f82a98614661f4 Jul 22 00:38:54.135006 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:54.134975 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nrvbn"] Jul 22 00:38:54.138844 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:38:54.138804 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b5eeae9_2497_4dc4_b0ef_25a654135223.slice/crio-29f8a1d093644a7aed911494bf94fb5ca988d2a691a5d7db2e75be55afe222d6 WatchSource:0}: Error finding container 29f8a1d093644a7aed911494bf94fb5ca988d2a691a5d7db2e75be55afe222d6: Status 404 returned error can't find the container with id 29f8a1d093644a7aed911494bf94fb5ca988d2a691a5d7db2e75be55afe222d6 Jul 22 00:38:54.171396 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:38:54.171367 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-fkc72\" is forbidden: User \"system:node:ip-10-0-139-192.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-192.ec2.internal' and this object" podUID="d32cf1fc-fc00-48aa-90bd-55e8730603ff" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fkc72" Jul 22 00:38:54.173509 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:38:54.173475 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-9j7sw\" is forbidden: User \"system:node:ip-10-0-139-192.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-192.ec2.internal' and this object" podUID="8ff59e62-7fa8-4e20-ae28-9d575e5bb6da" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9j7sw" Jul 22 00:38:54.175718 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:38:54.175654 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-9j7sw\" is forbidden: User \"system:node:ip-10-0-139-192.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-192.ec2.internal' and this object" podUID="8ff59e62-7fa8-4e20-ae28-9d575e5bb6da" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9j7sw" Jul 22 00:38:54.177747 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:38:54.177723 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-fkc72\" is forbidden: User \"system:node:ip-10-0-139-192.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-192.ec2.internal' and this object" podUID="d32cf1fc-fc00-48aa-90bd-55e8730603ff" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fkc72" Jul 22 00:38:54.850008 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:54.849927 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nrvbn" event={"ID":"0b5eeae9-2497-4dc4-b0ef-25a654135223","Type":"ContainerStarted","Data":"93e36dc3693d430561e287270b9fb24a20560421e3076c0de28cc2ab050ec8f7"} Jul 22 00:38:54.850427 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:54.850025 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nrvbn" Jul 22 00:38:54.850427 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:54.850052 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nrvbn" event={"ID":"0b5eeae9-2497-4dc4-b0ef-25a654135223","Type":"ContainerStarted","Data":"29f8a1d093644a7aed911494bf94fb5ca988d2a691a5d7db2e75be55afe222d6"} Jul 22 00:38:54.852379 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:38:54.852352 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-9j7sw\" is forbidden: User \"system:node:ip-10-0-139-192.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-192.ec2.internal' and this object" podUID="8ff59e62-7fa8-4e20-ae28-9d575e5bb6da" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9j7sw" Jul 22 00:38:54.853238 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:54.853213 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jkvgc" event={"ID":"70ce4a2f-6c5a-49cd-8286-679a6f4bff1a","Type":"ContainerStarted","Data":"1e940bf0e41284eaa371decc8728d003ae1bec1d089604268f98a0d7b21c7e1c"} Jul 22 00:38:54.853317 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:54.853260 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jkvgc" event={"ID":"70ce4a2f-6c5a-49cd-8286-679a6f4bff1a","Type":"ContainerStarted","Data":"6b1de0da7e2832b374118b01e7b8625b63ea739dac9f4b90b5f82a98614661f4"} Jul 22 00:38:54.853358 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:54.853317 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jkvgc" Jul 22 00:38:54.870551 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:54.870513 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nrvbn" podStartSLOduration=1.870500491 podStartE2EDuration="1.870500491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:38:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:38:54.868836179 +0000 UTC m=+720.225594055" watchObservedRunningTime="2026-07-22 00:38:54.870500491 +0000 UTC m=+720.227258367" Jul 22 00:38:54.870935 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:38:54.870907 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-fkc72\" is forbidden: User \"system:node:ip-10-0-139-192.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-192.ec2.internal' and this object" podUID="d32cf1fc-fc00-48aa-90bd-55e8730603ff" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fkc72" Jul 22 00:38:54.874372 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:38:54.874352 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-fkc72\" is forbidden: User \"system:node:ip-10-0-139-192.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-192.ec2.internal' and this object" podUID="d32cf1fc-fc00-48aa-90bd-55e8730603ff" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fkc72" Jul 22 00:38:54.876686 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:38:54.876664 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-9j7sw\" is forbidden: User \"system:node:ip-10-0-139-192.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-192.ec2.internal' and this object" podUID="8ff59e62-7fa8-4e20-ae28-9d575e5bb6da" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9j7sw" Jul 22 00:38:54.899332 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:54.899293 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jkvgc" podStartSLOduration=1.899280696 podStartE2EDuration="1.899280696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:38:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:38:54.898223392 +0000 UTC m=+720.254981292" watchObservedRunningTime="2026-07-22 00:38:54.899280696 +0000 UTC m=+720.256038573" Jul 22 00:38:55.088406 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:38:55.088369 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-9j7sw\" is forbidden: User \"system:node:ip-10-0-139-192.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-192.ec2.internal' and this object" podUID="8ff59e62-7fa8-4e20-ae28-9d575e5bb6da" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9j7sw" Jul 22 00:38:55.089159 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:55.089139 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ff59e62-7fa8-4e20-ae28-9d575e5bb6da" path="/var/lib/kubelet/pods/8ff59e62-7fa8-4e20-ae28-9d575e5bb6da/volumes" Jul 22 00:38:55.089482 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:38:55.089470 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d32cf1fc-fc00-48aa-90bd-55e8730603ff" path="/var/lib/kubelet/pods/d32cf1fc-fc00-48aa-90bd-55e8730603ff/volumes" Jul 22 00:38:55.090912 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:38:55.090892 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-fkc72\" is forbidden: User \"system:node:ip-10-0-139-192.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-192.ec2.internal' and this object" podUID="d32cf1fc-fc00-48aa-90bd-55e8730603ff" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fkc72" Jul 22 00:39:05.860124 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:05.860087 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nrvbn" Jul 22 00:39:05.860597 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:05.860172 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jkvgc" Jul 22 00:39:09.613277 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:09.613235 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jkvgc"] Jul 22 00:39:09.613758 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:09.613454 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jkvgc" podUID="70ce4a2f-6c5a-49cd-8286-679a6f4bff1a" containerName="manager" containerID="cri-o://1e940bf0e41284eaa371decc8728d003ae1bec1d089604268f98a0d7b21c7e1c" gracePeriod=10 Jul 22 00:39:09.820307 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:09.820274 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n4tg8"] Jul 22 00:39:09.824037 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:09.824018 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n4tg8" Jul 22 00:39:09.841053 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:09.841026 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n4tg8"] Jul 22 00:39:09.873511 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:09.873458 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jkvgc" Jul 22 00:39:09.931650 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:09.931619 2574 generic.go:356] "Generic (PLEG): container finished" podID="70ce4a2f-6c5a-49cd-8286-679a6f4bff1a" containerID="1e940bf0e41284eaa371decc8728d003ae1bec1d089604268f98a0d7b21c7e1c" exitCode=0 Jul 22 00:39:09.931803 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:09.931672 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jkvgc" Jul 22 00:39:09.931803 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:09.931704 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jkvgc" event={"ID":"70ce4a2f-6c5a-49cd-8286-679a6f4bff1a","Type":"ContainerDied","Data":"1e940bf0e41284eaa371decc8728d003ae1bec1d089604268f98a0d7b21c7e1c"} Jul 22 00:39:09.931803 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:09.931758 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jkvgc" event={"ID":"70ce4a2f-6c5a-49cd-8286-679a6f4bff1a","Type":"ContainerDied","Data":"6b1de0da7e2832b374118b01e7b8625b63ea739dac9f4b90b5f82a98614661f4"} Jul 22 00:39:09.931803 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:09.931789 2574 scope.go:117] "RemoveContainer" containerID="1e940bf0e41284eaa371decc8728d003ae1bec1d089604268f98a0d7b21c7e1c" Jul 22 00:39:09.940194 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:09.940178 2574 scope.go:117] "RemoveContainer" containerID="1e940bf0e41284eaa371decc8728d003ae1bec1d089604268f98a0d7b21c7e1c" Jul 22 00:39:09.940450 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:39:09.940433 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e940bf0e41284eaa371decc8728d003ae1bec1d089604268f98a0d7b21c7e1c\": container with ID starting with 1e940bf0e41284eaa371decc8728d003ae1bec1d089604268f98a0d7b21c7e1c not found: ID does not exist" containerID="1e940bf0e41284eaa371decc8728d003ae1bec1d089604268f98a0d7b21c7e1c" Jul 22 00:39:09.940494 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:09.940458 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e940bf0e41284eaa371decc8728d003ae1bec1d089604268f98a0d7b21c7e1c"} err="failed to get container status \"1e940bf0e41284eaa371decc8728d003ae1bec1d089604268f98a0d7b21c7e1c\": rpc error: code = NotFound desc = could not find container \"1e940bf0e41284eaa371decc8728d003ae1bec1d089604268f98a0d7b21c7e1c\": container with ID starting with 1e940bf0e41284eaa371decc8728d003ae1bec1d089604268f98a0d7b21c7e1c not found: ID does not exist" Jul 22 00:39:09.968756 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:09.968734 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/70ce4a2f-6c5a-49cd-8286-679a6f4bff1a-extensions-socket-volume\") pod \"70ce4a2f-6c5a-49cd-8286-679a6f4bff1a\" (UID: \"70ce4a2f-6c5a-49cd-8286-679a6f4bff1a\") " Jul 22 00:39:09.968846 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:09.968774 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-29vg6\" (UniqueName: \"kubernetes.io/projected/70ce4a2f-6c5a-49cd-8286-679a6f4bff1a-kube-api-access-29vg6\") pod \"70ce4a2f-6c5a-49cd-8286-679a6f4bff1a\" (UID: \"70ce4a2f-6c5a-49cd-8286-679a6f4bff1a\") " Jul 22 00:39:09.968969 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:09.968955 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw7jt\" (UniqueName: \"kubernetes.io/projected/7e69590f-f707-4031-8a59-24a76c3210f9-kube-api-access-fw7jt\") pod \"kuadrant-operator-controller-manager-86878c769f-n4tg8\" (UID: \"7e69590f-f707-4031-8a59-24a76c3210f9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n4tg8" Jul 22 00:39:09.969009 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:09.968990 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7e69590f-f707-4031-8a59-24a76c3210f9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-n4tg8\" (UID: \"7e69590f-f707-4031-8a59-24a76c3210f9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n4tg8" Jul 22 00:39:09.969139 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:09.969117 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70ce4a2f-6c5a-49cd-8286-679a6f4bff1a-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "70ce4a2f-6c5a-49cd-8286-679a6f4bff1a" (UID: "70ce4a2f-6c5a-49cd-8286-679a6f4bff1a"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 00:39:09.970758 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:09.970739 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70ce4a2f-6c5a-49cd-8286-679a6f4bff1a-kube-api-access-29vg6" (OuterVolumeSpecName: "kube-api-access-29vg6") pod "70ce4a2f-6c5a-49cd-8286-679a6f4bff1a" (UID: "70ce4a2f-6c5a-49cd-8286-679a6f4bff1a"). InnerVolumeSpecName "kube-api-access-29vg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 00:39:10.069790 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:10.069755 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw7jt\" (UniqueName: \"kubernetes.io/projected/7e69590f-f707-4031-8a59-24a76c3210f9-kube-api-access-fw7jt\") pod \"kuadrant-operator-controller-manager-86878c769f-n4tg8\" (UID: \"7e69590f-f707-4031-8a59-24a76c3210f9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n4tg8" Jul 22 00:39:10.069947 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:10.069807 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7e69590f-f707-4031-8a59-24a76c3210f9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-n4tg8\" (UID: \"7e69590f-f707-4031-8a59-24a76c3210f9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n4tg8" Jul 22 00:39:10.069947 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:10.069846 2574 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/70ce4a2f-6c5a-49cd-8286-679a6f4bff1a-extensions-socket-volume\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 00:39:10.069947 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:10.069858 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29vg6\" (UniqueName: \"kubernetes.io/projected/70ce4a2f-6c5a-49cd-8286-679a6f4bff1a-kube-api-access-29vg6\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 00:39:10.070154 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:10.070139 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7e69590f-f707-4031-8a59-24a76c3210f9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-n4tg8\" (UID: \"7e69590f-f707-4031-8a59-24a76c3210f9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n4tg8" Jul 22 00:39:10.077715 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:10.077694 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw7jt\" (UniqueName: \"kubernetes.io/projected/7e69590f-f707-4031-8a59-24a76c3210f9-kube-api-access-fw7jt\") pod \"kuadrant-operator-controller-manager-86878c769f-n4tg8\" (UID: \"7e69590f-f707-4031-8a59-24a76c3210f9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n4tg8" Jul 22 00:39:10.135520 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:10.135497 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n4tg8" Jul 22 00:39:10.258888 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:10.258852 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jkvgc"] Jul 22 00:39:10.262292 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:10.262268 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jkvgc"] Jul 22 00:39:10.270354 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:10.270218 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n4tg8"] Jul 22 00:39:10.273221 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:39:10.273187 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e69590f_f707_4031_8a59_24a76c3210f9.slice/crio-e42fa7f87b3e1d75ad1944f9d00b3c6701d0d35a285a10c4115e4a937524c86a WatchSource:0}: Error finding container e42fa7f87b3e1d75ad1944f9d00b3c6701d0d35a285a10c4115e4a937524c86a: Status 404 returned error can't find the container with id e42fa7f87b3e1d75ad1944f9d00b3c6701d0d35a285a10c4115e4a937524c86a Jul 22 00:39:10.939665 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:10.939611 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n4tg8" event={"ID":"7e69590f-f707-4031-8a59-24a76c3210f9","Type":"ContainerStarted","Data":"2e071eb059aae87ce32912b13a0edb60869cf952d772e95ed14ad76a7582313e"} Jul 22 00:39:10.940126 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:10.939695 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n4tg8" event={"ID":"7e69590f-f707-4031-8a59-24a76c3210f9","Type":"ContainerStarted","Data":"e42fa7f87b3e1d75ad1944f9d00b3c6701d0d35a285a10c4115e4a937524c86a"} Jul 22 00:39:10.940126 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:10.939727 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n4tg8" Jul 22 00:39:10.958981 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:10.958931 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n4tg8" podStartSLOduration=1.9589166279999999 podStartE2EDuration="1.958916628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:39:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:39:10.958392824 +0000 UTC m=+736.315150721" watchObservedRunningTime="2026-07-22 00:39:10.958916628 +0000 UTC m=+736.315674505" Jul 22 00:39:11.088993 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:11.088960 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70ce4a2f-6c5a-49cd-8286-679a6f4bff1a" path="/var/lib/kubelet/pods/70ce4a2f-6c5a-49cd-8286-679a6f4bff1a/volumes" Jul 22 00:39:21.947032 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:21.947003 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n4tg8" Jul 22 00:39:41.264799 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:41.264764 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-shh5z"] Jul 22 00:39:41.265269 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:41.265112 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70ce4a2f-6c5a-49cd-8286-679a6f4bff1a" containerName="manager" Jul 22 00:39:41.265269 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:41.265123 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="70ce4a2f-6c5a-49cd-8286-679a6f4bff1a" containerName="manager" Jul 22 00:39:41.265269 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:41.265178 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="70ce4a2f-6c5a-49cd-8286-679a6f4bff1a" containerName="manager" Jul 22 00:39:41.268422 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:41.268404 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-shh5z" Jul 22 00:39:41.270913 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:41.270892 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 22 00:39:41.271014 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:41.270934 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-mphx6\"" Jul 22 00:39:41.274879 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:41.274854 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-shh5z"] Jul 22 00:39:41.335323 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:41.335289 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqkgx\" (UniqueName: \"kubernetes.io/projected/8a5237e3-29aa-488d-9c8f-cc8f2ff2a265-kube-api-access-dqkgx\") pod \"limitador-limitador-64c94fd64f-shh5z\" (UID: \"8a5237e3-29aa-488d-9c8f-cc8f2ff2a265\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-shh5z" Jul 22 00:39:41.335541 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:41.335431 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8a5237e3-29aa-488d-9c8f-cc8f2ff2a265-config-file\") pod \"limitador-limitador-64c94fd64f-shh5z\" (UID: \"8a5237e3-29aa-488d-9c8f-cc8f2ff2a265\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-shh5z" Jul 22 00:39:41.358953 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:41.358921 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-shh5z"] Jul 22 00:39:41.436656 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:41.436595 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8a5237e3-29aa-488d-9c8f-cc8f2ff2a265-config-file\") pod \"limitador-limitador-64c94fd64f-shh5z\" (UID: \"8a5237e3-29aa-488d-9c8f-cc8f2ff2a265\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-shh5z" Jul 22 00:39:41.436831 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:41.436704 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqkgx\" (UniqueName: \"kubernetes.io/projected/8a5237e3-29aa-488d-9c8f-cc8f2ff2a265-kube-api-access-dqkgx\") pod \"limitador-limitador-64c94fd64f-shh5z\" (UID: \"8a5237e3-29aa-488d-9c8f-cc8f2ff2a265\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-shh5z" Jul 22 00:39:41.437312 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:41.437290 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8a5237e3-29aa-488d-9c8f-cc8f2ff2a265-config-file\") pod \"limitador-limitador-64c94fd64f-shh5z\" (UID: \"8a5237e3-29aa-488d-9c8f-cc8f2ff2a265\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-shh5z" Jul 22 00:39:41.445268 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:41.445237 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqkgx\" (UniqueName: \"kubernetes.io/projected/8a5237e3-29aa-488d-9c8f-cc8f2ff2a265-kube-api-access-dqkgx\") pod \"limitador-limitador-64c94fd64f-shh5z\" (UID: \"8a5237e3-29aa-488d-9c8f-cc8f2ff2a265\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-shh5z" Jul 22 00:39:41.582221 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:41.582142 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-shh5z" Jul 22 00:39:41.731970 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:41.731947 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-shh5z"] Jul 22 00:39:41.734474 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:39:41.734442 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a5237e3_29aa_488d_9c8f_cc8f2ff2a265.slice/crio-d7079e5fd3a6c6fc510a1383754a2be4ff7766a22991abdd979d4ecc8095a129 WatchSource:0}: Error finding container d7079e5fd3a6c6fc510a1383754a2be4ff7766a22991abdd979d4ecc8095a129: Status 404 returned error can't find the container with id d7079e5fd3a6c6fc510a1383754a2be4ff7766a22991abdd979d4ecc8095a129 Jul 22 00:39:42.094283 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:42.094244 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-shh5z" event={"ID":"8a5237e3-29aa-488d-9c8f-cc8f2ff2a265","Type":"ContainerStarted","Data":"d7079e5fd3a6c6fc510a1383754a2be4ff7766a22991abdd979d4ecc8095a129"} Jul 22 00:39:45.110781 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:45.110739 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-shh5z" event={"ID":"8a5237e3-29aa-488d-9c8f-cc8f2ff2a265","Type":"ContainerStarted","Data":"084feb87782f7dd2463aa02fc1a1866009aa167a189de35639aecaaa63dfebf0"} Jul 22 00:39:45.111174 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:45.110829 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-shh5z" Jul 22 00:39:45.130173 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:45.130121 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-shh5z" podStartSLOduration=1.44922146 podStartE2EDuration="4.130106814s" totalImagesPullingTime="2.680885354s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:39:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:39:45.127365149 +0000 UTC m=+770.484123041" watchObservedRunningTime="2026-07-22 00:39:45.130106814 +0000 UTC m=+770.486864691" Jul 22 00:39:55.821327 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:55.821284 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-shh5z"] Jul 22 00:39:55.821842 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:55.821517 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-shh5z" podUID="8a5237e3-29aa-488d-9c8f-cc8f2ff2a265" containerName="limitador" containerID="cri-o://084feb87782f7dd2463aa02fc1a1866009aa167a189de35639aecaaa63dfebf0" gracePeriod=30 Jul 22 00:39:55.822220 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:55.822204 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-shh5z" Jul 22 00:39:56.374198 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:56.374174 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-shh5z" Jul 22 00:39:56.468619 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:56.468525 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8a5237e3-29aa-488d-9c8f-cc8f2ff2a265-config-file\") pod \"8a5237e3-29aa-488d-9c8f-cc8f2ff2a265\" (UID: \"8a5237e3-29aa-488d-9c8f-cc8f2ff2a265\") " Jul 22 00:39:56.468619 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:56.468616 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dqkgx\" (UniqueName: \"kubernetes.io/projected/8a5237e3-29aa-488d-9c8f-cc8f2ff2a265-kube-api-access-dqkgx\") pod \"8a5237e3-29aa-488d-9c8f-cc8f2ff2a265\" (UID: \"8a5237e3-29aa-488d-9c8f-cc8f2ff2a265\") " Jul 22 00:39:56.468968 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:56.468943 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a5237e3-29aa-488d-9c8f-cc8f2ff2a265-config-file" (OuterVolumeSpecName: "config-file") pod "8a5237e3-29aa-488d-9c8f-cc8f2ff2a265" (UID: "8a5237e3-29aa-488d-9c8f-cc8f2ff2a265"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 00:39:56.470565 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:56.470548 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a5237e3-29aa-488d-9c8f-cc8f2ff2a265-kube-api-access-dqkgx" (OuterVolumeSpecName: "kube-api-access-dqkgx") pod "8a5237e3-29aa-488d-9c8f-cc8f2ff2a265" (UID: "8a5237e3-29aa-488d-9c8f-cc8f2ff2a265"). InnerVolumeSpecName "kube-api-access-dqkgx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 00:39:56.569948 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:56.569917 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqkgx\" (UniqueName: \"kubernetes.io/projected/8a5237e3-29aa-488d-9c8f-cc8f2ff2a265-kube-api-access-dqkgx\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 00:39:56.569948 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:56.569942 2574 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8a5237e3-29aa-488d-9c8f-cc8f2ff2a265-config-file\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 00:39:57.169029 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:57.168996 2574 generic.go:356] "Generic (PLEG): container finished" podID="8a5237e3-29aa-488d-9c8f-cc8f2ff2a265" containerID="084feb87782f7dd2463aa02fc1a1866009aa167a189de35639aecaaa63dfebf0" exitCode=0 Jul 22 00:39:57.169503 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:57.169047 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-shh5z" Jul 22 00:39:57.169503 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:57.169077 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-shh5z" event={"ID":"8a5237e3-29aa-488d-9c8f-cc8f2ff2a265","Type":"ContainerDied","Data":"084feb87782f7dd2463aa02fc1a1866009aa167a189de35639aecaaa63dfebf0"} Jul 22 00:39:57.169503 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:57.169139 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-shh5z" event={"ID":"8a5237e3-29aa-488d-9c8f-cc8f2ff2a265","Type":"ContainerDied","Data":"d7079e5fd3a6c6fc510a1383754a2be4ff7766a22991abdd979d4ecc8095a129"} Jul 22 00:39:57.169503 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:57.169188 2574 scope.go:117] "RemoveContainer" containerID="084feb87782f7dd2463aa02fc1a1866009aa167a189de35639aecaaa63dfebf0" Jul 22 00:39:57.177652 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:57.177636 2574 scope.go:117] "RemoveContainer" containerID="084feb87782f7dd2463aa02fc1a1866009aa167a189de35639aecaaa63dfebf0" Jul 22 00:39:57.177900 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:39:57.177882 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"084feb87782f7dd2463aa02fc1a1866009aa167a189de35639aecaaa63dfebf0\": container with ID starting with 084feb87782f7dd2463aa02fc1a1866009aa167a189de35639aecaaa63dfebf0 not found: ID does not exist" containerID="084feb87782f7dd2463aa02fc1a1866009aa167a189de35639aecaaa63dfebf0" Jul 22 00:39:57.177950 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:57.177909 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"084feb87782f7dd2463aa02fc1a1866009aa167a189de35639aecaaa63dfebf0"} err="failed to get container status \"084feb87782f7dd2463aa02fc1a1866009aa167a189de35639aecaaa63dfebf0\": rpc error: code = NotFound desc = could not find container \"084feb87782f7dd2463aa02fc1a1866009aa167a189de35639aecaaa63dfebf0\": container with ID starting with 084feb87782f7dd2463aa02fc1a1866009aa167a189de35639aecaaa63dfebf0 not found: ID does not exist" Jul 22 00:39:57.184774 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:57.184751 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-shh5z"] Jul 22 00:39:57.188428 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:57.188407 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-shh5z"] Jul 22 00:39:59.089473 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:59.089376 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a5237e3-29aa-488d-9c8f-cc8f2ff2a265" path="/var/lib/kubelet/pods/8a5237e3-29aa-488d-9c8f-cc8f2ff2a265/volumes" Jul 22 00:39:59.540373 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:59.540340 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-tk6gw"] Jul 22 00:39:59.540838 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:59.540818 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a5237e3-29aa-488d-9c8f-cc8f2ff2a265" containerName="limitador" Jul 22 00:39:59.540838 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:59.540837 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a5237e3-29aa-488d-9c8f-cc8f2ff2a265" containerName="limitador" Jul 22 00:39:59.540999 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:59.540936 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a5237e3-29aa-488d-9c8f-cc8f2ff2a265" containerName="limitador" Jul 22 00:39:59.545351 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:59.545331 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-tk6gw" Jul 22 00:39:59.547804 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:59.547780 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 22 00:39:59.548848 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:59.548826 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-6hvqs\"" Jul 22 00:39:59.549147 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:59.548920 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 22 00:39:59.549281 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:59.549264 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 00:39:59.550570 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:59.550540 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-tk6gw"] Jul 22 00:39:59.694701 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:59.694670 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb42p\" (UniqueName: \"kubernetes.io/projected/c8fa4ab2-d239-432a-97c9-cd4eb1281cd0-kube-api-access-vb42p\") pod \"postgres-69fb4594cf-tk6gw\" (UID: \"c8fa4ab2-d239-432a-97c9-cd4eb1281cd0\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-tk6gw" Jul 22 00:39:59.694866 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:59.694714 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c8fa4ab2-d239-432a-97c9-cd4eb1281cd0-data\") pod \"postgres-69fb4594cf-tk6gw\" (UID: \"c8fa4ab2-d239-432a-97c9-cd4eb1281cd0\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-tk6gw" Jul 22 00:39:59.795640 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:59.795558 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb42p\" (UniqueName: \"kubernetes.io/projected/c8fa4ab2-d239-432a-97c9-cd4eb1281cd0-kube-api-access-vb42p\") pod \"postgres-69fb4594cf-tk6gw\" (UID: \"c8fa4ab2-d239-432a-97c9-cd4eb1281cd0\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-tk6gw" Jul 22 00:39:59.795640 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:59.795624 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c8fa4ab2-d239-432a-97c9-cd4eb1281cd0-data\") pod \"postgres-69fb4594cf-tk6gw\" (UID: \"c8fa4ab2-d239-432a-97c9-cd4eb1281cd0\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-tk6gw" Jul 22 00:39:59.795955 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:59.795939 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c8fa4ab2-d239-432a-97c9-cd4eb1281cd0-data\") pod \"postgres-69fb4594cf-tk6gw\" (UID: \"c8fa4ab2-d239-432a-97c9-cd4eb1281cd0\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-tk6gw" Jul 22 00:39:59.803853 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:59.803830 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb42p\" (UniqueName: \"kubernetes.io/projected/c8fa4ab2-d239-432a-97c9-cd4eb1281cd0-kube-api-access-vb42p\") pod \"postgres-69fb4594cf-tk6gw\" (UID: \"c8fa4ab2-d239-432a-97c9-cd4eb1281cd0\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-tk6gw" Jul 22 00:39:59.857108 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:59.857081 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-tk6gw" Jul 22 00:39:59.984638 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:39:59.984591 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-tk6gw"] Jul 22 00:39:59.987198 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:39:59.987162 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8fa4ab2_d239_432a_97c9_cd4eb1281cd0.slice/crio-6590ce4f698b3bc83a43a488eb992f44e43963ffff4503da9600e4e341ad5d53 WatchSource:0}: Error finding container 6590ce4f698b3bc83a43a488eb992f44e43963ffff4503da9600e4e341ad5d53: Status 404 returned error can't find the container with id 6590ce4f698b3bc83a43a488eb992f44e43963ffff4503da9600e4e341ad5d53 Jul 22 00:40:00.185855 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:40:00.185820 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-tk6gw" event={"ID":"c8fa4ab2-d239-432a-97c9-cd4eb1281cd0","Type":"ContainerStarted","Data":"6590ce4f698b3bc83a43a488eb992f44e43963ffff4503da9600e4e341ad5d53"} Jul 22 00:40:08.233579 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:40:08.233543 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-tk6gw" event={"ID":"c8fa4ab2-d239-432a-97c9-cd4eb1281cd0","Type":"ContainerStarted","Data":"8403caa2b6b57d370aa666245b3e648a28ce17b9517c3baf04d161eafbb70469"} Jul 22 00:40:08.234170 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:40:08.233670 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-tk6gw" Jul 22 00:40:08.249425 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:40:08.249372 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-tk6gw" podStartSLOduration=1.418028694 podStartE2EDuration="9.24935724s" totalImagesPullingTime="7.831328546s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:39:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:40:08.248433362 +0000 UTC m=+793.605191239" watchObservedRunningTime="2026-07-22 00:40:08.24935724 +0000 UTC m=+793.606115135" Jul 22 00:40:14.268120 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:40:14.268086 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-tk6gw" Jul 22 00:40:21.110928 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:40:21.110851 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-d99964cbf-ffj56"] Jul 22 00:40:21.115459 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:40:21.115434 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d99964cbf-ffj56" Jul 22 00:40:21.118013 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:40:21.117988 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 22 00:40:21.118013 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:40:21.118003 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-qrk2k\"" Jul 22 00:40:21.118013 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:40:21.117993 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 22 00:40:21.131717 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:40:21.131694 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-d99964cbf-ffj56"] Jul 22 00:40:21.177195 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:40:21.177158 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9p7fk\" (UniqueName: \"kubernetes.io/projected/0bfda591-30db-4fbd-ba4b-1b0e3d8bc1f2-kube-api-access-9p7fk\") pod \"maas-controller-d99964cbf-ffj56\" (UID: \"0bfda591-30db-4fbd-ba4b-1b0e3d8bc1f2\") " pod="opendatahub/maas-controller-d99964cbf-ffj56" Jul 22 00:40:21.177349 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:40:21.177252 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0bfda591-30db-4fbd-ba4b-1b0e3d8bc1f2-webhook-cert\") pod \"maas-controller-d99964cbf-ffj56\" (UID: \"0bfda591-30db-4fbd-ba4b-1b0e3d8bc1f2\") " pod="opendatahub/maas-controller-d99964cbf-ffj56" Jul 22 00:40:21.277673 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:40:21.277641 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0bfda591-30db-4fbd-ba4b-1b0e3d8bc1f2-webhook-cert\") pod \"maas-controller-d99964cbf-ffj56\" (UID: \"0bfda591-30db-4fbd-ba4b-1b0e3d8bc1f2\") " pod="opendatahub/maas-controller-d99964cbf-ffj56" Jul 22 00:40:21.277857 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:40:21.277714 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9p7fk\" (UniqueName: \"kubernetes.io/projected/0bfda591-30db-4fbd-ba4b-1b0e3d8bc1f2-kube-api-access-9p7fk\") pod \"maas-controller-d99964cbf-ffj56\" (UID: \"0bfda591-30db-4fbd-ba4b-1b0e3d8bc1f2\") " pod="opendatahub/maas-controller-d99964cbf-ffj56" Jul 22 00:40:21.277857 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:40:21.277776 2574 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 22 00:40:21.277857 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:40:21.277839 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bfda591-30db-4fbd-ba4b-1b0e3d8bc1f2-webhook-cert podName:0bfda591-30db-4fbd-ba4b-1b0e3d8bc1f2 nodeName:}" failed. No retries permitted until 2026-07-22 00:40:21.777822196 +0000 UTC m=+807.134580051 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/0bfda591-30db-4fbd-ba4b-1b0e3d8bc1f2-webhook-cert") pod "maas-controller-d99964cbf-ffj56" (UID: "0bfda591-30db-4fbd-ba4b-1b0e3d8bc1f2") : secret "maas-controller-webhook-cert" not found Jul 22 00:40:21.281055 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:40:21.281030 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-84cdf579f4-llbj2"] Jul 22 00:40:21.283302 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:40:21.283282 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84cdf579f4-llbj2" Jul 22 00:40:21.293953 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:40:21.293927 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-84cdf579f4-llbj2"] Jul 22 00:40:21.295705 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:40:21.295681 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9p7fk\" (UniqueName: \"kubernetes.io/projected/0bfda591-30db-4fbd-ba4b-1b0e3d8bc1f2-kube-api-access-9p7fk\") pod \"maas-controller-d99964cbf-ffj56\" (UID: \"0bfda591-30db-4fbd-ba4b-1b0e3d8bc1f2\") " pod="opendatahub/maas-controller-d99964cbf-ffj56" Jul 22 00:40:21.378902 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:40:21.378870 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px7cc\" (UniqueName: \"kubernetes.io/projected/6206e899-dad6-4971-98f1-4c1bcaf0b0cc-kube-api-access-px7cc\") pod \"maas-controller-84cdf579f4-llbj2\" (UID: \"6206e899-dad6-4971-98f1-4c1bcaf0b0cc\") " pod="opendatahub/maas-controller-84cdf579f4-llbj2" Jul 22 00:40:21.379045 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:40:21.378969 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6206e899-dad6-4971-98f1-4c1bcaf0b0cc-webhook-cert\") pod \"maas-controller-84cdf579f4-llbj2\" (UID: \"6206e899-dad6-4971-98f1-4c1bcaf0b0cc\") " pod="opendatahub/maas-controller-84cdf579f4-llbj2" Jul 22 00:40:21.479451 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:40:21.479416 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px7cc\" (UniqueName: \"kubernetes.io/projected/6206e899-dad6-4971-98f1-4c1bcaf0b0cc-kube-api-access-px7cc\") pod \"maas-controller-84cdf579f4-llbj2\" (UID: \"6206e899-dad6-4971-98f1-4c1bcaf0b0cc\") " pod="opendatahub/maas-controller-84cdf579f4-llbj2" Jul 22 00:40:21.479653 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:40:21.479509 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6206e899-dad6-4971-98f1-4c1bcaf0b0cc-webhook-cert\") pod \"maas-controller-84cdf579f4-llbj2\" (UID: \"6206e899-dad6-4971-98f1-4c1bcaf0b0cc\") " pod="opendatahub/maas-controller-84cdf579f4-llbj2" Jul 22 00:40:21.482014 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:40:21.481991 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6206e899-dad6-4971-98f1-4c1bcaf0b0cc-webhook-cert\") pod \"maas-controller-84cdf579f4-llbj2\" (UID: \"6206e899-dad6-4971-98f1-4c1bcaf0b0cc\") " pod="opendatahub/maas-controller-84cdf579f4-llbj2" Jul 22 00:40:21.487546 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:40:21.487518 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-px7cc\" (UniqueName: \"kubernetes.io/projected/6206e899-dad6-4971-98f1-4c1bcaf0b0cc-kube-api-access-px7cc\") pod \"maas-controller-84cdf579f4-llbj2\" (UID: \"6206e899-dad6-4971-98f1-4c1bcaf0b0cc\") " pod="opendatahub/maas-controller-84cdf579f4-llbj2" Jul 22 00:40:21.594360 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:40:21.594326 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84cdf579f4-llbj2" Jul 22 00:40:21.723505 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:40:21.723482 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-84cdf579f4-llbj2"] Jul 22 00:40:21.724996 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:40:21.724970 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6206e899_dad6_4971_98f1_4c1bcaf0b0cc.slice/crio-a3af4478fd71fb424b08bba44862b3ace5618e5461365320d8ff69ad4a2acb68 WatchSource:0}: Error finding container a3af4478fd71fb424b08bba44862b3ace5618e5461365320d8ff69ad4a2acb68: Status 404 returned error can't find the container with id a3af4478fd71fb424b08bba44862b3ace5618e5461365320d8ff69ad4a2acb68 Jul 22 00:40:21.781644 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:40:21.781584 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0bfda591-30db-4fbd-ba4b-1b0e3d8bc1f2-webhook-cert\") pod \"maas-controller-d99964cbf-ffj56\" (UID: \"0bfda591-30db-4fbd-ba4b-1b0e3d8bc1f2\") " pod="opendatahub/maas-controller-d99964cbf-ffj56" Jul 22 00:40:21.783971 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:40:21.783946 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0bfda591-30db-4fbd-ba4b-1b0e3d8bc1f2-webhook-cert\") pod \"maas-controller-d99964cbf-ffj56\" (UID: \"0bfda591-30db-4fbd-ba4b-1b0e3d8bc1f2\") " pod="opendatahub/maas-controller-d99964cbf-ffj56" Jul 22 00:40:22.033260 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:40:22.033173 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d99964cbf-ffj56" Jul 22 00:40:22.162082 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:40:22.162048 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-d99964cbf-ffj56"] Jul 22 00:40:22.164973 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:40:22.164944 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bfda591_30db_4fbd_ba4b_1b0e3d8bc1f2.slice/crio-92cd91e1f40970dcf4f5cbe25cc3a799eda5e62890955610741f6eb005478aa5 WatchSource:0}: Error finding container 92cd91e1f40970dcf4f5cbe25cc3a799eda5e62890955610741f6eb005478aa5: Status 404 returned error can't find the container with id 92cd91e1f40970dcf4f5cbe25cc3a799eda5e62890955610741f6eb005478aa5 Jul 22 00:40:22.308752 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:40:22.308669 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d99964cbf-ffj56" event={"ID":"0bfda591-30db-4fbd-ba4b-1b0e3d8bc1f2","Type":"ContainerStarted","Data":"92cd91e1f40970dcf4f5cbe25cc3a799eda5e62890955610741f6eb005478aa5"} Jul 22 00:40:22.309719 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:40:22.309695 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84cdf579f4-llbj2" event={"ID":"6206e899-dad6-4971-98f1-4c1bcaf0b0cc","Type":"ContainerStarted","Data":"a3af4478fd71fb424b08bba44862b3ace5618e5461365320d8ff69ad4a2acb68"} Jul 22 00:40:25.331109 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:40:25.331066 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d99964cbf-ffj56" event={"ID":"0bfda591-30db-4fbd-ba4b-1b0e3d8bc1f2","Type":"ContainerStarted","Data":"762fa1a2f1f477f7e0d8faf6f03c486d1892968e12df5f02c6acb67ad71bb597"} Jul 22 00:40:25.331571 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:40:25.331187 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-d99964cbf-ffj56" Jul 22 00:40:25.332746 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:40:25.332718 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84cdf579f4-llbj2" event={"ID":"6206e899-dad6-4971-98f1-4c1bcaf0b0cc","Type":"ContainerStarted","Data":"99ff97b54c3d57ba806cf741f4fc8c5e364edc2a5da0c23d073405fa3f0e375c"} Jul 22 00:40:25.332869 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:40:25.332791 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-84cdf579f4-llbj2" Jul 22 00:40:25.348953 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:40:25.348911 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-d99964cbf-ffj56" podStartSLOduration=2.109061556 podStartE2EDuration="4.348897282s" totalImagesPullingTime="2.239835726s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:40:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:40:25.347545847 +0000 UTC m=+810.704303724" watchObservedRunningTime="2026-07-22 00:40:25.348897282 +0000 UTC m=+810.705655212" Jul 22 00:40:25.364880 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:40:25.364832 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-84cdf579f4-llbj2" podStartSLOduration=1.691493919 podStartE2EDuration="4.364815041s" totalImagesPullingTime="2.673321122s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:40:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:40:25.362530962 +0000 UTC m=+810.719288851" watchObservedRunningTime="2026-07-22 00:40:25.364815041 +0000 UTC m=+810.721572940" Jul 22 00:40:36.346461 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:40:36.346431 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-d99964cbf-ffj56" Jul 22 00:40:36.346940 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:40:36.346596 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-84cdf579f4-llbj2" Jul 22 00:40:36.401503 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:40:36.401460 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-d99964cbf-ffj56"] Jul 22 00:40:36.403551 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:40:36.403488 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-d99964cbf-ffj56" podUID="0bfda591-30db-4fbd-ba4b-1b0e3d8bc1f2" containerName="manager" containerID="cri-o://762fa1a2f1f477f7e0d8faf6f03c486d1892968e12df5f02c6acb67ad71bb597" gracePeriod=30 Jul 22 00:40:36.651989 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:40:36.651967 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d99964cbf-ffj56" Jul 22 00:40:36.712701 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:40:36.712670 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0bfda591-30db-4fbd-ba4b-1b0e3d8bc1f2-webhook-cert\") pod \"0bfda591-30db-4fbd-ba4b-1b0e3d8bc1f2\" (UID: \"0bfda591-30db-4fbd-ba4b-1b0e3d8bc1f2\") " Jul 22 00:40:36.712876 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:40:36.712717 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9p7fk\" (UniqueName: \"kubernetes.io/projected/0bfda591-30db-4fbd-ba4b-1b0e3d8bc1f2-kube-api-access-9p7fk\") pod \"0bfda591-30db-4fbd-ba4b-1b0e3d8bc1f2\" (UID: \"0bfda591-30db-4fbd-ba4b-1b0e3d8bc1f2\") " Jul 22 00:40:36.714838 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:40:36.714810 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bfda591-30db-4fbd-ba4b-1b0e3d8bc1f2-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "0bfda591-30db-4fbd-ba4b-1b0e3d8bc1f2" (UID: "0bfda591-30db-4fbd-ba4b-1b0e3d8bc1f2"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 00:40:36.714943 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:40:36.714842 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bfda591-30db-4fbd-ba4b-1b0e3d8bc1f2-kube-api-access-9p7fk" (OuterVolumeSpecName: "kube-api-access-9p7fk") pod "0bfda591-30db-4fbd-ba4b-1b0e3d8bc1f2" (UID: "0bfda591-30db-4fbd-ba4b-1b0e3d8bc1f2"). InnerVolumeSpecName "kube-api-access-9p7fk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 00:40:36.813583 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:40:36.813537 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9p7fk\" (UniqueName: \"kubernetes.io/projected/0bfda591-30db-4fbd-ba4b-1b0e3d8bc1f2-kube-api-access-9p7fk\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 00:40:36.813583 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:40:36.813586 2574 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0bfda591-30db-4fbd-ba4b-1b0e3d8bc1f2-webhook-cert\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 00:40:37.409994 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:40:37.409958 2574 generic.go:356] "Generic (PLEG): container finished" podID="0bfda591-30db-4fbd-ba4b-1b0e3d8bc1f2" containerID="762fa1a2f1f477f7e0d8faf6f03c486d1892968e12df5f02c6acb67ad71bb597" exitCode=0 Jul 22 00:40:37.410467 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:40:37.410004 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d99964cbf-ffj56" Jul 22 00:40:37.410467 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:40:37.410010 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d99964cbf-ffj56" event={"ID":"0bfda591-30db-4fbd-ba4b-1b0e3d8bc1f2","Type":"ContainerDied","Data":"762fa1a2f1f477f7e0d8faf6f03c486d1892968e12df5f02c6acb67ad71bb597"} Jul 22 00:40:37.410467 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:40:37.410074 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d99964cbf-ffj56" event={"ID":"0bfda591-30db-4fbd-ba4b-1b0e3d8bc1f2","Type":"ContainerDied","Data":"92cd91e1f40970dcf4f5cbe25cc3a799eda5e62890955610741f6eb005478aa5"} Jul 22 00:40:37.410467 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:40:37.410104 2574 scope.go:117] "RemoveContainer" containerID="762fa1a2f1f477f7e0d8faf6f03c486d1892968e12df5f02c6acb67ad71bb597" Jul 22 00:40:37.418545 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:40:37.418524 2574 scope.go:117] "RemoveContainer" containerID="762fa1a2f1f477f7e0d8faf6f03c486d1892968e12df5f02c6acb67ad71bb597" Jul 22 00:40:37.418848 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:40:37.418818 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"762fa1a2f1f477f7e0d8faf6f03c486d1892968e12df5f02c6acb67ad71bb597\": container with ID starting with 762fa1a2f1f477f7e0d8faf6f03c486d1892968e12df5f02c6acb67ad71bb597 not found: ID does not exist" containerID="762fa1a2f1f477f7e0d8faf6f03c486d1892968e12df5f02c6acb67ad71bb597" Jul 22 00:40:37.418935 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:40:37.418861 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"762fa1a2f1f477f7e0d8faf6f03c486d1892968e12df5f02c6acb67ad71bb597"} err="failed to get container status \"762fa1a2f1f477f7e0d8faf6f03c486d1892968e12df5f02c6acb67ad71bb597\": rpc error: code = NotFound desc = could not find container \"762fa1a2f1f477f7e0d8faf6f03c486d1892968e12df5f02c6acb67ad71bb597\": container with ID starting with 762fa1a2f1f477f7e0d8faf6f03c486d1892968e12df5f02c6acb67ad71bb597 not found: ID does not exist" Jul 22 00:40:37.430364 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:40:37.430343 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-d99964cbf-ffj56"] Jul 22 00:40:37.433764 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:40:37.433745 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-d99964cbf-ffj56"] Jul 22 00:40:39.093663 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:40:39.093629 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0bfda591-30db-4fbd-ba4b-1b0e3d8bc1f2" path="/var/lib/kubelet/pods/0bfda591-30db-4fbd-ba4b-1b0e3d8bc1f2/volumes" Jul 22 00:40:56.861756 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:40:56.861723 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5664d797bd-rrrgw"] Jul 22 00:40:56.862162 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:40:56.862060 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bfda591-30db-4fbd-ba4b-1b0e3d8bc1f2" containerName="manager" Jul 22 00:40:56.862162 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:40:56.862071 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bfda591-30db-4fbd-ba4b-1b0e3d8bc1f2" containerName="manager" Jul 22 00:40:56.862162 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:40:56.862126 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="0bfda591-30db-4fbd-ba4b-1b0e3d8bc1f2" containerName="manager" Jul 22 00:40:56.866583 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:40:56.866564 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5664d797bd-rrrgw" Jul 22 00:40:56.869262 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:40:56.869240 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 22 00:40:56.869386 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:40:56.869273 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-k22j6\"" Jul 22 00:40:56.874679 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:40:56.874476 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5664d797bd-rrrgw"] Jul 22 00:40:57.002097 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:40:57.002062 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d2e465e-d76a-43f4-b4e1-003d89b0bb53-maas-api-tls\") pod \"maas-api-5664d797bd-rrrgw\" (UID: \"0d2e465e-d76a-43f4-b4e1-003d89b0bb53\") " pod="odh-ai-gateway-infra/maas-api-5664d797bd-rrrgw" Jul 22 00:40:57.002287 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:40:57.002111 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d2e465e-d76a-43f4-b4e1-003d89b0bb53-openshift-service-ca\") pod \"maas-api-5664d797bd-rrrgw\" (UID: \"0d2e465e-d76a-43f4-b4e1-003d89b0bb53\") " pod="odh-ai-gateway-infra/maas-api-5664d797bd-rrrgw" Jul 22 00:40:57.002287 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:40:57.002233 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9k9p\" (UniqueName: \"kubernetes.io/projected/0d2e465e-d76a-43f4-b4e1-003d89b0bb53-kube-api-access-f9k9p\") pod \"maas-api-5664d797bd-rrrgw\" (UID: \"0d2e465e-d76a-43f4-b4e1-003d89b0bb53\") " pod="odh-ai-gateway-infra/maas-api-5664d797bd-rrrgw" Jul 22 00:40:57.103098 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:40:57.103059 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9k9p\" (UniqueName: \"kubernetes.io/projected/0d2e465e-d76a-43f4-b4e1-003d89b0bb53-kube-api-access-f9k9p\") pod \"maas-api-5664d797bd-rrrgw\" (UID: \"0d2e465e-d76a-43f4-b4e1-003d89b0bb53\") " pod="odh-ai-gateway-infra/maas-api-5664d797bd-rrrgw" Jul 22 00:40:57.103257 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:40:57.103128 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d2e465e-d76a-43f4-b4e1-003d89b0bb53-maas-api-tls\") pod \"maas-api-5664d797bd-rrrgw\" (UID: \"0d2e465e-d76a-43f4-b4e1-003d89b0bb53\") " pod="odh-ai-gateway-infra/maas-api-5664d797bd-rrrgw" Jul 22 00:40:57.103257 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:40:57.103167 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d2e465e-d76a-43f4-b4e1-003d89b0bb53-openshift-service-ca\") pod \"maas-api-5664d797bd-rrrgw\" (UID: \"0d2e465e-d76a-43f4-b4e1-003d89b0bb53\") " pod="odh-ai-gateway-infra/maas-api-5664d797bd-rrrgw" Jul 22 00:40:57.103328 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:40:57.103261 2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found Jul 22 00:40:57.103361 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:40:57.103346 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d2e465e-d76a-43f4-b4e1-003d89b0bb53-maas-api-tls podName:0d2e465e-d76a-43f4-b4e1-003d89b0bb53 nodeName:}" failed. No retries permitted until 2026-07-22 00:40:57.603325109 +0000 UTC m=+842.960082979 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0d2e465e-d76a-43f4-b4e1-003d89b0bb53-maas-api-tls") pod "maas-api-5664d797bd-rrrgw" (UID: "0d2e465e-d76a-43f4-b4e1-003d89b0bb53") : secret "maas-api-serving-cert" not found Jul 22 00:40:57.103972 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:40:57.103953 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d2e465e-d76a-43f4-b4e1-003d89b0bb53-openshift-service-ca\") pod \"maas-api-5664d797bd-rrrgw\" (UID: \"0d2e465e-d76a-43f4-b4e1-003d89b0bb53\") " pod="odh-ai-gateway-infra/maas-api-5664d797bd-rrrgw" Jul 22 00:40:57.112037 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:40:57.111979 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9k9p\" (UniqueName: \"kubernetes.io/projected/0d2e465e-d76a-43f4-b4e1-003d89b0bb53-kube-api-access-f9k9p\") pod \"maas-api-5664d797bd-rrrgw\" (UID: \"0d2e465e-d76a-43f4-b4e1-003d89b0bb53\") " pod="odh-ai-gateway-infra/maas-api-5664d797bd-rrrgw" Jul 22 00:40:57.608547 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:40:57.608514 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d2e465e-d76a-43f4-b4e1-003d89b0bb53-maas-api-tls\") pod \"maas-api-5664d797bd-rrrgw\" (UID: \"0d2e465e-d76a-43f4-b4e1-003d89b0bb53\") " pod="odh-ai-gateway-infra/maas-api-5664d797bd-rrrgw" Jul 22 00:40:57.611018 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:40:57.610993 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d2e465e-d76a-43f4-b4e1-003d89b0bb53-maas-api-tls\") pod \"maas-api-5664d797bd-rrrgw\" (UID: \"0d2e465e-d76a-43f4-b4e1-003d89b0bb53\") " pod="odh-ai-gateway-infra/maas-api-5664d797bd-rrrgw" Jul 22 00:40:57.778847 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:40:57.778811 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5664d797bd-rrrgw" Jul 22 00:40:57.926421 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:40:57.926396 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5664d797bd-rrrgw"] Jul 22 00:40:57.928470 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:40:57.928437 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d2e465e_d76a_43f4_b4e1_003d89b0bb53.slice/crio-116a864a43be0d8e5461cce89a3aae5f551b9f1f56f77cb4df36e129009ba21b WatchSource:0}: Error finding container 116a864a43be0d8e5461cce89a3aae5f551b9f1f56f77cb4df36e129009ba21b: Status 404 returned error can't find the container with id 116a864a43be0d8e5461cce89a3aae5f551b9f1f56f77cb4df36e129009ba21b Jul 22 00:40:58.515646 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:40:58.515589 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5664d797bd-rrrgw" event={"ID":"0d2e465e-d76a-43f4-b4e1-003d89b0bb53","Type":"ContainerStarted","Data":"116a864a43be0d8e5461cce89a3aae5f551b9f1f56f77cb4df36e129009ba21b"} Jul 22 00:41:00.526410 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:41:00.526379 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5664d797bd-rrrgw" event={"ID":"0d2e465e-d76a-43f4-b4e1-003d89b0bb53","Type":"ContainerStarted","Data":"a3a35a25585c78db9d4e1533308c81ccf4164925c9ade95e789451c047a7e5ba"} Jul 22 00:41:00.526904 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:41:00.526446 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5664d797bd-rrrgw" Jul 22 00:41:00.543584 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:41:00.543532 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5664d797bd-rrrgw" podStartSLOduration=2.492555265 podStartE2EDuration="4.543487483s" totalImagesPullingTime="2.050932218s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:40:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:41:00.542715614 +0000 UTC m=+845.899473494" watchObservedRunningTime="2026-07-22 00:41:00.543487483 +0000 UTC m=+845.900245363" Jul 22 00:41:06.537213 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:41:06.537171 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5664d797bd-rrrgw" Jul 22 00:41:30.415979 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:41:30.415929 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ph8m4"] Jul 22 00:41:30.420732 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:41:30.420709 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ph8m4" Jul 22 00:41:30.423356 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:41:30.423336 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 22 00:41:30.423468 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:41:30.423448 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 22 00:41:30.424546 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:41:30.424526 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-jsfjh\"" Jul 22 00:41:30.424663 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:41:30.424528 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 22 00:41:30.430104 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:41:30.430085 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ph8m4"] Jul 22 00:41:30.476357 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:41:30.476313 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/034f4343-633a-45ff-9fee-7eccd8cf2e15-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ph8m4\" (UID: \"034f4343-633a-45ff-9fee-7eccd8cf2e15\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ph8m4" Jul 22 00:41:30.476510 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:41:30.476418 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lj7nj\" (UniqueName: \"kubernetes.io/projected/034f4343-633a-45ff-9fee-7eccd8cf2e15-kube-api-access-lj7nj\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ph8m4\" (UID: \"034f4343-633a-45ff-9fee-7eccd8cf2e15\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ph8m4" Jul 22 00:41:30.476510 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:41:30.476486 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/034f4343-633a-45ff-9fee-7eccd8cf2e15-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ph8m4\" (UID: \"034f4343-633a-45ff-9fee-7eccd8cf2e15\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ph8m4" Jul 22 00:41:30.476510 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:41:30.476506 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/034f4343-633a-45ff-9fee-7eccd8cf2e15-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ph8m4\" (UID: \"034f4343-633a-45ff-9fee-7eccd8cf2e15\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ph8m4" Jul 22 00:41:30.476962 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:41:30.476540 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/034f4343-633a-45ff-9fee-7eccd8cf2e15-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ph8m4\" (UID: \"034f4343-633a-45ff-9fee-7eccd8cf2e15\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ph8m4" Jul 22 00:41:30.476962 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:41:30.476567 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/034f4343-633a-45ff-9fee-7eccd8cf2e15-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ph8m4\" (UID: \"034f4343-633a-45ff-9fee-7eccd8cf2e15\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ph8m4" Jul 22 00:41:30.577858 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:41:30.577815 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/034f4343-633a-45ff-9fee-7eccd8cf2e15-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ph8m4\" (UID: \"034f4343-633a-45ff-9fee-7eccd8cf2e15\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ph8m4" Jul 22 00:41:30.578037 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:41:30.577869 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj7nj\" (UniqueName: \"kubernetes.io/projected/034f4343-633a-45ff-9fee-7eccd8cf2e15-kube-api-access-lj7nj\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ph8m4\" (UID: \"034f4343-633a-45ff-9fee-7eccd8cf2e15\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ph8m4" Jul 22 00:41:30.578037 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:41:30.577916 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/034f4343-633a-45ff-9fee-7eccd8cf2e15-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ph8m4\" (UID: \"034f4343-633a-45ff-9fee-7eccd8cf2e15\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ph8m4" Jul 22 00:41:30.578037 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:41:30.577934 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/034f4343-633a-45ff-9fee-7eccd8cf2e15-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ph8m4\" (UID: \"034f4343-633a-45ff-9fee-7eccd8cf2e15\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ph8m4" Jul 22 00:41:30.578037 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:41:30.577956 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/034f4343-633a-45ff-9fee-7eccd8cf2e15-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ph8m4\" (UID: \"034f4343-633a-45ff-9fee-7eccd8cf2e15\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ph8m4" Jul 22 00:41:30.578037 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:41:30.577983 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/034f4343-633a-45ff-9fee-7eccd8cf2e15-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ph8m4\" (UID: \"034f4343-633a-45ff-9fee-7eccd8cf2e15\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ph8m4" Jul 22 00:41:30.578292 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:41:30.578259 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/034f4343-633a-45ff-9fee-7eccd8cf2e15-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ph8m4\" (UID: \"034f4343-633a-45ff-9fee-7eccd8cf2e15\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ph8m4" Jul 22 00:41:30.578344 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:41:30.578287 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/034f4343-633a-45ff-9fee-7eccd8cf2e15-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ph8m4\" (UID: \"034f4343-633a-45ff-9fee-7eccd8cf2e15\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ph8m4" Jul 22 00:41:30.578386 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:41:30.578343 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/034f4343-633a-45ff-9fee-7eccd8cf2e15-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ph8m4\" (UID: \"034f4343-633a-45ff-9fee-7eccd8cf2e15\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ph8m4" Jul 22 00:41:30.580219 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:41:30.580195 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/034f4343-633a-45ff-9fee-7eccd8cf2e15-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ph8m4\" (UID: \"034f4343-633a-45ff-9fee-7eccd8cf2e15\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ph8m4" Jul 22 00:41:30.580387 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:41:30.580371 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/034f4343-633a-45ff-9fee-7eccd8cf2e15-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ph8m4\" (UID: \"034f4343-633a-45ff-9fee-7eccd8cf2e15\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ph8m4" Jul 22 00:41:30.586041 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:41:30.586015 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lj7nj\" (UniqueName: \"kubernetes.io/projected/034f4343-633a-45ff-9fee-7eccd8cf2e15-kube-api-access-lj7nj\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ph8m4\" (UID: \"034f4343-633a-45ff-9fee-7eccd8cf2e15\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ph8m4" Jul 22 00:41:30.731628 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:41:30.731508 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ph8m4" Jul 22 00:41:30.864964 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:41:30.864942 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ph8m4"] Jul 22 00:41:30.867383 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:41:30.867346 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod034f4343_633a_45ff_9fee_7eccd8cf2e15.slice/crio-ee60044282d3b8314dc915944fb06e4305be6bae62f6ee2bec0ef2d1e9a8d9fa WatchSource:0}: Error finding container ee60044282d3b8314dc915944fb06e4305be6bae62f6ee2bec0ef2d1e9a8d9fa: Status 404 returned error can't find the container with id ee60044282d3b8314dc915944fb06e4305be6bae62f6ee2bec0ef2d1e9a8d9fa Jul 22 00:41:31.680705 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:41:31.680661 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ph8m4" event={"ID":"034f4343-633a-45ff-9fee-7eccd8cf2e15","Type":"ContainerStarted","Data":"ee60044282d3b8314dc915944fb06e4305be6bae62f6ee2bec0ef2d1e9a8d9fa"} Jul 22 00:41:32.688173 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:41:32.688138 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ph8m4" event={"ID":"034f4343-633a-45ff-9fee-7eccd8cf2e15","Type":"ContainerStarted","Data":"7576630e6ffb427d8ba16b6a0ff13eb4e5681263558c9155a8c178313f6fef1d"} Jul 22 00:41:32.706281 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:41:32.706228 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ph8m4" podStartSLOduration=1.529049429 podStartE2EDuration="2.706211894s" totalImagesPullingTime="1.177162465s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:41:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:41:32.704385734 +0000 UTC m=+878.061143612" watchObservedRunningTime="2026-07-22 00:41:32.706211894 +0000 UTC m=+878.062969771" Jul 22 00:41:40.732730 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:41:40.732701 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ph8m4" Jul 22 00:41:40.733213 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:41:40.732786 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ph8m4" Jul 22 00:41:40.745212 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:41:40.745189 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ph8m4" Jul 22 00:41:41.744839 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:41:41.744804 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ph8m4" Jul 22 00:41:52.497940 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:41:52.497862 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fng9jf"] Jul 22 00:41:52.501417 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:41:52.501394 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fng9jf" Jul 22 00:41:52.504129 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:41:52.504110 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 22 00:41:52.509865 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:41:52.509843 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fng9jf"] Jul 22 00:41:52.687088 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:41:52.687051 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5eed5fe-1c85-4646-a373-f4fb80d65542-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fng9jf\" (UID: \"d5eed5fe-1c85-4646-a373-f4fb80d65542\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fng9jf" Jul 22 00:41:52.687262 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:41:52.687092 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5eed5fe-1c85-4646-a373-f4fb80d65542-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fng9jf\" (UID: \"d5eed5fe-1c85-4646-a373-f4fb80d65542\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fng9jf" Jul 22 00:41:52.687262 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:41:52.687127 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5eed5fe-1c85-4646-a373-f4fb80d65542-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fng9jf\" (UID: \"d5eed5fe-1c85-4646-a373-f4fb80d65542\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fng9jf" Jul 22 00:41:52.687262 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:41:52.687157 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5eed5fe-1c85-4646-a373-f4fb80d65542-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fng9jf\" (UID: \"d5eed5fe-1c85-4646-a373-f4fb80d65542\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fng9jf" Jul 22 00:41:52.687262 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:41:52.687175 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5bhq\" (UniqueName: \"kubernetes.io/projected/d5eed5fe-1c85-4646-a373-f4fb80d65542-kube-api-access-t5bhq\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fng9jf\" (UID: \"d5eed5fe-1c85-4646-a373-f4fb80d65542\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fng9jf" Jul 22 00:41:52.687262 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:41:52.687209 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5eed5fe-1c85-4646-a373-f4fb80d65542-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fng9jf\" (UID: \"d5eed5fe-1c85-4646-a373-f4fb80d65542\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fng9jf" Jul 22 00:41:52.788279 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:41:52.788191 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5eed5fe-1c85-4646-a373-f4fb80d65542-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fng9jf\" (UID: \"d5eed5fe-1c85-4646-a373-f4fb80d65542\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fng9jf" Jul 22 00:41:52.788279 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:41:52.788232 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5eed5fe-1c85-4646-a373-f4fb80d65542-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fng9jf\" (UID: \"d5eed5fe-1c85-4646-a373-f4fb80d65542\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fng9jf" Jul 22 00:41:52.788279 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:41:52.788262 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5eed5fe-1c85-4646-a373-f4fb80d65542-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fng9jf\" (UID: \"d5eed5fe-1c85-4646-a373-f4fb80d65542\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fng9jf" Jul 22 00:41:52.788558 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:41:52.788290 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5bhq\" (UniqueName: \"kubernetes.io/projected/d5eed5fe-1c85-4646-a373-f4fb80d65542-kube-api-access-t5bhq\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fng9jf\" (UID: \"d5eed5fe-1c85-4646-a373-f4fb80d65542\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fng9jf" Jul 22 00:41:52.788558 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:41:52.788321 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5eed5fe-1c85-4646-a373-f4fb80d65542-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fng9jf\" (UID: \"d5eed5fe-1c85-4646-a373-f4fb80d65542\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fng9jf" Jul 22 00:41:52.788558 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:41:52.788422 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5eed5fe-1c85-4646-a373-f4fb80d65542-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fng9jf\" (UID: \"d5eed5fe-1c85-4646-a373-f4fb80d65542\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fng9jf" Jul 22 00:41:52.788751 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:41:52.788715 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5eed5fe-1c85-4646-a373-f4fb80d65542-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fng9jf\" (UID: \"d5eed5fe-1c85-4646-a373-f4fb80d65542\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fng9jf" Jul 22 00:41:52.788851 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:41:52.788821 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5eed5fe-1c85-4646-a373-f4fb80d65542-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fng9jf\" (UID: \"d5eed5fe-1c85-4646-a373-f4fb80d65542\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fng9jf" Jul 22 00:41:52.789098 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:41:52.789076 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5eed5fe-1c85-4646-a373-f4fb80d65542-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fng9jf\" (UID: \"d5eed5fe-1c85-4646-a373-f4fb80d65542\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fng9jf" Jul 22 00:41:52.790906 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:41:52.790887 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5eed5fe-1c85-4646-a373-f4fb80d65542-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fng9jf\" (UID: \"d5eed5fe-1c85-4646-a373-f4fb80d65542\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fng9jf" Jul 22 00:41:52.791293 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:41:52.791273 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5eed5fe-1c85-4646-a373-f4fb80d65542-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fng9jf\" (UID: \"d5eed5fe-1c85-4646-a373-f4fb80d65542\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fng9jf" Jul 22 00:41:52.796476 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:41:52.796458 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5bhq\" (UniqueName: \"kubernetes.io/projected/d5eed5fe-1c85-4646-a373-f4fb80d65542-kube-api-access-t5bhq\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fng9jf\" (UID: \"d5eed5fe-1c85-4646-a373-f4fb80d65542\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fng9jf" Jul 22 00:41:52.839189 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:41:52.839155 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fng9jf" Jul 22 00:41:52.976018 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:41:52.975989 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fng9jf"] Jul 22 00:41:52.977763 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:41:52.977737 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5eed5fe_1c85_4646_a373_f4fb80d65542.slice/crio-6e550811f2cad3f68da2057353190bc085acf7885b9fcbb694b10707bf1dba91 WatchSource:0}: Error finding container 6e550811f2cad3f68da2057353190bc085acf7885b9fcbb694b10707bf1dba91: Status 404 returned error can't find the container with id 6e550811f2cad3f68da2057353190bc085acf7885b9fcbb694b10707bf1dba91 Jul 22 00:41:53.799149 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:41:53.799111 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fng9jf" event={"ID":"d5eed5fe-1c85-4646-a373-f4fb80d65542","Type":"ContainerStarted","Data":"503465785af5ad279b0390752080c4bb5c60e75711f22026efe17620b70b8aa2"} Jul 22 00:41:53.799525 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:41:53.799168 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fng9jf" event={"ID":"d5eed5fe-1c85-4646-a373-f4fb80d65542","Type":"ContainerStarted","Data":"6e550811f2cad3f68da2057353190bc085acf7885b9fcbb694b10707bf1dba91"} Jul 22 00:41:53.816865 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:41:53.816824 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fng9jf" podStartSLOduration=1.6519460449999999 podStartE2EDuration="1.816810692s" totalImagesPullingTime="164.864647ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:41:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:41:53.815230918 +0000 UTC m=+899.171988807" watchObservedRunningTime="2026-07-22 00:41:53.816810692 +0000 UTC m=+899.173568568" Jul 22 00:41:55.014255 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:41:55.014222 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-hb69v_bf7b370a-c67c-4c3c-b3ba-7533131eace2/console-operator/1.log" Jul 22 00:41:55.014704 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:41:55.014285 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-hb69v_bf7b370a-c67c-4c3c-b3ba-7533131eace2/console-operator/1.log" Jul 22 00:41:55.018331 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:41:55.018313 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hr5fn_947d311a-ec4b-42e7-a0fb-ee5edc0caf8d/ovn-acl-logging/0.log" Jul 22 00:41:55.018448 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:41:55.018337 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hr5fn_947d311a-ec4b-42e7-a0fb-ee5edc0caf8d/ovn-acl-logging/0.log" Jul 22 00:42:02.839343 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:42:02.839311 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fng9jf" Jul 22 00:42:02.842022 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:42:02.839369 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fng9jf" Jul 22 00:42:02.852192 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:42:02.852167 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fng9jf" Jul 22 00:42:03.860704 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:42:03.860672 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fng9jf" Jul 22 00:44:53.587905 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:44:53.587817 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-84cdf579f4-llbj2"] Jul 22 00:44:53.588437 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:44:53.588110 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-84cdf579f4-llbj2" podUID="6206e899-dad6-4971-98f1-4c1bcaf0b0cc" containerName="manager" containerID="cri-o://99ff97b54c3d57ba806cf741f4fc8c5e364edc2a5da0c23d073405fa3f0e375c" gracePeriod=30 Jul 22 00:44:53.750904 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:44:53.750872 2574 generic.go:356] "Generic (PLEG): container finished" podID="6206e899-dad6-4971-98f1-4c1bcaf0b0cc" containerID="99ff97b54c3d57ba806cf741f4fc8c5e364edc2a5da0c23d073405fa3f0e375c" exitCode=0 Jul 22 00:44:53.751062 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:44:53.750949 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84cdf579f4-llbj2" event={"ID":"6206e899-dad6-4971-98f1-4c1bcaf0b0cc","Type":"ContainerDied","Data":"99ff97b54c3d57ba806cf741f4fc8c5e364edc2a5da0c23d073405fa3f0e375c"} Jul 22 00:44:53.842868 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:44:53.842801 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84cdf579f4-llbj2" Jul 22 00:44:53.951445 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:44:53.951413 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-px7cc\" (UniqueName: \"kubernetes.io/projected/6206e899-dad6-4971-98f1-4c1bcaf0b0cc-kube-api-access-px7cc\") pod \"6206e899-dad6-4971-98f1-4c1bcaf0b0cc\" (UID: \"6206e899-dad6-4971-98f1-4c1bcaf0b0cc\") " Jul 22 00:44:53.951656 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:44:53.951509 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6206e899-dad6-4971-98f1-4c1bcaf0b0cc-webhook-cert\") pod \"6206e899-dad6-4971-98f1-4c1bcaf0b0cc\" (UID: \"6206e899-dad6-4971-98f1-4c1bcaf0b0cc\") " Jul 22 00:44:53.953477 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:44:53.953456 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6206e899-dad6-4971-98f1-4c1bcaf0b0cc-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "6206e899-dad6-4971-98f1-4c1bcaf0b0cc" (UID: "6206e899-dad6-4971-98f1-4c1bcaf0b0cc"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 00:44:53.953612 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:44:53.953577 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6206e899-dad6-4971-98f1-4c1bcaf0b0cc-kube-api-access-px7cc" (OuterVolumeSpecName: "kube-api-access-px7cc") pod "6206e899-dad6-4971-98f1-4c1bcaf0b0cc" (UID: "6206e899-dad6-4971-98f1-4c1bcaf0b0cc"). InnerVolumeSpecName "kube-api-access-px7cc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 00:44:54.052363 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:44:54.052327 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-px7cc\" (UniqueName: \"kubernetes.io/projected/6206e899-dad6-4971-98f1-4c1bcaf0b0cc-kube-api-access-px7cc\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 00:44:54.052363 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:44:54.052355 2574 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6206e899-dad6-4971-98f1-4c1bcaf0b0cc-webhook-cert\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 00:44:54.757694 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:44:54.757664 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84cdf579f4-llbj2" Jul 22 00:44:54.757694 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:44:54.757689 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84cdf579f4-llbj2" event={"ID":"6206e899-dad6-4971-98f1-4c1bcaf0b0cc","Type":"ContainerDied","Data":"a3af4478fd71fb424b08bba44862b3ace5618e5461365320d8ff69ad4a2acb68"} Jul 22 00:44:54.758203 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:44:54.757744 2574 scope.go:117] "RemoveContainer" containerID="99ff97b54c3d57ba806cf741f4fc8c5e364edc2a5da0c23d073405fa3f0e375c" Jul 22 00:44:54.779540 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:44:54.779517 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-84cdf579f4-llbj2"] Jul 22 00:44:54.782923 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:44:54.782902 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-84cdf579f4-llbj2"] Jul 22 00:44:55.089701 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:44:55.089672 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6206e899-dad6-4971-98f1-4c1bcaf0b0cc" path="/var/lib/kubelet/pods/6206e899-dad6-4971-98f1-4c1bcaf0b0cc/volumes" Jul 22 00:45:00.139618 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:45:00.139562 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29744685-59rzf"] Jul 22 00:45:00.140064 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:45:00.139923 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6206e899-dad6-4971-98f1-4c1bcaf0b0cc" containerName="manager" Jul 22 00:45:00.140064 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:45:00.139935 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="6206e899-dad6-4971-98f1-4c1bcaf0b0cc" containerName="manager" Jul 22 00:45:00.140064 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:45:00.140004 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="6206e899-dad6-4971-98f1-4c1bcaf0b0cc" containerName="manager" Jul 22 00:45:00.144142 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:45:00.144124 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744685-59rzf" Jul 22 00:45:00.146883 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:45:00.146859 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tmggk\"" Jul 22 00:45:00.150862 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:45:00.150840 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29744685-59rzf"] Jul 22 00:45:00.155378 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:45:00.155362 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744685-59rzf" Jul 22 00:45:00.285550 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:45:00.285520 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29744685-59rzf"] Jul 22 00:45:00.287082 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:45:00.287046 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f978879_8672_49a1_9982_93fee590cc5a.slice/crio-52e23a1f10f0aea88a8aa4ad55b105e74a805a900c2fedb570b1c58278e34d52 WatchSource:0}: Error finding container 52e23a1f10f0aea88a8aa4ad55b105e74a805a900c2fedb570b1c58278e34d52: Status 404 returned error can't find the container with id 52e23a1f10f0aea88a8aa4ad55b105e74a805a900c2fedb570b1c58278e34d52 Jul 22 00:45:00.289037 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:45:00.289015 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 00:45:00.788907 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:45:00.788871 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744685-59rzf" event={"ID":"0f978879-8672-49a1-9982-93fee590cc5a","Type":"ContainerStarted","Data":"52e23a1f10f0aea88a8aa4ad55b105e74a805a900c2fedb570b1c58278e34d52"} Jul 22 00:45:03.805255 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:45:03.805171 2574 generic.go:356] "Generic (PLEG): container finished" podID="0f978879-8672-49a1-9982-93fee590cc5a" containerID="14a03ec3d7e674786103d8bc6e47d0813b3df02f872ca191c76bf0ad8f12980a" exitCode=0 Jul 22 00:45:03.805626 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:45:03.805257 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744685-59rzf" event={"ID":"0f978879-8672-49a1-9982-93fee590cc5a","Type":"ContainerDied","Data":"14a03ec3d7e674786103d8bc6e47d0813b3df02f872ca191c76bf0ad8f12980a"} Jul 22 00:45:04.948329 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:45:04.948307 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744685-59rzf" Jul 22 00:45:05.817012 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:45:05.816978 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744685-59rzf" event={"ID":"0f978879-8672-49a1-9982-93fee590cc5a","Type":"ContainerDied","Data":"52e23a1f10f0aea88a8aa4ad55b105e74a805a900c2fedb570b1c58278e34d52"} Jul 22 00:45:05.817190 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:45:05.817028 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52e23a1f10f0aea88a8aa4ad55b105e74a805a900c2fedb570b1c58278e34d52" Jul 22 00:45:05.817190 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:45:05.816990 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744685-59rzf" Jul 22 00:45:55.574992 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:45:55.574955 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-84cdf579f4-xnjg9"] Jul 22 00:45:55.575512 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:45:55.575383 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f978879-8672-49a1-9982-93fee590cc5a" containerName="cleanup" Jul 22 00:45:55.575512 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:45:55.575403 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f978879-8672-49a1-9982-93fee590cc5a" containerName="cleanup" Jul 22 00:45:55.575512 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:45:55.575495 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f978879-8672-49a1-9982-93fee590cc5a" containerName="cleanup" Jul 22 00:45:55.578607 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:45:55.578580 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84cdf579f4-xnjg9" Jul 22 00:45:55.581260 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:45:55.581237 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 22 00:45:55.582458 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:45:55.582434 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-qrk2k\"" Jul 22 00:45:55.582549 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:45:55.582454 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 22 00:45:55.588668 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:45:55.588621 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-84cdf579f4-xnjg9"] Jul 22 00:45:55.697922 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:45:55.697874 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2k6n\" (UniqueName: \"kubernetes.io/projected/0ddf4a74-ccd8-4579-afd6-7359f595e139-kube-api-access-w2k6n\") pod \"maas-controller-84cdf579f4-xnjg9\" (UID: \"0ddf4a74-ccd8-4579-afd6-7359f595e139\") " pod="opendatahub/maas-controller-84cdf579f4-xnjg9" Jul 22 00:45:55.697922 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:45:55.697919 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0ddf4a74-ccd8-4579-afd6-7359f595e139-webhook-cert\") pod \"maas-controller-84cdf579f4-xnjg9\" (UID: \"0ddf4a74-ccd8-4579-afd6-7359f595e139\") " pod="opendatahub/maas-controller-84cdf579f4-xnjg9" Jul 22 00:45:55.798645 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:45:55.798591 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2k6n\" (UniqueName: \"kubernetes.io/projected/0ddf4a74-ccd8-4579-afd6-7359f595e139-kube-api-access-w2k6n\") pod \"maas-controller-84cdf579f4-xnjg9\" (UID: \"0ddf4a74-ccd8-4579-afd6-7359f595e139\") " pod="opendatahub/maas-controller-84cdf579f4-xnjg9" Jul 22 00:45:55.798820 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:45:55.798669 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0ddf4a74-ccd8-4579-afd6-7359f595e139-webhook-cert\") pod \"maas-controller-84cdf579f4-xnjg9\" (UID: \"0ddf4a74-ccd8-4579-afd6-7359f595e139\") " pod="opendatahub/maas-controller-84cdf579f4-xnjg9" Jul 22 00:45:55.801101 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:45:55.801078 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0ddf4a74-ccd8-4579-afd6-7359f595e139-webhook-cert\") pod \"maas-controller-84cdf579f4-xnjg9\" (UID: \"0ddf4a74-ccd8-4579-afd6-7359f595e139\") " pod="opendatahub/maas-controller-84cdf579f4-xnjg9" Jul 22 00:45:55.806517 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:45:55.806491 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2k6n\" (UniqueName: \"kubernetes.io/projected/0ddf4a74-ccd8-4579-afd6-7359f595e139-kube-api-access-w2k6n\") pod \"maas-controller-84cdf579f4-xnjg9\" (UID: \"0ddf4a74-ccd8-4579-afd6-7359f595e139\") " pod="opendatahub/maas-controller-84cdf579f4-xnjg9" Jul 22 00:45:55.890043 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:45:55.890015 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84cdf579f4-xnjg9" Jul 22 00:45:56.019991 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:45:56.019961 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-84cdf579f4-xnjg9"] Jul 22 00:45:56.021904 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:45:56.021876 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ddf4a74_ccd8_4579_afd6_7359f595e139.slice/crio-92101cc77943eb6df93e9b676cbc951be52508448bd9b0c361342e3689fb52f2 WatchSource:0}: Error finding container 92101cc77943eb6df93e9b676cbc951be52508448bd9b0c361342e3689fb52f2: Status 404 returned error can't find the container with id 92101cc77943eb6df93e9b676cbc951be52508448bd9b0c361342e3689fb52f2 Jul 22 00:45:56.088655 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:45:56.088622 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84cdf579f4-xnjg9" event={"ID":"0ddf4a74-ccd8-4579-afd6-7359f595e139","Type":"ContainerStarted","Data":"92101cc77943eb6df93e9b676cbc951be52508448bd9b0c361342e3689fb52f2"} Jul 22 00:45:57.094536 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:45:57.094501 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84cdf579f4-xnjg9" event={"ID":"0ddf4a74-ccd8-4579-afd6-7359f595e139","Type":"ContainerStarted","Data":"49d91f120e436c77a7fdad9cb37509f9994173c6c5cf09b33e7aa81afb0c062b"} Jul 22 00:45:57.094952 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:45:57.094563 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-84cdf579f4-xnjg9" Jul 22 00:45:57.111056 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:45:57.111010 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-84cdf579f4-xnjg9" podStartSLOduration=1.721455138 podStartE2EDuration="2.11099566s" totalImagesPullingTime="389.540522ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:45:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:45:57.110199276 +0000 UTC m=+1142.466957153" watchObservedRunningTime="2026-07-22 00:45:57.11099566 +0000 UTC m=+1142.467753579" Jul 22 00:46:08.102262 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:46:08.102227 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-84cdf579f4-xnjg9" Jul 22 00:46:12.755944 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:46:12.755897 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-84cdf579f4-xnjg9"] Jul 22 00:46:12.756421 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:46:12.756155 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-84cdf579f4-xnjg9" podUID="0ddf4a74-ccd8-4579-afd6-7359f595e139" containerName="manager" containerID="cri-o://49d91f120e436c77a7fdad9cb37509f9994173c6c5cf09b33e7aa81afb0c062b" gracePeriod=30 Jul 22 00:46:13.002028 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:46:13.001998 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84cdf579f4-xnjg9" Jul 22 00:46:13.046891 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:46:13.046819 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w2k6n\" (UniqueName: \"kubernetes.io/projected/0ddf4a74-ccd8-4579-afd6-7359f595e139-kube-api-access-w2k6n\") pod \"0ddf4a74-ccd8-4579-afd6-7359f595e139\" (UID: \"0ddf4a74-ccd8-4579-afd6-7359f595e139\") " Jul 22 00:46:13.046891 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:46:13.046873 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0ddf4a74-ccd8-4579-afd6-7359f595e139-webhook-cert\") pod \"0ddf4a74-ccd8-4579-afd6-7359f595e139\" (UID: \"0ddf4a74-ccd8-4579-afd6-7359f595e139\") " Jul 22 00:46:13.048944 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:46:13.048916 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ddf4a74-ccd8-4579-afd6-7359f595e139-kube-api-access-w2k6n" (OuterVolumeSpecName: "kube-api-access-w2k6n") pod "0ddf4a74-ccd8-4579-afd6-7359f595e139" (UID: "0ddf4a74-ccd8-4579-afd6-7359f595e139"). InnerVolumeSpecName "kube-api-access-w2k6n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 00:46:13.049056 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:46:13.048950 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ddf4a74-ccd8-4579-afd6-7359f595e139-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "0ddf4a74-ccd8-4579-afd6-7359f595e139" (UID: "0ddf4a74-ccd8-4579-afd6-7359f595e139"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 00:46:13.147583 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:46:13.147544 2574 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0ddf4a74-ccd8-4579-afd6-7359f595e139-webhook-cert\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 00:46:13.147583 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:46:13.147575 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2k6n\" (UniqueName: \"kubernetes.io/projected/0ddf4a74-ccd8-4579-afd6-7359f595e139-kube-api-access-w2k6n\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 00:46:13.181084 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:46:13.181055 2574 generic.go:356] "Generic (PLEG): container finished" podID="0ddf4a74-ccd8-4579-afd6-7359f595e139" containerID="49d91f120e436c77a7fdad9cb37509f9994173c6c5cf09b33e7aa81afb0c062b" exitCode=0 Jul 22 00:46:13.181229 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:46:13.181097 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84cdf579f4-xnjg9" Jul 22 00:46:13.181229 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:46:13.181102 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84cdf579f4-xnjg9" event={"ID":"0ddf4a74-ccd8-4579-afd6-7359f595e139","Type":"ContainerDied","Data":"49d91f120e436c77a7fdad9cb37509f9994173c6c5cf09b33e7aa81afb0c062b"} Jul 22 00:46:13.181229 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:46:13.181143 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84cdf579f4-xnjg9" event={"ID":"0ddf4a74-ccd8-4579-afd6-7359f595e139","Type":"ContainerDied","Data":"92101cc77943eb6df93e9b676cbc951be52508448bd9b0c361342e3689fb52f2"} Jul 22 00:46:13.181229 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:46:13.181165 2574 scope.go:117] "RemoveContainer" containerID="49d91f120e436c77a7fdad9cb37509f9994173c6c5cf09b33e7aa81afb0c062b" Jul 22 00:46:13.189888 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:46:13.189869 2574 scope.go:117] "RemoveContainer" containerID="49d91f120e436c77a7fdad9cb37509f9994173c6c5cf09b33e7aa81afb0c062b" Jul 22 00:46:13.190131 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:46:13.190110 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49d91f120e436c77a7fdad9cb37509f9994173c6c5cf09b33e7aa81afb0c062b\": container with ID starting with 49d91f120e436c77a7fdad9cb37509f9994173c6c5cf09b33e7aa81afb0c062b not found: ID does not exist" containerID="49d91f120e436c77a7fdad9cb37509f9994173c6c5cf09b33e7aa81afb0c062b" Jul 22 00:46:13.190173 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:46:13.190139 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49d91f120e436c77a7fdad9cb37509f9994173c6c5cf09b33e7aa81afb0c062b"} err="failed to get container status \"49d91f120e436c77a7fdad9cb37509f9994173c6c5cf09b33e7aa81afb0c062b\": rpc error: code = NotFound desc = could not find container \"49d91f120e436c77a7fdad9cb37509f9994173c6c5cf09b33e7aa81afb0c062b\": container with ID starting with 49d91f120e436c77a7fdad9cb37509f9994173c6c5cf09b33e7aa81afb0c062b not found: ID does not exist" Jul 22 00:46:13.198178 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:46:13.198158 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-84cdf579f4-xnjg9"] Jul 22 00:46:13.201809 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:46:13.201788 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-84cdf579f4-xnjg9"] Jul 22 00:46:14.011066 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:46:14.011035 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-84cdf579f4-c6prf"] Jul 22 00:46:14.011435 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:46:14.011373 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ddf4a74-ccd8-4579-afd6-7359f595e139" containerName="manager" Jul 22 00:46:14.011435 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:46:14.011383 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ddf4a74-ccd8-4579-afd6-7359f595e139" containerName="manager" Jul 22 00:46:14.011507 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:46:14.011451 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ddf4a74-ccd8-4579-afd6-7359f595e139" containerName="manager" Jul 22 00:46:14.014449 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:46:14.014432 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84cdf579f4-c6prf" Jul 22 00:46:14.018154 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:46:14.018133 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 22 00:46:14.018154 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:46:14.018143 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 22 00:46:14.018315 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:46:14.018141 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-qrk2k\"" Jul 22 00:46:14.022927 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:46:14.022903 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-84cdf579f4-c6prf"] Jul 22 00:46:14.056824 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:46:14.056799 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a5124289-201e-4af1-bc93-4bd7b9a8895f-webhook-cert\") pod \"maas-controller-84cdf579f4-c6prf\" (UID: \"a5124289-201e-4af1-bc93-4bd7b9a8895f\") " pod="opendatahub/maas-controller-84cdf579f4-c6prf" Jul 22 00:46:14.056960 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:46:14.056855 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdxnn\" (UniqueName: \"kubernetes.io/projected/a5124289-201e-4af1-bc93-4bd7b9a8895f-kube-api-access-xdxnn\") pod \"maas-controller-84cdf579f4-c6prf\" (UID: \"a5124289-201e-4af1-bc93-4bd7b9a8895f\") " pod="opendatahub/maas-controller-84cdf579f4-c6prf" Jul 22 00:46:14.158158 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:46:14.158123 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a5124289-201e-4af1-bc93-4bd7b9a8895f-webhook-cert\") pod \"maas-controller-84cdf579f4-c6prf\" (UID: \"a5124289-201e-4af1-bc93-4bd7b9a8895f\") " pod="opendatahub/maas-controller-84cdf579f4-c6prf" Jul 22 00:46:14.158329 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:46:14.158194 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdxnn\" (UniqueName: \"kubernetes.io/projected/a5124289-201e-4af1-bc93-4bd7b9a8895f-kube-api-access-xdxnn\") pod \"maas-controller-84cdf579f4-c6prf\" (UID: \"a5124289-201e-4af1-bc93-4bd7b9a8895f\") " pod="opendatahub/maas-controller-84cdf579f4-c6prf" Jul 22 00:46:14.160588 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:46:14.160564 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a5124289-201e-4af1-bc93-4bd7b9a8895f-webhook-cert\") pod \"maas-controller-84cdf579f4-c6prf\" (UID: \"a5124289-201e-4af1-bc93-4bd7b9a8895f\") " pod="opendatahub/maas-controller-84cdf579f4-c6prf" Jul 22 00:46:14.166078 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:46:14.166053 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdxnn\" (UniqueName: \"kubernetes.io/projected/a5124289-201e-4af1-bc93-4bd7b9a8895f-kube-api-access-xdxnn\") pod \"maas-controller-84cdf579f4-c6prf\" (UID: \"a5124289-201e-4af1-bc93-4bd7b9a8895f\") " pod="opendatahub/maas-controller-84cdf579f4-c6prf" Jul 22 00:46:14.327844 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:46:14.327743 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84cdf579f4-c6prf" Jul 22 00:46:14.459126 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:46:14.459099 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-84cdf579f4-c6prf"] Jul 22 00:46:14.461319 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:46:14.461292 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5124289_201e_4af1_bc93_4bd7b9a8895f.slice/crio-178566c5a8fbc42f5bd3276587f9962374cbb1f1481c69ae55797b5809b2141b WatchSource:0}: Error finding container 178566c5a8fbc42f5bd3276587f9962374cbb1f1481c69ae55797b5809b2141b: Status 404 returned error can't find the container with id 178566c5a8fbc42f5bd3276587f9962374cbb1f1481c69ae55797b5809b2141b Jul 22 00:46:15.089963 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:46:15.089921 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ddf4a74-ccd8-4579-afd6-7359f595e139" path="/var/lib/kubelet/pods/0ddf4a74-ccd8-4579-afd6-7359f595e139/volumes" Jul 22 00:46:15.194193 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:46:15.194161 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84cdf579f4-c6prf" event={"ID":"a5124289-201e-4af1-bc93-4bd7b9a8895f","Type":"ContainerStarted","Data":"178566c5a8fbc42f5bd3276587f9962374cbb1f1481c69ae55797b5809b2141b"} Jul 22 00:46:16.200120 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:46:16.200081 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84cdf579f4-c6prf" event={"ID":"a5124289-201e-4af1-bc93-4bd7b9a8895f","Type":"ContainerStarted","Data":"31d7d160ae01d4a4ddae538a4e2fd6da44651ba7124e3e8aeb4f5c588bc5c200"} Jul 22 00:46:16.200526 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:46:16.200210 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-84cdf579f4-c6prf" Jul 22 00:46:16.217086 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:46:16.217028 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-84cdf579f4-c6prf" podStartSLOduration=2.032043402 podStartE2EDuration="3.217014506s" totalImagesPullingTime="1.184971104s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:46:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:46:16.215383334 +0000 UTC m=+1161.572141225" watchObservedRunningTime="2026-07-22 00:46:16.217014506 +0000 UTC m=+1161.573772382" Jul 22 00:46:27.207731 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:46:27.207695 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-84cdf579f4-c6prf" Jul 22 00:46:55.050220 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:46:55.050186 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-hb69v_bf7b370a-c67c-4c3c-b3ba-7533131eace2/console-operator/1.log" Jul 22 00:46:55.054958 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:46:55.054922 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-hb69v_bf7b370a-c67c-4c3c-b3ba-7533131eace2/console-operator/1.log" Jul 22 00:46:55.058594 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:46:55.058574 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hr5fn_947d311a-ec4b-42e7-a0fb-ee5edc0caf8d/ovn-acl-logging/0.log" Jul 22 00:46:55.060514 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:46:55.060497 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hr5fn_947d311a-ec4b-42e7-a0fb-ee5edc0caf8d/ovn-acl-logging/0.log" Jul 22 00:50:54.087180 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:50:54.087092 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n4tg8"] Jul 22 00:50:54.087792 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:50:54.087330 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n4tg8" podUID="7e69590f-f707-4031-8a59-24a76c3210f9" containerName="manager" containerID="cri-o://2e071eb059aae87ce32912b13a0edb60869cf952d772e95ed14ad76a7582313e" gracePeriod=10 Jul 22 00:50:57.339543 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:50:57.339515 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n4tg8" Jul 22 00:50:57.454986 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:50:57.454886 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fw7jt\" (UniqueName: \"kubernetes.io/projected/7e69590f-f707-4031-8a59-24a76c3210f9-kube-api-access-fw7jt\") pod \"7e69590f-f707-4031-8a59-24a76c3210f9\" (UID: \"7e69590f-f707-4031-8a59-24a76c3210f9\") " Jul 22 00:50:57.454986 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:50:57.454954 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7e69590f-f707-4031-8a59-24a76c3210f9-extensions-socket-volume\") pod \"7e69590f-f707-4031-8a59-24a76c3210f9\" (UID: \"7e69590f-f707-4031-8a59-24a76c3210f9\") " Jul 22 00:50:57.455288 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:50:57.455263 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e69590f-f707-4031-8a59-24a76c3210f9-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "7e69590f-f707-4031-8a59-24a76c3210f9" (UID: "7e69590f-f707-4031-8a59-24a76c3210f9"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 00:50:57.456999 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:50:57.456980 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e69590f-f707-4031-8a59-24a76c3210f9-kube-api-access-fw7jt" (OuterVolumeSpecName: "kube-api-access-fw7jt") pod "7e69590f-f707-4031-8a59-24a76c3210f9" (UID: "7e69590f-f707-4031-8a59-24a76c3210f9"). InnerVolumeSpecName "kube-api-access-fw7jt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 00:50:57.555649 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:50:57.555577 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fw7jt\" (UniqueName: \"kubernetes.io/projected/7e69590f-f707-4031-8a59-24a76c3210f9-kube-api-access-fw7jt\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 00:50:57.555649 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:50:57.555645 2574 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7e69590f-f707-4031-8a59-24a76c3210f9-extensions-socket-volume\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 00:50:57.696923 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:50:57.696887 2574 generic.go:356] "Generic (PLEG): container finished" podID="7e69590f-f707-4031-8a59-24a76c3210f9" containerID="2e071eb059aae87ce32912b13a0edb60869cf952d772e95ed14ad76a7582313e" exitCode=0 Jul 22 00:50:57.697096 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:50:57.696931 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n4tg8" event={"ID":"7e69590f-f707-4031-8a59-24a76c3210f9","Type":"ContainerDied","Data":"2e071eb059aae87ce32912b13a0edb60869cf952d772e95ed14ad76a7582313e"} Jul 22 00:50:57.697096 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:50:57.696983 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n4tg8" event={"ID":"7e69590f-f707-4031-8a59-24a76c3210f9","Type":"ContainerDied","Data":"e42fa7f87b3e1d75ad1944f9d00b3c6701d0d35a285a10c4115e4a937524c86a"} Jul 22 00:50:57.697096 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:50:57.697007 2574 scope.go:117] "RemoveContainer" containerID="2e071eb059aae87ce32912b13a0edb60869cf952d772e95ed14ad76a7582313e" Jul 22 00:50:57.697096 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:50:57.696938 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n4tg8" Jul 22 00:50:57.709654 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:50:57.709633 2574 scope.go:117] "RemoveContainer" containerID="2e071eb059aae87ce32912b13a0edb60869cf952d772e95ed14ad76a7582313e" Jul 22 00:50:57.709938 ip-10-0-139-192 kubenswrapper[2574]: E0722 00:50:57.709917 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e071eb059aae87ce32912b13a0edb60869cf952d772e95ed14ad76a7582313e\": container with ID starting with 2e071eb059aae87ce32912b13a0edb60869cf952d772e95ed14ad76a7582313e not found: ID does not exist" containerID="2e071eb059aae87ce32912b13a0edb60869cf952d772e95ed14ad76a7582313e" Jul 22 00:50:57.710013 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:50:57.709950 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e071eb059aae87ce32912b13a0edb60869cf952d772e95ed14ad76a7582313e"} err="failed to get container status \"2e071eb059aae87ce32912b13a0edb60869cf952d772e95ed14ad76a7582313e\": rpc error: code = NotFound desc = could not find container \"2e071eb059aae87ce32912b13a0edb60869cf952d772e95ed14ad76a7582313e\": container with ID starting with 2e071eb059aae87ce32912b13a0edb60869cf952d772e95ed14ad76a7582313e not found: ID does not exist" Jul 22 00:50:57.718937 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:50:57.718912 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n4tg8"] Jul 22 00:50:57.724627 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:50:57.724580 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n4tg8"] Jul 22 00:50:59.089398 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:50:59.089363 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e69590f-f707-4031-8a59-24a76c3210f9" path="/var/lib/kubelet/pods/7e69590f-f707-4031-8a59-24a76c3210f9/volumes" Jul 22 00:51:55.088418 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:51:55.088390 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-hb69v_bf7b370a-c67c-4c3c-b3ba-7533131eace2/console-operator/1.log" Jul 22 00:51:55.091970 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:51:55.091947 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-hb69v_bf7b370a-c67c-4c3c-b3ba-7533131eace2/console-operator/1.log" Jul 22 00:51:55.092239 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:51:55.092221 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hr5fn_947d311a-ec4b-42e7-a0fb-ee5edc0caf8d/ovn-acl-logging/0.log" Jul 22 00:51:55.095678 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:51:55.095660 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hr5fn_947d311a-ec4b-42e7-a0fb-ee5edc0caf8d/ovn-acl-logging/0.log" Jul 22 00:51:55.294198 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:51:55.294164 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ngvsl"] Jul 22 00:51:55.294551 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:51:55.294539 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e69590f-f707-4031-8a59-24a76c3210f9" containerName="manager" Jul 22 00:51:55.294619 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:51:55.294553 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e69590f-f707-4031-8a59-24a76c3210f9" containerName="manager" Jul 22 00:51:55.294659 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:51:55.294649 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e69590f-f707-4031-8a59-24a76c3210f9" containerName="manager" Jul 22 00:51:55.297951 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:51:55.297935 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ngvsl" Jul 22 00:51:55.300391 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:51:55.300372 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-pr929\"" Jul 22 00:51:55.309286 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:51:55.309259 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ngvsl"] Jul 22 00:51:55.466976 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:51:55.466931 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4ttj\" (UniqueName: \"kubernetes.io/projected/93ed6faf-7f88-4c7a-bc55-f4f710f6e050-kube-api-access-s4ttj\") pod \"kuadrant-operator-controller-manager-86878c769f-ngvsl\" (UID: \"93ed6faf-7f88-4c7a-bc55-f4f710f6e050\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ngvsl" Jul 22 00:51:55.467192 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:51:55.467012 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/93ed6faf-7f88-4c7a-bc55-f4f710f6e050-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-ngvsl\" (UID: \"93ed6faf-7f88-4c7a-bc55-f4f710f6e050\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ngvsl" Jul 22 00:51:55.568058 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:51:55.568019 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/93ed6faf-7f88-4c7a-bc55-f4f710f6e050-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-ngvsl\" (UID: \"93ed6faf-7f88-4c7a-bc55-f4f710f6e050\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ngvsl" Jul 22 00:51:55.568234 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:51:55.568103 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4ttj\" (UniqueName: \"kubernetes.io/projected/93ed6faf-7f88-4c7a-bc55-f4f710f6e050-kube-api-access-s4ttj\") pod \"kuadrant-operator-controller-manager-86878c769f-ngvsl\" (UID: \"93ed6faf-7f88-4c7a-bc55-f4f710f6e050\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ngvsl" Jul 22 00:51:55.568474 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:51:55.568448 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/93ed6faf-7f88-4c7a-bc55-f4f710f6e050-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-ngvsl\" (UID: \"93ed6faf-7f88-4c7a-bc55-f4f710f6e050\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ngvsl" Jul 22 00:51:55.576806 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:51:55.576777 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4ttj\" (UniqueName: \"kubernetes.io/projected/93ed6faf-7f88-4c7a-bc55-f4f710f6e050-kube-api-access-s4ttj\") pod \"kuadrant-operator-controller-manager-86878c769f-ngvsl\" (UID: \"93ed6faf-7f88-4c7a-bc55-f4f710f6e050\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ngvsl" Jul 22 00:51:55.609323 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:51:55.609285 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ngvsl" Jul 22 00:51:55.750730 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:51:55.750703 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ngvsl"] Jul 22 00:51:55.753426 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:51:55.753400 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93ed6faf_7f88_4c7a_bc55_f4f710f6e050.slice/crio-43b6b6630e0402f135a4ed9963a479c33492ad88fc589f20306ca49bd7ea4052 WatchSource:0}: Error finding container 43b6b6630e0402f135a4ed9963a479c33492ad88fc589f20306ca49bd7ea4052: Status 404 returned error can't find the container with id 43b6b6630e0402f135a4ed9963a479c33492ad88fc589f20306ca49bd7ea4052 Jul 22 00:51:55.996241 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:51:55.996198 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ngvsl" event={"ID":"93ed6faf-7f88-4c7a-bc55-f4f710f6e050","Type":"ContainerStarted","Data":"15b2487b53a9a80108aeeb1f4891418e96ac194c8fd75f4e19cd7c69acfe4d79"} Jul 22 00:51:55.996421 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:51:55.996263 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ngvsl" event={"ID":"93ed6faf-7f88-4c7a-bc55-f4f710f6e050","Type":"ContainerStarted","Data":"43b6b6630e0402f135a4ed9963a479c33492ad88fc589f20306ca49bd7ea4052"} Jul 22 00:51:55.996421 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:51:55.996288 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ngvsl" Jul 22 00:51:56.014706 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:51:56.014583 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ngvsl" podStartSLOduration=1.014565977 podStartE2EDuration="1.014565977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:51:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:51:56.013490497 +0000 UTC m=+1501.370248431" watchObservedRunningTime="2026-07-22 00:51:56.014565977 +0000 UTC m=+1501.371323894" Jul 22 00:52:07.003736 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:52:07.003701 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ngvsl" Jul 22 00:56:55.122738 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:56:55.122708 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-hb69v_bf7b370a-c67c-4c3c-b3ba-7533131eace2/console-operator/1.log" Jul 22 00:56:55.126631 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:56:55.126594 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hr5fn_947d311a-ec4b-42e7-a0fb-ee5edc0caf8d/ovn-acl-logging/0.log" Jul 22 00:56:55.126997 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:56:55.126980 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-hb69v_bf7b370a-c67c-4c3c-b3ba-7533131eace2/console-operator/1.log" Jul 22 00:56:55.130896 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:56:55.130877 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hr5fn_947d311a-ec4b-42e7-a0fb-ee5edc0caf8d/ovn-acl-logging/0.log" Jul 22 00:58:47.351891 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:58:47.351833 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-827fc75f-589d446fd9-bcxmg"] Jul 22 00:58:47.355387 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:58:47.355368 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-827fc75f-589d446fd9-bcxmg" Jul 22 00:58:47.358809 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:58:47.358780 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-827fc75f-serving-cert\"" Jul 22 00:58:47.372172 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:58:47.372139 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-827fc75f-589d446fd9-bcxmg"] Jul 22 00:58:47.471482 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:58:47.471432 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/609ef5f2-0b90-4b1f-8bda-ea89a8d01a7d-maas-api-tls\") pod \"maas-api-e2e-ait-827fc75f-589d446fd9-bcxmg\" (UID: \"609ef5f2-0b90-4b1f-8bda-ea89a8d01a7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-827fc75f-589d446fd9-bcxmg" Jul 22 00:58:47.471707 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:58:47.471504 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/609ef5f2-0b90-4b1f-8bda-ea89a8d01a7d-openshift-service-ca\") pod \"maas-api-e2e-ait-827fc75f-589d446fd9-bcxmg\" (UID: \"609ef5f2-0b90-4b1f-8bda-ea89a8d01a7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-827fc75f-589d446fd9-bcxmg" Jul 22 00:58:47.471707 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:58:47.471556 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99q6x\" (UniqueName: \"kubernetes.io/projected/609ef5f2-0b90-4b1f-8bda-ea89a8d01a7d-kube-api-access-99q6x\") pod \"maas-api-e2e-ait-827fc75f-589d446fd9-bcxmg\" (UID: \"609ef5f2-0b90-4b1f-8bda-ea89a8d01a7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-827fc75f-589d446fd9-bcxmg" Jul 22 00:58:47.572897 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:58:47.572862 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/609ef5f2-0b90-4b1f-8bda-ea89a8d01a7d-maas-api-tls\") pod \"maas-api-e2e-ait-827fc75f-589d446fd9-bcxmg\" (UID: \"609ef5f2-0b90-4b1f-8bda-ea89a8d01a7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-827fc75f-589d446fd9-bcxmg" Jul 22 00:58:47.573055 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:58:47.572904 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/609ef5f2-0b90-4b1f-8bda-ea89a8d01a7d-openshift-service-ca\") pod \"maas-api-e2e-ait-827fc75f-589d446fd9-bcxmg\" (UID: \"609ef5f2-0b90-4b1f-8bda-ea89a8d01a7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-827fc75f-589d446fd9-bcxmg" Jul 22 00:58:47.573055 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:58:47.572938 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99q6x\" (UniqueName: \"kubernetes.io/projected/609ef5f2-0b90-4b1f-8bda-ea89a8d01a7d-kube-api-access-99q6x\") pod \"maas-api-e2e-ait-827fc75f-589d446fd9-bcxmg\" (UID: \"609ef5f2-0b90-4b1f-8bda-ea89a8d01a7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-827fc75f-589d446fd9-bcxmg" Jul 22 00:58:47.573611 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:58:47.573577 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/609ef5f2-0b90-4b1f-8bda-ea89a8d01a7d-openshift-service-ca\") pod \"maas-api-e2e-ait-827fc75f-589d446fd9-bcxmg\" (UID: \"609ef5f2-0b90-4b1f-8bda-ea89a8d01a7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-827fc75f-589d446fd9-bcxmg" Jul 22 00:58:47.575694 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:58:47.575672 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/609ef5f2-0b90-4b1f-8bda-ea89a8d01a7d-maas-api-tls\") pod \"maas-api-e2e-ait-827fc75f-589d446fd9-bcxmg\" (UID: \"609ef5f2-0b90-4b1f-8bda-ea89a8d01a7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-827fc75f-589d446fd9-bcxmg" Jul 22 00:58:47.581126 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:58:47.581103 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99q6x\" (UniqueName: \"kubernetes.io/projected/609ef5f2-0b90-4b1f-8bda-ea89a8d01a7d-kube-api-access-99q6x\") pod \"maas-api-e2e-ait-827fc75f-589d446fd9-bcxmg\" (UID: \"609ef5f2-0b90-4b1f-8bda-ea89a8d01a7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-827fc75f-589d446fd9-bcxmg" Jul 22 00:58:47.665030 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:58:47.664984 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-827fc75f-589d446fd9-bcxmg" Jul 22 00:58:47.805254 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:58:47.805092 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-827fc75f-589d446fd9-bcxmg"] Jul 22 00:58:47.807912 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:58:47.807886 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod609ef5f2_0b90_4b1f_8bda_ea89a8d01a7d.slice/crio-94fc12c29c9e47826a7fdbdea0e0e99ee6c1ce2755397141e345f5b3993c24e7 WatchSource:0}: Error finding container 94fc12c29c9e47826a7fdbdea0e0e99ee6c1ce2755397141e345f5b3993c24e7: Status 404 returned error can't find the container with id 94fc12c29c9e47826a7fdbdea0e0e99ee6c1ce2755397141e345f5b3993c24e7 Jul 22 00:58:47.809384 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:58:47.809362 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 00:58:48.191235 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:58:48.191199 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-827fc75f-589d446fd9-bcxmg" event={"ID":"609ef5f2-0b90-4b1f-8bda-ea89a8d01a7d","Type":"ContainerStarted","Data":"94fc12c29c9e47826a7fdbdea0e0e99ee6c1ce2755397141e345f5b3993c24e7"} Jul 22 00:58:49.197415 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:58:49.197381 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-827fc75f-589d446fd9-bcxmg" event={"ID":"609ef5f2-0b90-4b1f-8bda-ea89a8d01a7d","Type":"ContainerStarted","Data":"07c6305f12a29aa06015c8996f1b4f6764fc021c10187193b7a73bd0820d0408"} Jul 22 00:58:49.197932 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:58:49.197473 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-827fc75f-589d446fd9-bcxmg" Jul 22 00:58:49.216896 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:58:49.216846 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-827fc75f-589d446fd9-bcxmg" podStartSLOduration=1.742015357 podStartE2EDuration="2.216831834s" totalImagesPullingTime="474.816477ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:58:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:58:49.215002456 +0000 UTC m=+1914.571760333" watchObservedRunningTime="2026-07-22 00:58:49.216831834 +0000 UTC m=+1914.573589710" Jul 22 00:58:55.211032 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:58:55.211005 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-827fc75f-589d446fd9-bcxmg" Jul 22 00:59:12.787174 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:59:12.787129 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-827fc75f-w8ctp"] Jul 22 00:59:12.790591 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:59:12.790570 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-827fc75f-w8ctp" Jul 22 00:59:12.793416 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:59:12.793386 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tmggk\"" Jul 22 00:59:12.796882 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:59:12.796825 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-827fc75f-w8ctp"] Jul 22 00:59:12.892087 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:59:12.892042 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5935c4f7-a235-45da-b8a0-3323ff516066-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-827fc75f-w8ctp\" (UID: \"5935c4f7-a235-45da-b8a0-3323ff516066\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-827fc75f-w8ctp" Jul 22 00:59:12.993574 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:59:12.993539 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5935c4f7-a235-45da-b8a0-3323ff516066-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-827fc75f-w8ctp\" (UID: \"5935c4f7-a235-45da-b8a0-3323ff516066\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-827fc75f-w8ctp" Jul 22 00:59:12.994170 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:59:12.994149 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5935c4f7-a235-45da-b8a0-3323ff516066-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-827fc75f-w8ctp\" (UID: \"5935c4f7-a235-45da-b8a0-3323ff516066\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-827fc75f-w8ctp" Jul 22 00:59:13.104312 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:59:13.104231 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-827fc75f-w8ctp" Jul 22 00:59:13.244105 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:59:13.244059 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-827fc75f-w8ctp"] Jul 22 00:59:13.246097 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:59:13.246072 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5935c4f7_a235_45da_b8a0_3323ff516066.slice/crio-2de7b8cc56d93888e8307299d33cee40b10ce04879b72e7626984aec05f2577c WatchSource:0}: Error finding container 2de7b8cc56d93888e8307299d33cee40b10ce04879b72e7626984aec05f2577c: Status 404 returned error can't find the container with id 2de7b8cc56d93888e8307299d33cee40b10ce04879b72e7626984aec05f2577c Jul 22 00:59:13.331555 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:59:13.331520 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-827fc75f-w8ctp" event={"ID":"5935c4f7-a235-45da-b8a0-3323ff516066","Type":"ContainerStarted","Data":"2de7b8cc56d93888e8307299d33cee40b10ce04879b72e7626984aec05f2577c"} Jul 22 00:59:14.337674 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:59:14.337562 2574 generic.go:356] "Generic (PLEG): container finished" podID="5935c4f7-a235-45da-b8a0-3323ff516066" containerID="b8c3dcf762d15fec823fd45bbae3441ed91ec37c4cfb0e0e0d6e548e36557906" exitCode=0 Jul 22 00:59:14.337674 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:59:14.337653 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-827fc75f-w8ctp" event={"ID":"5935c4f7-a235-45da-b8a0-3323ff516066","Type":"ContainerDied","Data":"b8c3dcf762d15fec823fd45bbae3441ed91ec37c4cfb0e0e0d6e548e36557906"} Jul 22 00:59:15.480352 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:59:15.480324 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-827fc75f-w8ctp" Jul 22 00:59:15.619225 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:59:15.619049 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5935c4f7-a235-45da-b8a0-3323ff516066-maas-api-service-ca\") pod \"5935c4f7-a235-45da-b8a0-3323ff516066\" (UID: \"5935c4f7-a235-45da-b8a0-3323ff516066\") " Jul 22 00:59:15.619395 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:59:15.619369 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5935c4f7-a235-45da-b8a0-3323ff516066-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5935c4f7-a235-45da-b8a0-3323ff516066" (UID: "5935c4f7-a235-45da-b8a0-3323ff516066"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 00:59:15.720463 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:59:15.720431 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5935c4f7-a235-45da-b8a0-3323ff516066-maas-api-service-ca\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 00:59:16.350254 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:59:16.350223 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-827fc75f-w8ctp" Jul 22 00:59:16.350437 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:59:16.350260 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-827fc75f-w8ctp" event={"ID":"5935c4f7-a235-45da-b8a0-3323ff516066","Type":"ContainerDied","Data":"2de7b8cc56d93888e8307299d33cee40b10ce04879b72e7626984aec05f2577c"} Jul 22 00:59:16.350437 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:59:16.350306 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2de7b8cc56d93888e8307299d33cee40b10ce04879b72e7626984aec05f2577c" Jul 22 00:59:23.156761 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:59:23.156717 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-827fc75f-589d446fd9-bcxmg"] Jul 22 00:59:23.157387 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:59:23.156966 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-827fc75f-589d446fd9-bcxmg" podUID="609ef5f2-0b90-4b1f-8bda-ea89a8d01a7d" containerName="maas-api" containerID="cri-o://07c6305f12a29aa06015c8996f1b4f6764fc021c10187193b7a73bd0820d0408" gracePeriod=30 Jul 22 00:59:23.388559 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:59:23.388525 2574 generic.go:356] "Generic (PLEG): container finished" podID="609ef5f2-0b90-4b1f-8bda-ea89a8d01a7d" containerID="07c6305f12a29aa06015c8996f1b4f6764fc021c10187193b7a73bd0820d0408" exitCode=0 Jul 22 00:59:23.388730 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:59:23.388568 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-827fc75f-589d446fd9-bcxmg" event={"ID":"609ef5f2-0b90-4b1f-8bda-ea89a8d01a7d","Type":"ContainerDied","Data":"07c6305f12a29aa06015c8996f1b4f6764fc021c10187193b7a73bd0820d0408"} Jul 22 00:59:23.409119 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:59:23.409052 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-827fc75f-589d446fd9-bcxmg" Jul 22 00:59:23.487916 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:59:23.487881 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-99q6x\" (UniqueName: \"kubernetes.io/projected/609ef5f2-0b90-4b1f-8bda-ea89a8d01a7d-kube-api-access-99q6x\") pod \"609ef5f2-0b90-4b1f-8bda-ea89a8d01a7d\" (UID: \"609ef5f2-0b90-4b1f-8bda-ea89a8d01a7d\") " Jul 22 00:59:23.488104 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:59:23.487940 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/609ef5f2-0b90-4b1f-8bda-ea89a8d01a7d-openshift-service-ca\") pod \"609ef5f2-0b90-4b1f-8bda-ea89a8d01a7d\" (UID: \"609ef5f2-0b90-4b1f-8bda-ea89a8d01a7d\") " Jul 22 00:59:23.488104 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:59:23.487972 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/609ef5f2-0b90-4b1f-8bda-ea89a8d01a7d-maas-api-tls\") pod \"609ef5f2-0b90-4b1f-8bda-ea89a8d01a7d\" (UID: \"609ef5f2-0b90-4b1f-8bda-ea89a8d01a7d\") " Jul 22 00:59:23.488340 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:59:23.488316 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/609ef5f2-0b90-4b1f-8bda-ea89a8d01a7d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "609ef5f2-0b90-4b1f-8bda-ea89a8d01a7d" (UID: "609ef5f2-0b90-4b1f-8bda-ea89a8d01a7d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 00:59:23.490204 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:59:23.490171 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/609ef5f2-0b90-4b1f-8bda-ea89a8d01a7d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "609ef5f2-0b90-4b1f-8bda-ea89a8d01a7d" (UID: "609ef5f2-0b90-4b1f-8bda-ea89a8d01a7d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 00:59:23.490300 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:59:23.490206 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/609ef5f2-0b90-4b1f-8bda-ea89a8d01a7d-kube-api-access-99q6x" (OuterVolumeSpecName: "kube-api-access-99q6x") pod "609ef5f2-0b90-4b1f-8bda-ea89a8d01a7d" (UID: "609ef5f2-0b90-4b1f-8bda-ea89a8d01a7d"). InnerVolumeSpecName "kube-api-access-99q6x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 00:59:23.589158 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:59:23.589121 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/609ef5f2-0b90-4b1f-8bda-ea89a8d01a7d-openshift-service-ca\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 00:59:23.589158 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:59:23.589149 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/609ef5f2-0b90-4b1f-8bda-ea89a8d01a7d-maas-api-tls\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 00:59:23.589158 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:59:23.589158 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99q6x\" (UniqueName: \"kubernetes.io/projected/609ef5f2-0b90-4b1f-8bda-ea89a8d01a7d-kube-api-access-99q6x\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 00:59:24.398444 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:59:24.398413 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-827fc75f-589d446fd9-bcxmg" Jul 22 00:59:24.398947 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:59:24.398448 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-827fc75f-589d446fd9-bcxmg" event={"ID":"609ef5f2-0b90-4b1f-8bda-ea89a8d01a7d","Type":"ContainerDied","Data":"94fc12c29c9e47826a7fdbdea0e0e99ee6c1ce2755397141e345f5b3993c24e7"} Jul 22 00:59:24.398947 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:59:24.398512 2574 scope.go:117] "RemoveContainer" containerID="07c6305f12a29aa06015c8996f1b4f6764fc021c10187193b7a73bd0820d0408" Jul 22 00:59:24.420499 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:59:24.420470 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-827fc75f-589d446fd9-bcxmg"] Jul 22 00:59:24.422686 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:59:24.422668 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-827fc75f-589d446fd9-bcxmg"] Jul 22 00:59:25.088386 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:59:25.088351 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="609ef5f2-0b90-4b1f-8bda-ea89a8d01a7d" path="/var/lib/kubelet/pods/609ef5f2-0b90-4b1f-8bda-ea89a8d01a7d/volumes" Jul 22 00:59:27.861188 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:59:27.861150 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-827fc75f-w8ctp"] Jul 22 00:59:27.863938 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:59:27.863908 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-827fc75f-w8ctp"] Jul 22 00:59:29.088846 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:59:29.088815 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5935c4f7-a235-45da-b8a0-3323ff516066" path="/var/lib/kubelet/pods/5935c4f7-a235-45da-b8a0-3323ff516066/volumes" Jul 22 00:59:31.629177 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:59:31.629141 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f98ed7b9-76459c7fcb-w6tpp"] Jul 22 00:59:31.629842 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:59:31.629820 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5935c4f7-a235-45da-b8a0-3323ff516066" containerName="revoke-keys" Jul 22 00:59:31.629943 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:59:31.629844 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5935c4f7-a235-45da-b8a0-3323ff516066" containerName="revoke-keys" Jul 22 00:59:31.629943 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:59:31.629874 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="609ef5f2-0b90-4b1f-8bda-ea89a8d01a7d" containerName="maas-api" Jul 22 00:59:31.629943 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:59:31.629883 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="609ef5f2-0b90-4b1f-8bda-ea89a8d01a7d" containerName="maas-api" Jul 22 00:59:31.630096 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:59:31.629980 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="609ef5f2-0b90-4b1f-8bda-ea89a8d01a7d" containerName="maas-api" Jul 22 00:59:31.630096 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:59:31.629994 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="5935c4f7-a235-45da-b8a0-3323ff516066" containerName="revoke-keys" Jul 22 00:59:31.633410 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:59:31.633388 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f98ed7b9-76459c7fcb-w6tpp" Jul 22 00:59:31.636240 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:59:31.636211 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-f98ed7b9-serving-cert\"" Jul 22 00:59:31.645112 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:59:31.645085 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f98ed7b9-76459c7fcb-w6tpp"] Jul 22 00:59:31.762550 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:59:31.762514 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9d4d10d-ed6f-4b0e-abb3-a6403634e6f1-openshift-service-ca\") pod \"maas-api-e2e-migrate-f98ed7b9-76459c7fcb-w6tpp\" (UID: \"a9d4d10d-ed6f-4b0e-abb3-a6403634e6f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f98ed7b9-76459c7fcb-w6tpp" Jul 22 00:59:31.762728 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:59:31.762593 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9d4d10d-ed6f-4b0e-abb3-a6403634e6f1-maas-api-tls\") pod \"maas-api-e2e-migrate-f98ed7b9-76459c7fcb-w6tpp\" (UID: \"a9d4d10d-ed6f-4b0e-abb3-a6403634e6f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f98ed7b9-76459c7fcb-w6tpp" Jul 22 00:59:31.762728 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:59:31.762643 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm225\" (UniqueName: \"kubernetes.io/projected/a9d4d10d-ed6f-4b0e-abb3-a6403634e6f1-kube-api-access-pm225\") pod \"maas-api-e2e-migrate-f98ed7b9-76459c7fcb-w6tpp\" (UID: \"a9d4d10d-ed6f-4b0e-abb3-a6403634e6f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f98ed7b9-76459c7fcb-w6tpp" Jul 22 00:59:31.863927 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:59:31.863892 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9d4d10d-ed6f-4b0e-abb3-a6403634e6f1-openshift-service-ca\") pod \"maas-api-e2e-migrate-f98ed7b9-76459c7fcb-w6tpp\" (UID: \"a9d4d10d-ed6f-4b0e-abb3-a6403634e6f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f98ed7b9-76459c7fcb-w6tpp" Jul 22 00:59:31.864116 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:59:31.863956 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9d4d10d-ed6f-4b0e-abb3-a6403634e6f1-maas-api-tls\") pod \"maas-api-e2e-migrate-f98ed7b9-76459c7fcb-w6tpp\" (UID: \"a9d4d10d-ed6f-4b0e-abb3-a6403634e6f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f98ed7b9-76459c7fcb-w6tpp" Jul 22 00:59:31.864116 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:59:31.863979 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm225\" (UniqueName: \"kubernetes.io/projected/a9d4d10d-ed6f-4b0e-abb3-a6403634e6f1-kube-api-access-pm225\") pod \"maas-api-e2e-migrate-f98ed7b9-76459c7fcb-w6tpp\" (UID: \"a9d4d10d-ed6f-4b0e-abb3-a6403634e6f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f98ed7b9-76459c7fcb-w6tpp" Jul 22 00:59:31.864700 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:59:31.864679 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9d4d10d-ed6f-4b0e-abb3-a6403634e6f1-openshift-service-ca\") pod \"maas-api-e2e-migrate-f98ed7b9-76459c7fcb-w6tpp\" (UID: \"a9d4d10d-ed6f-4b0e-abb3-a6403634e6f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f98ed7b9-76459c7fcb-w6tpp" Jul 22 00:59:31.866540 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:59:31.866516 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9d4d10d-ed6f-4b0e-abb3-a6403634e6f1-maas-api-tls\") pod \"maas-api-e2e-migrate-f98ed7b9-76459c7fcb-w6tpp\" (UID: \"a9d4d10d-ed6f-4b0e-abb3-a6403634e6f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f98ed7b9-76459c7fcb-w6tpp" Jul 22 00:59:31.872278 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:59:31.872255 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm225\" (UniqueName: \"kubernetes.io/projected/a9d4d10d-ed6f-4b0e-abb3-a6403634e6f1-kube-api-access-pm225\") pod \"maas-api-e2e-migrate-f98ed7b9-76459c7fcb-w6tpp\" (UID: \"a9d4d10d-ed6f-4b0e-abb3-a6403634e6f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f98ed7b9-76459c7fcb-w6tpp" Jul 22 00:59:31.946265 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:59:31.946172 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f98ed7b9-76459c7fcb-w6tpp" Jul 22 00:59:32.095102 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:59:32.095078 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f98ed7b9-76459c7fcb-w6tpp"] Jul 22 00:59:32.096863 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:59:32.096835 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9d4d10d_ed6f_4b0e_abb3_a6403634e6f1.slice/crio-56d6d408f13639a7e3cef7815d53deb2a1e7305007a7b5acc1e16424e64553b0 WatchSource:0}: Error finding container 56d6d408f13639a7e3cef7815d53deb2a1e7305007a7b5acc1e16424e64553b0: Status 404 returned error can't find the container with id 56d6d408f13639a7e3cef7815d53deb2a1e7305007a7b5acc1e16424e64553b0 Jul 22 00:59:32.444535 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:59:32.444501 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f98ed7b9-76459c7fcb-w6tpp" event={"ID":"a9d4d10d-ed6f-4b0e-abb3-a6403634e6f1","Type":"ContainerStarted","Data":"56d6d408f13639a7e3cef7815d53deb2a1e7305007a7b5acc1e16424e64553b0"} Jul 22 00:59:33.451668 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:59:33.451621 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f98ed7b9-76459c7fcb-w6tpp" event={"ID":"a9d4d10d-ed6f-4b0e-abb3-a6403634e6f1","Type":"ContainerStarted","Data":"2d9de6299d7f2688bf8b8d300a9191f4cf69b8781ffd58bf01c0e404f60324a8"} Jul 22 00:59:33.452065 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:59:33.451700 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f98ed7b9-76459c7fcb-w6tpp" Jul 22 00:59:33.469442 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:59:33.469392 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f98ed7b9-76459c7fcb-w6tpp" podStartSLOduration=2.162817969 podStartE2EDuration="2.469376771s" totalImagesPullingTime="306.558802ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:59:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:59:33.466959421 +0000 UTC m=+1958.823717317" watchObservedRunningTime="2026-07-22 00:59:33.469376771 +0000 UTC m=+1958.826134648" Jul 22 00:59:39.463512 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:59:39.463484 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f98ed7b9-76459c7fcb-w6tpp" Jul 22 00:59:56.556581 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:59:56.556548 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f98ed7b9-pml88"] Jul 22 00:59:56.561322 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:59:56.561305 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f98ed7b9-pml88" Jul 22 00:59:56.564216 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:59:56.564035 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tmggk\"" Jul 22 00:59:56.567553 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:59:56.567523 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f98ed7b9-pml88"] Jul 22 00:59:56.680981 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:59:56.680944 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06acfc6e-d404-492e-a240-f9839a4b372d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f98ed7b9-pml88\" (UID: \"06acfc6e-d404-492e-a240-f9839a4b372d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f98ed7b9-pml88" Jul 22 00:59:56.782076 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:59:56.782036 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06acfc6e-d404-492e-a240-f9839a4b372d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f98ed7b9-pml88\" (UID: \"06acfc6e-d404-492e-a240-f9839a4b372d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f98ed7b9-pml88" Jul 22 00:59:56.782752 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:59:56.782730 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06acfc6e-d404-492e-a240-f9839a4b372d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f98ed7b9-pml88\" (UID: \"06acfc6e-d404-492e-a240-f9839a4b372d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f98ed7b9-pml88" Jul 22 00:59:56.874156 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:59:56.874073 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f98ed7b9-pml88" Jul 22 00:59:57.007220 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:59:57.007139 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f98ed7b9-pml88"] Jul 22 00:59:57.009637 ip-10-0-139-192 kubenswrapper[2574]: W0722 00:59:57.009589 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06acfc6e_d404_492e_a240_f9839a4b372d.slice/crio-a308bff9cc236f9bb04e0a0ee94a65298dda641c483222ded554f1457b40dc36 WatchSource:0}: Error finding container a308bff9cc236f9bb04e0a0ee94a65298dda641c483222ded554f1457b40dc36: Status 404 returned error can't find the container with id a308bff9cc236f9bb04e0a0ee94a65298dda641c483222ded554f1457b40dc36 Jul 22 00:59:57.584931 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:59:57.584845 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f98ed7b9-pml88" event={"ID":"06acfc6e-d404-492e-a240-f9839a4b372d","Type":"ContainerStarted","Data":"53c1ebae67f5243c1bbf7df90d2536203655314698a39d5b68a5bdc4c27ea2a0"} Jul 22 00:59:57.584931 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:59:57.584903 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f98ed7b9-pml88" event={"ID":"06acfc6e-d404-492e-a240-f9839a4b372d","Type":"ContainerStarted","Data":"a308bff9cc236f9bb04e0a0ee94a65298dda641c483222ded554f1457b40dc36"} Jul 22 00:59:57.601257 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:59:57.601207 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f98ed7b9-pml88" podStartSLOduration=1.60119309 podStartE2EDuration="1.60119309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 00:59:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 00:59:57.60008397 +0000 UTC m=+1982.956841846" watchObservedRunningTime="2026-07-22 00:59:57.60119309 +0000 UTC m=+1982.957950967" Jul 22 00:59:58.591454 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:59:58.591413 2574 generic.go:356] "Generic (PLEG): container finished" podID="06acfc6e-d404-492e-a240-f9839a4b372d" containerID="53c1ebae67f5243c1bbf7df90d2536203655314698a39d5b68a5bdc4c27ea2a0" exitCode=0 Jul 22 00:59:58.592051 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:59:58.591505 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f98ed7b9-pml88" event={"ID":"06acfc6e-d404-492e-a240-f9839a4b372d","Type":"ContainerDied","Data":"53c1ebae67f5243c1bbf7df90d2536203655314698a39d5b68a5bdc4c27ea2a0"} Jul 22 00:59:59.728771 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:59:59.728748 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f98ed7b9-pml88" Jul 22 00:59:59.807215 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:59:59.807185 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06acfc6e-d404-492e-a240-f9839a4b372d-maas-api-service-ca\") pod \"06acfc6e-d404-492e-a240-f9839a4b372d\" (UID: \"06acfc6e-d404-492e-a240-f9839a4b372d\") " Jul 22 00:59:59.807556 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:59:59.807525 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06acfc6e-d404-492e-a240-f9839a4b372d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "06acfc6e-d404-492e-a240-f9839a4b372d" (UID: "06acfc6e-d404-492e-a240-f9839a4b372d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 00:59:59.908319 ip-10-0-139-192 kubenswrapper[2574]: I0722 00:59:59.908288 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06acfc6e-d404-492e-a240-f9839a4b372d-maas-api-service-ca\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 01:00:00.140199 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:00.140163 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29744700-7t9f4"] Jul 22 01:00:00.140743 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:00.140725 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06acfc6e-d404-492e-a240-f9839a4b372d" containerName="revoke-keys" Jul 22 01:00:00.140786 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:00.140747 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="06acfc6e-d404-492e-a240-f9839a4b372d" containerName="revoke-keys" Jul 22 01:00:00.140860 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:00.140849 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="06acfc6e-d404-492e-a240-f9839a4b372d" containerName="revoke-keys" Jul 22 01:00:00.144160 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:00.144144 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-f98ed7b9-29744700-gpv5q"] Jul 22 01:00:00.144287 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:00.144270 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744700-7t9f4" Jul 22 01:00:00.147615 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:00.147574 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-f98ed7b9-29744700-gpv5q" Jul 22 01:00:00.150464 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:00.150443 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29744700-7t9f4"] Jul 22 01:00:00.153384 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:00.153357 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-f98ed7b9-29744700-gpv5q"] Jul 22 01:00:00.158406 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:00.158350 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744700-7t9f4" Jul 22 01:00:00.165972 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:00.165951 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-f98ed7b9-29744700-gpv5q" Jul 22 01:00:00.308300 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:00.308274 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29744700-7t9f4"] Jul 22 01:00:00.310440 ip-10-0-139-192 kubenswrapper[2574]: W0722 01:00:00.310411 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c94d0ef_0181_4658_9920_73a1f0c0d3d1.slice/crio-d78e5b5e3f4062d2227a71e42fa376a2be58a227e02a80d2be5b401bdc14492d WatchSource:0}: Error finding container d78e5b5e3f4062d2227a71e42fa376a2be58a227e02a80d2be5b401bdc14492d: Status 404 returned error can't find the container with id d78e5b5e3f4062d2227a71e42fa376a2be58a227e02a80d2be5b401bdc14492d Jul 22 01:00:00.325827 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:00.325804 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-f98ed7b9-29744700-gpv5q"] Jul 22 01:00:00.326833 ip-10-0-139-192 kubenswrapper[2574]: W0722 01:00:00.326797 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a60e76f_ca79_4fe6_ac08_41b43f7a2979.slice/crio-a8092f9b45af9013551494d29e77d60fd8ccd64ad6fe520ee9651e4bac0a7b6b WatchSource:0}: Error finding container a8092f9b45af9013551494d29e77d60fd8ccd64ad6fe520ee9651e4bac0a7b6b: Status 404 returned error can't find the container with id a8092f9b45af9013551494d29e77d60fd8ccd64ad6fe520ee9651e4bac0a7b6b Jul 22 01:00:00.603454 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:00.603413 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744700-7t9f4" event={"ID":"9c94d0ef-0181-4658-9920-73a1f0c0d3d1","Type":"ContainerStarted","Data":"d78e5b5e3f4062d2227a71e42fa376a2be58a227e02a80d2be5b401bdc14492d"} Jul 22 01:00:00.604721 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:00.604687 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-f98ed7b9-29744700-gpv5q" event={"ID":"3a60e76f-ca79-4fe6-ac08-41b43f7a2979","Type":"ContainerStarted","Data":"a8092f9b45af9013551494d29e77d60fd8ccd64ad6fe520ee9651e4bac0a7b6b"} Jul 22 01:00:00.605913 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:00.605885 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f98ed7b9-pml88" event={"ID":"06acfc6e-d404-492e-a240-f9839a4b372d","Type":"ContainerDied","Data":"a308bff9cc236f9bb04e0a0ee94a65298dda641c483222ded554f1457b40dc36"} Jul 22 01:00:00.606014 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:00.605926 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a308bff9cc236f9bb04e0a0ee94a65298dda641c483222ded554f1457b40dc36" Jul 22 01:00:00.606014 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:00.605926 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f98ed7b9-pml88" Jul 22 01:00:01.612737 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:01.612657 2574 generic.go:356] "Generic (PLEG): container finished" podID="3a60e76f-ca79-4fe6-ac08-41b43f7a2979" containerID="ae74c371f06cec8c021387ae75b6b0025ea32a08205f43bee6ecc9f8259f8600" exitCode=0 Jul 22 01:00:01.613131 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:01.612738 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-f98ed7b9-29744700-gpv5q" event={"ID":"3a60e76f-ca79-4fe6-ac08-41b43f7a2979","Type":"ContainerDied","Data":"ae74c371f06cec8c021387ae75b6b0025ea32a08205f43bee6ecc9f8259f8600"} Jul 22 01:00:01.614127 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:01.614104 2574 generic.go:356] "Generic (PLEG): container finished" podID="9c94d0ef-0181-4658-9920-73a1f0c0d3d1" containerID="15bc1e5054d675fd5ee11db1b8fdf44a5b68e7762edcb5689d3bdbca026b6a9a" exitCode=0 Jul 22 01:00:01.614266 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:01.614188 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744700-7t9f4" event={"ID":"9c94d0ef-0181-4658-9920-73a1f0c0d3d1","Type":"ContainerDied","Data":"15bc1e5054d675fd5ee11db1b8fdf44a5b68e7762edcb5689d3bdbca026b6a9a"} Jul 22 01:00:02.769505 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:02.769482 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744700-7t9f4" Jul 22 01:00:02.784221 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:02.784194 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-f98ed7b9-29744700-gpv5q" Jul 22 01:00:03.627581 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:03.627551 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-f98ed7b9-29744700-gpv5q" Jul 22 01:00:03.627771 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:03.627586 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-f98ed7b9-29744700-gpv5q" event={"ID":"3a60e76f-ca79-4fe6-ac08-41b43f7a2979","Type":"ContainerDied","Data":"a8092f9b45af9013551494d29e77d60fd8ccd64ad6fe520ee9651e4bac0a7b6b"} Jul 22 01:00:03.627771 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:03.627658 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a8092f9b45af9013551494d29e77d60fd8ccd64ad6fe520ee9651e4bac0a7b6b" Jul 22 01:00:03.628899 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:03.628868 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744700-7t9f4" Jul 22 01:00:03.629042 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:03.628901 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744700-7t9f4" event={"ID":"9c94d0ef-0181-4658-9920-73a1f0c0d3d1","Type":"ContainerDied","Data":"d78e5b5e3f4062d2227a71e42fa376a2be58a227e02a80d2be5b401bdc14492d"} Jul 22 01:00:03.629042 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:03.628932 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d78e5b5e3f4062d2227a71e42fa376a2be58a227e02a80d2be5b401bdc14492d" Jul 22 01:00:07.002669 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:07.002623 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f98ed7b9-76459c7fcb-w6tpp"] Jul 22 01:00:07.003096 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:07.002909 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f98ed7b9-76459c7fcb-w6tpp" podUID="a9d4d10d-ed6f-4b0e-abb3-a6403634e6f1" containerName="maas-api" containerID="cri-o://2d9de6299d7f2688bf8b8d300a9191f4cf69b8781ffd58bf01c0e404f60324a8" gracePeriod=30 Jul 22 01:00:07.023196 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:07.023171 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-f98ed7b9-29744700-gpv5q"] Jul 22 01:00:07.028475 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:07.028446 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-f98ed7b9-29744700-gpv5q"] Jul 22 01:00:07.088857 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:07.088826 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a60e76f-ca79-4fe6-ac08-41b43f7a2979" path="/var/lib/kubelet/pods/3a60e76f-ca79-4fe6-ac08-41b43f7a2979/volumes" Jul 22 01:00:07.256926 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:07.256862 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f98ed7b9-76459c7fcb-w6tpp" Jul 22 01:00:07.372779 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:07.372741 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9d4d10d-ed6f-4b0e-abb3-a6403634e6f1-openshift-service-ca\") pod \"a9d4d10d-ed6f-4b0e-abb3-a6403634e6f1\" (UID: \"a9d4d10d-ed6f-4b0e-abb3-a6403634e6f1\") " Jul 22 01:00:07.372982 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:07.372795 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9d4d10d-ed6f-4b0e-abb3-a6403634e6f1-maas-api-tls\") pod \"a9d4d10d-ed6f-4b0e-abb3-a6403634e6f1\" (UID: \"a9d4d10d-ed6f-4b0e-abb3-a6403634e6f1\") " Jul 22 01:00:07.372982 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:07.372845 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pm225\" (UniqueName: \"kubernetes.io/projected/a9d4d10d-ed6f-4b0e-abb3-a6403634e6f1-kube-api-access-pm225\") pod \"a9d4d10d-ed6f-4b0e-abb3-a6403634e6f1\" (UID: \"a9d4d10d-ed6f-4b0e-abb3-a6403634e6f1\") " Jul 22 01:00:07.373139 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:07.373111 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9d4d10d-ed6f-4b0e-abb3-a6403634e6f1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a9d4d10d-ed6f-4b0e-abb3-a6403634e6f1" (UID: "a9d4d10d-ed6f-4b0e-abb3-a6403634e6f1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 01:00:07.374962 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:07.374940 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9d4d10d-ed6f-4b0e-abb3-a6403634e6f1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a9d4d10d-ed6f-4b0e-abb3-a6403634e6f1" (UID: "a9d4d10d-ed6f-4b0e-abb3-a6403634e6f1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 01:00:07.375151 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:07.375132 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9d4d10d-ed6f-4b0e-abb3-a6403634e6f1-kube-api-access-pm225" (OuterVolumeSpecName: "kube-api-access-pm225") pod "a9d4d10d-ed6f-4b0e-abb3-a6403634e6f1" (UID: "a9d4d10d-ed6f-4b0e-abb3-a6403634e6f1"). InnerVolumeSpecName "kube-api-access-pm225". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:00:07.473880 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:07.473844 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9d4d10d-ed6f-4b0e-abb3-a6403634e6f1-openshift-service-ca\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 01:00:07.473880 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:07.473874 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9d4d10d-ed6f-4b0e-abb3-a6403634e6f1-maas-api-tls\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 01:00:07.473880 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:07.473884 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pm225\" (UniqueName: \"kubernetes.io/projected/a9d4d10d-ed6f-4b0e-abb3-a6403634e6f1-kube-api-access-pm225\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 01:00:07.657301 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:07.657263 2574 generic.go:356] "Generic (PLEG): container finished" podID="a9d4d10d-ed6f-4b0e-abb3-a6403634e6f1" containerID="2d9de6299d7f2688bf8b8d300a9191f4cf69b8781ffd58bf01c0e404f60324a8" exitCode=0 Jul 22 01:00:07.657477 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:07.657316 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f98ed7b9-76459c7fcb-w6tpp" Jul 22 01:00:07.657477 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:07.657350 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f98ed7b9-76459c7fcb-w6tpp" event={"ID":"a9d4d10d-ed6f-4b0e-abb3-a6403634e6f1","Type":"ContainerDied","Data":"2d9de6299d7f2688bf8b8d300a9191f4cf69b8781ffd58bf01c0e404f60324a8"} Jul 22 01:00:07.657477 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:07.657409 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f98ed7b9-76459c7fcb-w6tpp" event={"ID":"a9d4d10d-ed6f-4b0e-abb3-a6403634e6f1","Type":"ContainerDied","Data":"56d6d408f13639a7e3cef7815d53deb2a1e7305007a7b5acc1e16424e64553b0"} Jul 22 01:00:07.657477 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:07.657433 2574 scope.go:117] "RemoveContainer" containerID="2d9de6299d7f2688bf8b8d300a9191f4cf69b8781ffd58bf01c0e404f60324a8" Jul 22 01:00:07.670293 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:07.670275 2574 scope.go:117] "RemoveContainer" containerID="2d9de6299d7f2688bf8b8d300a9191f4cf69b8781ffd58bf01c0e404f60324a8" Jul 22 01:00:07.670524 ip-10-0-139-192 kubenswrapper[2574]: E0722 01:00:07.670506 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d9de6299d7f2688bf8b8d300a9191f4cf69b8781ffd58bf01c0e404f60324a8\": container with ID starting with 2d9de6299d7f2688bf8b8d300a9191f4cf69b8781ffd58bf01c0e404f60324a8 not found: ID does not exist" containerID="2d9de6299d7f2688bf8b8d300a9191f4cf69b8781ffd58bf01c0e404f60324a8" Jul 22 01:00:07.670570 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:07.670533 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d9de6299d7f2688bf8b8d300a9191f4cf69b8781ffd58bf01c0e404f60324a8"} err="failed to get container status \"2d9de6299d7f2688bf8b8d300a9191f4cf69b8781ffd58bf01c0e404f60324a8\": rpc error: code = NotFound desc = could not find container \"2d9de6299d7f2688bf8b8d300a9191f4cf69b8781ffd58bf01c0e404f60324a8\": container with ID starting with 2d9de6299d7f2688bf8b8d300a9191f4cf69b8781ffd58bf01c0e404f60324a8 not found: ID does not exist" Jul 22 01:00:07.685501 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:07.685478 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f98ed7b9-76459c7fcb-w6tpp"] Jul 22 01:00:07.687774 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:07.687756 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f98ed7b9-76459c7fcb-w6tpp"] Jul 22 01:00:09.088530 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:09.088496 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9d4d10d-ed6f-4b0e-abb3-a6403634e6f1" path="/var/lib/kubelet/pods/a9d4d10d-ed6f-4b0e-abb3-a6403634e6f1/volumes" Jul 22 01:00:11.631625 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:11.631568 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f98ed7b9-pml88"] Jul 22 01:00:11.635438 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:11.635412 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f98ed7b9-pml88"] Jul 22 01:00:13.088841 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:13.088802 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06acfc6e-d404-492e-a240-f9839a4b372d" path="/var/lib/kubelet/pods/06acfc6e-d404-492e-a240-f9839a4b372d/volumes" Jul 22 01:00:14.775232 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:14.775194 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b80956c0-66668c8b46-fzj4x"] Jul 22 01:00:14.775759 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:14.775740 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c94d0ef-0181-4658-9920-73a1f0c0d3d1" containerName="cleanup" Jul 22 01:00:14.775812 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:14.775762 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c94d0ef-0181-4658-9920-73a1f0c0d3d1" containerName="cleanup" Jul 22 01:00:14.775812 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:14.775788 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a60e76f-ca79-4fe6-ac08-41b43f7a2979" containerName="cleanup" Jul 22 01:00:14.775812 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:14.775799 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a60e76f-ca79-4fe6-ac08-41b43f7a2979" containerName="cleanup" Jul 22 01:00:14.775911 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:14.775818 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9d4d10d-ed6f-4b0e-abb3-a6403634e6f1" containerName="maas-api" Jul 22 01:00:14.775911 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:14.775826 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9d4d10d-ed6f-4b0e-abb3-a6403634e6f1" containerName="maas-api" Jul 22 01:00:14.775973 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:14.775931 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a60e76f-ca79-4fe6-ac08-41b43f7a2979" containerName="cleanup" Jul 22 01:00:14.775973 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:14.775946 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c94d0ef-0181-4658-9920-73a1f0c0d3d1" containerName="cleanup" Jul 22 01:00:14.775973 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:14.775957 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9d4d10d-ed6f-4b0e-abb3-a6403634e6f1" containerName="maas-api" Jul 22 01:00:14.779399 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:14.779377 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b80956c0-66668c8b46-fzj4x" Jul 22 01:00:14.782175 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:14.782156 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-b80956c0-serving-cert\"" Jul 22 01:00:14.791664 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:14.791643 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b80956c0-66668c8b46-fzj4x"] Jul 22 01:00:14.941705 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:14.941672 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b68fb0f-05ee-4465-a118-db333a6fe02d-maas-api-tls\") pod \"maas-api-e2e-ait-b80956c0-66668c8b46-fzj4x\" (UID: \"7b68fb0f-05ee-4465-a118-db333a6fe02d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b80956c0-66668c8b46-fzj4x" Jul 22 01:00:14.941705 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:14.941707 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b68fb0f-05ee-4465-a118-db333a6fe02d-openshift-service-ca\") pod \"maas-api-e2e-ait-b80956c0-66668c8b46-fzj4x\" (UID: \"7b68fb0f-05ee-4465-a118-db333a6fe02d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b80956c0-66668c8b46-fzj4x" Jul 22 01:00:14.941957 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:14.941730 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmjfg\" (UniqueName: \"kubernetes.io/projected/7b68fb0f-05ee-4465-a118-db333a6fe02d-kube-api-access-rmjfg\") pod \"maas-api-e2e-ait-b80956c0-66668c8b46-fzj4x\" (UID: \"7b68fb0f-05ee-4465-a118-db333a6fe02d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b80956c0-66668c8b46-fzj4x" Jul 22 01:00:15.042788 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:15.042703 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b68fb0f-05ee-4465-a118-db333a6fe02d-maas-api-tls\") pod \"maas-api-e2e-ait-b80956c0-66668c8b46-fzj4x\" (UID: \"7b68fb0f-05ee-4465-a118-db333a6fe02d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b80956c0-66668c8b46-fzj4x" Jul 22 01:00:15.042788 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:15.042737 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b68fb0f-05ee-4465-a118-db333a6fe02d-openshift-service-ca\") pod \"maas-api-e2e-ait-b80956c0-66668c8b46-fzj4x\" (UID: \"7b68fb0f-05ee-4465-a118-db333a6fe02d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b80956c0-66668c8b46-fzj4x" Jul 22 01:00:15.042788 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:15.042761 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmjfg\" (UniqueName: \"kubernetes.io/projected/7b68fb0f-05ee-4465-a118-db333a6fe02d-kube-api-access-rmjfg\") pod \"maas-api-e2e-ait-b80956c0-66668c8b46-fzj4x\" (UID: \"7b68fb0f-05ee-4465-a118-db333a6fe02d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b80956c0-66668c8b46-fzj4x" Jul 22 01:00:15.043078 ip-10-0-139-192 kubenswrapper[2574]: E0722 01:00:15.042868 2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-b80956c0-serving-cert: secret "maas-api-e2e-ait-b80956c0-serving-cert" not found Jul 22 01:00:15.043078 ip-10-0-139-192 kubenswrapper[2574]: E0722 01:00:15.042958 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b68fb0f-05ee-4465-a118-db333a6fe02d-maas-api-tls podName:7b68fb0f-05ee-4465-a118-db333a6fe02d nodeName:}" failed. No retries permitted until 2026-07-22 01:00:15.542937684 +0000 UTC m=+2000.899695560 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7b68fb0f-05ee-4465-a118-db333a6fe02d-maas-api-tls") pod "maas-api-e2e-ait-b80956c0-66668c8b46-fzj4x" (UID: "7b68fb0f-05ee-4465-a118-db333a6fe02d") : secret "maas-api-e2e-ait-b80956c0-serving-cert" not found Jul 22 01:00:15.043438 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:15.043418 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b68fb0f-05ee-4465-a118-db333a6fe02d-openshift-service-ca\") pod \"maas-api-e2e-ait-b80956c0-66668c8b46-fzj4x\" (UID: \"7b68fb0f-05ee-4465-a118-db333a6fe02d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b80956c0-66668c8b46-fzj4x" Jul 22 01:00:15.052934 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:15.052909 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmjfg\" (UniqueName: \"kubernetes.io/projected/7b68fb0f-05ee-4465-a118-db333a6fe02d-kube-api-access-rmjfg\") pod \"maas-api-e2e-ait-b80956c0-66668c8b46-fzj4x\" (UID: \"7b68fb0f-05ee-4465-a118-db333a6fe02d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b80956c0-66668c8b46-fzj4x" Jul 22 01:00:15.548286 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:15.548245 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b68fb0f-05ee-4465-a118-db333a6fe02d-maas-api-tls\") pod \"maas-api-e2e-ait-b80956c0-66668c8b46-fzj4x\" (UID: \"7b68fb0f-05ee-4465-a118-db333a6fe02d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b80956c0-66668c8b46-fzj4x" Jul 22 01:00:15.550612 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:15.550580 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b68fb0f-05ee-4465-a118-db333a6fe02d-maas-api-tls\") pod \"maas-api-e2e-ait-b80956c0-66668c8b46-fzj4x\" (UID: \"7b68fb0f-05ee-4465-a118-db333a6fe02d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b80956c0-66668c8b46-fzj4x" Jul 22 01:00:15.693931 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:15.693901 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b80956c0-66668c8b46-fzj4x" Jul 22 01:00:15.837360 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:15.837333 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b80956c0-66668c8b46-fzj4x"] Jul 22 01:00:15.838458 ip-10-0-139-192 kubenswrapper[2574]: W0722 01:00:15.838425 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b68fb0f_05ee_4465_a118_db333a6fe02d.slice/crio-0936138159ff7b724acffb8e65bd6f9e113909f9dc39c0c377140bac8bcaaaa3 WatchSource:0}: Error finding container 0936138159ff7b724acffb8e65bd6f9e113909f9dc39c0c377140bac8bcaaaa3: Status 404 returned error can't find the container with id 0936138159ff7b724acffb8e65bd6f9e113909f9dc39c0c377140bac8bcaaaa3 Jul 22 01:00:16.709884 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:16.709842 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b80956c0-66668c8b46-fzj4x" event={"ID":"7b68fb0f-05ee-4465-a118-db333a6fe02d","Type":"ContainerStarted","Data":"0936138159ff7b724acffb8e65bd6f9e113909f9dc39c0c377140bac8bcaaaa3"} Jul 22 01:00:19.731744 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:19.731704 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b80956c0-66668c8b46-fzj4x" event={"ID":"7b68fb0f-05ee-4465-a118-db333a6fe02d","Type":"ContainerStarted","Data":"cf9f36a84d2f60547ad1590135dd5d600ebc7fa8a2592aed6d8701f8f5b16956"} Jul 22 01:00:19.732165 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:19.731838 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b80956c0-66668c8b46-fzj4x" Jul 22 01:00:19.751318 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:19.751263 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b80956c0-66668c8b46-fzj4x" podStartSLOduration=2.637085518 podStartE2EDuration="5.751248874s" totalImagesPullingTime="3.114163356s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:00:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:00:19.750402923 +0000 UTC m=+2005.107160822" watchObservedRunningTime="2026-07-22 01:00:19.751248874 +0000 UTC m=+2005.108006751" Jul 22 01:00:25.743257 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:25.743227 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b80956c0-66668c8b46-fzj4x" Jul 22 01:00:40.494734 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:40.494699 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b80956c0-52989"] Jul 22 01:00:40.498415 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:40.498395 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b80956c0-52989" Jul 22 01:00:40.501220 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:40.501200 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tmggk\"" Jul 22 01:00:40.506982 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:40.506956 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b80956c0-52989"] Jul 22 01:00:40.573913 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:40.573876 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b955cba-575f-4c50-bfb4-8d9baf3cc7f1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b80956c0-52989\" (UID: \"9b955cba-575f-4c50-bfb4-8d9baf3cc7f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b80956c0-52989" Jul 22 01:00:40.675247 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:40.675206 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b955cba-575f-4c50-bfb4-8d9baf3cc7f1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b80956c0-52989\" (UID: \"9b955cba-575f-4c50-bfb4-8d9baf3cc7f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b80956c0-52989" Jul 22 01:00:40.675881 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:40.675862 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b955cba-575f-4c50-bfb4-8d9baf3cc7f1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b80956c0-52989\" (UID: \"9b955cba-575f-4c50-bfb4-8d9baf3cc7f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b80956c0-52989" Jul 22 01:00:40.810622 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:40.810528 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b80956c0-52989" Jul 22 01:00:41.154304 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:41.154270 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b80956c0-52989"] Jul 22 01:00:41.157675 ip-10-0-139-192 kubenswrapper[2574]: W0722 01:00:41.157649 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b955cba_575f_4c50_bfb4_8d9baf3cc7f1.slice/crio-50dce11c49eef823eb633ee6cb9f725a0829ee2772f7d78b6e68495403229494 WatchSource:0}: Error finding container 50dce11c49eef823eb633ee6cb9f725a0829ee2772f7d78b6e68495403229494: Status 404 returned error can't find the container with id 50dce11c49eef823eb633ee6cb9f725a0829ee2772f7d78b6e68495403229494 Jul 22 01:00:41.852523 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:41.852485 2574 generic.go:356] "Generic (PLEG): container finished" podID="9b955cba-575f-4c50-bfb4-8d9baf3cc7f1" containerID="8a3f379ec46ee1ba9f8eae794d4e83b46f42ebff004cf1f9a5f40b61fdf4e03d" exitCode=0 Jul 22 01:00:41.852966 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:41.852570 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b80956c0-52989" event={"ID":"9b955cba-575f-4c50-bfb4-8d9baf3cc7f1","Type":"ContainerDied","Data":"8a3f379ec46ee1ba9f8eae794d4e83b46f42ebff004cf1f9a5f40b61fdf4e03d"} Jul 22 01:00:41.852966 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:41.852659 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b80956c0-52989" event={"ID":"9b955cba-575f-4c50-bfb4-8d9baf3cc7f1","Type":"ContainerStarted","Data":"50dce11c49eef823eb633ee6cb9f725a0829ee2772f7d78b6e68495403229494"} Jul 22 01:00:42.985070 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:42.985044 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b80956c0-52989" Jul 22 01:00:43.096805 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:43.096777 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b955cba-575f-4c50-bfb4-8d9baf3cc7f1-maas-api-service-ca\") pod \"9b955cba-575f-4c50-bfb4-8d9baf3cc7f1\" (UID: \"9b955cba-575f-4c50-bfb4-8d9baf3cc7f1\") " Jul 22 01:00:43.097104 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:43.097084 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b955cba-575f-4c50-bfb4-8d9baf3cc7f1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9b955cba-575f-4c50-bfb4-8d9baf3cc7f1" (UID: "9b955cba-575f-4c50-bfb4-8d9baf3cc7f1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 01:00:43.197589 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:43.197553 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b955cba-575f-4c50-bfb4-8d9baf3cc7f1-maas-api-service-ca\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 01:00:43.864372 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:43.864344 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b80956c0-52989" Jul 22 01:00:43.864573 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:43.864380 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b80956c0-52989" event={"ID":"9b955cba-575f-4c50-bfb4-8d9baf3cc7f1","Type":"ContainerDied","Data":"50dce11c49eef823eb633ee6cb9f725a0829ee2772f7d78b6e68495403229494"} Jul 22 01:00:43.864573 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:43.864429 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50dce11c49eef823eb633ee6cb9f725a0829ee2772f7d78b6e68495403229494" Jul 22 01:00:50.893130 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:50.893083 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b80956c0-66668c8b46-fzj4x"] Jul 22 01:00:50.893638 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:50.893316 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b80956c0-66668c8b46-fzj4x" podUID="7b68fb0f-05ee-4465-a118-db333a6fe02d" containerName="maas-api" containerID="cri-o://cf9f36a84d2f60547ad1590135dd5d600ebc7fa8a2592aed6d8701f8f5b16956" gracePeriod=30 Jul 22 01:00:51.148271 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:51.148217 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b80956c0-66668c8b46-fzj4x" Jul 22 01:00:51.272458 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:51.272428 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rmjfg\" (UniqueName: \"kubernetes.io/projected/7b68fb0f-05ee-4465-a118-db333a6fe02d-kube-api-access-rmjfg\") pod \"7b68fb0f-05ee-4465-a118-db333a6fe02d\" (UID: \"7b68fb0f-05ee-4465-a118-db333a6fe02d\") " Jul 22 01:00:51.272635 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:51.272490 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b68fb0f-05ee-4465-a118-db333a6fe02d-openshift-service-ca\") pod \"7b68fb0f-05ee-4465-a118-db333a6fe02d\" (UID: \"7b68fb0f-05ee-4465-a118-db333a6fe02d\") " Jul 22 01:00:51.272635 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:51.272513 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b68fb0f-05ee-4465-a118-db333a6fe02d-maas-api-tls\") pod \"7b68fb0f-05ee-4465-a118-db333a6fe02d\" (UID: \"7b68fb0f-05ee-4465-a118-db333a6fe02d\") " Jul 22 01:00:51.272896 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:51.272870 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b68fb0f-05ee-4465-a118-db333a6fe02d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7b68fb0f-05ee-4465-a118-db333a6fe02d" (UID: "7b68fb0f-05ee-4465-a118-db333a6fe02d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 01:00:51.274655 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:51.274635 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b68fb0f-05ee-4465-a118-db333a6fe02d-kube-api-access-rmjfg" (OuterVolumeSpecName: "kube-api-access-rmjfg") pod "7b68fb0f-05ee-4465-a118-db333a6fe02d" (UID: "7b68fb0f-05ee-4465-a118-db333a6fe02d"). InnerVolumeSpecName "kube-api-access-rmjfg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:00:51.274835 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:51.274821 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b68fb0f-05ee-4465-a118-db333a6fe02d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7b68fb0f-05ee-4465-a118-db333a6fe02d" (UID: "7b68fb0f-05ee-4465-a118-db333a6fe02d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 01:00:51.373771 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:51.373738 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b68fb0f-05ee-4465-a118-db333a6fe02d-openshift-service-ca\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 01:00:51.373771 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:51.373765 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b68fb0f-05ee-4465-a118-db333a6fe02d-maas-api-tls\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 01:00:51.373771 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:51.373776 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmjfg\" (UniqueName: \"kubernetes.io/projected/7b68fb0f-05ee-4465-a118-db333a6fe02d-kube-api-access-rmjfg\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 01:00:51.915237 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:51.915200 2574 generic.go:356] "Generic (PLEG): container finished" podID="7b68fb0f-05ee-4465-a118-db333a6fe02d" containerID="cf9f36a84d2f60547ad1590135dd5d600ebc7fa8a2592aed6d8701f8f5b16956" exitCode=0 Jul 22 01:00:51.915661 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:51.915250 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b80956c0-66668c8b46-fzj4x" Jul 22 01:00:51.915661 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:51.915286 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b80956c0-66668c8b46-fzj4x" event={"ID":"7b68fb0f-05ee-4465-a118-db333a6fe02d","Type":"ContainerDied","Data":"cf9f36a84d2f60547ad1590135dd5d600ebc7fa8a2592aed6d8701f8f5b16956"} Jul 22 01:00:51.915661 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:51.915345 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b80956c0-66668c8b46-fzj4x" event={"ID":"7b68fb0f-05ee-4465-a118-db333a6fe02d","Type":"ContainerDied","Data":"0936138159ff7b724acffb8e65bd6f9e113909f9dc39c0c377140bac8bcaaaa3"} Jul 22 01:00:51.915661 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:51.915371 2574 scope.go:117] "RemoveContainer" containerID="cf9f36a84d2f60547ad1590135dd5d600ebc7fa8a2592aed6d8701f8f5b16956" Jul 22 01:00:51.925482 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:51.925462 2574 scope.go:117] "RemoveContainer" containerID="cf9f36a84d2f60547ad1590135dd5d600ebc7fa8a2592aed6d8701f8f5b16956" Jul 22 01:00:51.925761 ip-10-0-139-192 kubenswrapper[2574]: E0722 01:00:51.925739 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf9f36a84d2f60547ad1590135dd5d600ebc7fa8a2592aed6d8701f8f5b16956\": container with ID starting with cf9f36a84d2f60547ad1590135dd5d600ebc7fa8a2592aed6d8701f8f5b16956 not found: ID does not exist" containerID="cf9f36a84d2f60547ad1590135dd5d600ebc7fa8a2592aed6d8701f8f5b16956" Jul 22 01:00:51.925836 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:51.925768 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf9f36a84d2f60547ad1590135dd5d600ebc7fa8a2592aed6d8701f8f5b16956"} err="failed to get container status \"cf9f36a84d2f60547ad1590135dd5d600ebc7fa8a2592aed6d8701f8f5b16956\": rpc error: code = NotFound desc = could not find container \"cf9f36a84d2f60547ad1590135dd5d600ebc7fa8a2592aed6d8701f8f5b16956\": container with ID starting with cf9f36a84d2f60547ad1590135dd5d600ebc7fa8a2592aed6d8701f8f5b16956 not found: ID does not exist" Jul 22 01:00:51.938010 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:51.937986 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b80956c0-66668c8b46-fzj4x"] Jul 22 01:00:51.941032 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:51.941013 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b80956c0-66668c8b46-fzj4x"] Jul 22 01:00:53.089170 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:53.089134 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b68fb0f-05ee-4465-a118-db333a6fe02d" path="/var/lib/kubelet/pods/7b68fb0f-05ee-4465-a118-db333a6fe02d/volumes" Jul 22 01:00:55.576587 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:55.576502 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b80956c0-52989"] Jul 22 01:00:55.580546 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:55.580521 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b80956c0-52989"] Jul 22 01:00:57.088302 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:00:57.088265 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b955cba-575f-4c50-bfb4-8d9baf3cc7f1" path="/var/lib/kubelet/pods/9b955cba-575f-4c50-bfb4-8d9baf3cc7f1/volumes" Jul 22 01:01:00.183848 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:01:00.183813 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5186cc87-5cf8b49df5-ml5wq"] Jul 22 01:01:00.184379 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:01:00.184364 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b68fb0f-05ee-4465-a118-db333a6fe02d" containerName="maas-api" Jul 22 01:01:00.184430 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:01:00.184383 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b68fb0f-05ee-4465-a118-db333a6fe02d" containerName="maas-api" Jul 22 01:01:00.184430 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:01:00.184401 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b955cba-575f-4c50-bfb4-8d9baf3cc7f1" containerName="revoke-keys" Jul 22 01:01:00.184430 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:01:00.184409 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b955cba-575f-4c50-bfb4-8d9baf3cc7f1" containerName="revoke-keys" Jul 22 01:01:00.184533 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:01:00.184484 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b68fb0f-05ee-4465-a118-db333a6fe02d" containerName="maas-api" Jul 22 01:01:00.184533 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:01:00.184496 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b955cba-575f-4c50-bfb4-8d9baf3cc7f1" containerName="revoke-keys" Jul 22 01:01:00.187774 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:01:00.187748 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5186cc87-5cf8b49df5-ml5wq" Jul 22 01:01:00.190680 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:01:00.190659 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-5186cc87-serving-cert\"" Jul 22 01:01:00.201229 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:01:00.201208 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5186cc87-5cf8b49df5-ml5wq"] Jul 22 01:01:00.355925 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:01:00.355881 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xc5w\" (UniqueName: \"kubernetes.io/projected/3a23cdb5-16a2-4281-955c-aa2c008fed66-kube-api-access-4xc5w\") pod \"maas-api-e2e-derive-5186cc87-5cf8b49df5-ml5wq\" (UID: \"3a23cdb5-16a2-4281-955c-aa2c008fed66\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5186cc87-5cf8b49df5-ml5wq" Jul 22 01:01:00.356121 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:01:00.356047 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a23cdb5-16a2-4281-955c-aa2c008fed66-openshift-service-ca\") pod \"maas-api-e2e-derive-5186cc87-5cf8b49df5-ml5wq\" (UID: \"3a23cdb5-16a2-4281-955c-aa2c008fed66\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5186cc87-5cf8b49df5-ml5wq" Jul 22 01:01:00.356121 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:01:00.356097 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a23cdb5-16a2-4281-955c-aa2c008fed66-maas-api-tls\") pod \"maas-api-e2e-derive-5186cc87-5cf8b49df5-ml5wq\" (UID: \"3a23cdb5-16a2-4281-955c-aa2c008fed66\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5186cc87-5cf8b49df5-ml5wq" Jul 22 01:01:00.457409 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:01:00.457324 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a23cdb5-16a2-4281-955c-aa2c008fed66-maas-api-tls\") pod \"maas-api-e2e-derive-5186cc87-5cf8b49df5-ml5wq\" (UID: \"3a23cdb5-16a2-4281-955c-aa2c008fed66\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5186cc87-5cf8b49df5-ml5wq" Jul 22 01:01:00.457409 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:01:00.457363 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xc5w\" (UniqueName: \"kubernetes.io/projected/3a23cdb5-16a2-4281-955c-aa2c008fed66-kube-api-access-4xc5w\") pod \"maas-api-e2e-derive-5186cc87-5cf8b49df5-ml5wq\" (UID: \"3a23cdb5-16a2-4281-955c-aa2c008fed66\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5186cc87-5cf8b49df5-ml5wq" Jul 22 01:01:00.457627 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:01:00.457441 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a23cdb5-16a2-4281-955c-aa2c008fed66-openshift-service-ca\") pod \"maas-api-e2e-derive-5186cc87-5cf8b49df5-ml5wq\" (UID: \"3a23cdb5-16a2-4281-955c-aa2c008fed66\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5186cc87-5cf8b49df5-ml5wq" Jul 22 01:01:00.457627 ip-10-0-139-192 kubenswrapper[2574]: E0722 01:01:00.457482 2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-5186cc87-serving-cert: secret "maas-api-e2e-derive-5186cc87-serving-cert" not found Jul 22 01:01:00.457627 ip-10-0-139-192 kubenswrapper[2574]: E0722 01:01:00.457558 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a23cdb5-16a2-4281-955c-aa2c008fed66-maas-api-tls podName:3a23cdb5-16a2-4281-955c-aa2c008fed66 nodeName:}" failed. No retries permitted until 2026-07-22 01:01:00.957540815 +0000 UTC m=+2046.314298673 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3a23cdb5-16a2-4281-955c-aa2c008fed66-maas-api-tls") pod "maas-api-e2e-derive-5186cc87-5cf8b49df5-ml5wq" (UID: "3a23cdb5-16a2-4281-955c-aa2c008fed66") : secret "maas-api-e2e-derive-5186cc87-serving-cert" not found Jul 22 01:01:00.458129 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:01:00.458113 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a23cdb5-16a2-4281-955c-aa2c008fed66-openshift-service-ca\") pod \"maas-api-e2e-derive-5186cc87-5cf8b49df5-ml5wq\" (UID: \"3a23cdb5-16a2-4281-955c-aa2c008fed66\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5186cc87-5cf8b49df5-ml5wq" Jul 22 01:01:00.471497 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:01:00.471473 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xc5w\" (UniqueName: \"kubernetes.io/projected/3a23cdb5-16a2-4281-955c-aa2c008fed66-kube-api-access-4xc5w\") pod \"maas-api-e2e-derive-5186cc87-5cf8b49df5-ml5wq\" (UID: \"3a23cdb5-16a2-4281-955c-aa2c008fed66\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5186cc87-5cf8b49df5-ml5wq" Jul 22 01:01:00.961725 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:01:00.961686 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a23cdb5-16a2-4281-955c-aa2c008fed66-maas-api-tls\") pod \"maas-api-e2e-derive-5186cc87-5cf8b49df5-ml5wq\" (UID: \"3a23cdb5-16a2-4281-955c-aa2c008fed66\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5186cc87-5cf8b49df5-ml5wq" Jul 22 01:01:00.964486 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:01:00.964457 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a23cdb5-16a2-4281-955c-aa2c008fed66-maas-api-tls\") pod \"maas-api-e2e-derive-5186cc87-5cf8b49df5-ml5wq\" (UID: \"3a23cdb5-16a2-4281-955c-aa2c008fed66\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5186cc87-5cf8b49df5-ml5wq" Jul 22 01:01:01.101125 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:01:01.101079 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5186cc87-5cf8b49df5-ml5wq" Jul 22 01:01:01.245045 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:01:01.245018 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5186cc87-5cf8b49df5-ml5wq"] Jul 22 01:01:01.247058 ip-10-0-139-192 kubenswrapper[2574]: W0722 01:01:01.247024 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a23cdb5_16a2_4281_955c_aa2c008fed66.slice/crio-21f8bb265066755cf3b8cf00d8c3dc5bf5ac4a49c6fff4f78acf3bdf306b5313 WatchSource:0}: Error finding container 21f8bb265066755cf3b8cf00d8c3dc5bf5ac4a49c6fff4f78acf3bdf306b5313: Status 404 returned error can't find the container with id 21f8bb265066755cf3b8cf00d8c3dc5bf5ac4a49c6fff4f78acf3bdf306b5313 Jul 22 01:01:01.973846 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:01:01.973810 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5186cc87-5cf8b49df5-ml5wq" event={"ID":"3a23cdb5-16a2-4281-955c-aa2c008fed66","Type":"ContainerStarted","Data":"21f8bb265066755cf3b8cf00d8c3dc5bf5ac4a49c6fff4f78acf3bdf306b5313"} Jul 22 01:01:03.989441 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:01:03.989402 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5186cc87-5cf8b49df5-ml5wq" event={"ID":"3a23cdb5-16a2-4281-955c-aa2c008fed66","Type":"ContainerStarted","Data":"220b8d82deaa59b0d392909cfacc5562f1720997a861f2f1896e3d2123935f61"} Jul 22 01:01:03.989893 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:01:03.989508 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5186cc87-5cf8b49df5-ml5wq" Jul 22 01:01:04.009767 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:01:04.009717 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5186cc87-5cf8b49df5-ml5wq" podStartSLOduration=2.29010954 podStartE2EDuration="4.009702469s" totalImagesPullingTime="1.719592929s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:01:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:01:04.007620858 +0000 UTC m=+2049.364378747" watchObservedRunningTime="2026-07-22 01:01:04.009702469 +0000 UTC m=+2049.366460346" Jul 22 01:01:10.000270 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:01:10.000239 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5186cc87-5cf8b49df5-ml5wq" Jul 22 01:01:24.824984 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:01:24.824896 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5186cc87-gll4f"] Jul 22 01:01:24.829251 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:01:24.829229 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5186cc87-gll4f" Jul 22 01:01:24.833041 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:01:24.833012 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tmggk\"" Jul 22 01:01:24.835743 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:01:24.835716 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5186cc87-gll4f"] Jul 22 01:01:24.876275 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:01:24.876236 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3542cfc-0c1a-45c1-81e1-41a2a9f90e32-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-5186cc87-gll4f\" (UID: \"f3542cfc-0c1a-45c1-81e1-41a2a9f90e32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5186cc87-gll4f" Jul 22 01:01:24.977497 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:01:24.977451 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3542cfc-0c1a-45c1-81e1-41a2a9f90e32-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-5186cc87-gll4f\" (UID: \"f3542cfc-0c1a-45c1-81e1-41a2a9f90e32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5186cc87-gll4f" Jul 22 01:01:24.978099 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:01:24.978079 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3542cfc-0c1a-45c1-81e1-41a2a9f90e32-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-5186cc87-gll4f\" (UID: \"f3542cfc-0c1a-45c1-81e1-41a2a9f90e32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5186cc87-gll4f" Jul 22 01:01:25.143160 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:01:25.143129 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5186cc87-gll4f" Jul 22 01:01:25.285102 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:01:25.285079 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5186cc87-gll4f"] Jul 22 01:01:25.287238 ip-10-0-139-192 kubenswrapper[2574]: W0722 01:01:25.287208 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3542cfc_0c1a_45c1_81e1_41a2a9f90e32.slice/crio-16246724bebc88ac07503b1a4e6c681bc5982cce3c421b5226fb47149b1ad392 WatchSource:0}: Error finding container 16246724bebc88ac07503b1a4e6c681bc5982cce3c421b5226fb47149b1ad392: Status 404 returned error can't find the container with id 16246724bebc88ac07503b1a4e6c681bc5982cce3c421b5226fb47149b1ad392 Jul 22 01:01:26.116617 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:01:26.116566 2574 generic.go:356] "Generic (PLEG): container finished" podID="f3542cfc-0c1a-45c1-81e1-41a2a9f90e32" containerID="ac80f9576fe78ea69c8cf2cb2498bc15f5e783c3ea75daa55bb631040055059b" exitCode=0 Jul 22 01:01:26.117010 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:01:26.116625 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5186cc87-gll4f" event={"ID":"f3542cfc-0c1a-45c1-81e1-41a2a9f90e32","Type":"ContainerDied","Data":"ac80f9576fe78ea69c8cf2cb2498bc15f5e783c3ea75daa55bb631040055059b"} Jul 22 01:01:26.117010 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:01:26.116689 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5186cc87-gll4f" event={"ID":"f3542cfc-0c1a-45c1-81e1-41a2a9f90e32","Type":"ContainerStarted","Data":"16246724bebc88ac07503b1a4e6c681bc5982cce3c421b5226fb47149b1ad392"} Jul 22 01:01:27.255376 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:01:27.255353 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5186cc87-gll4f" Jul 22 01:01:27.297845 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:01:27.297817 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3542cfc-0c1a-45c1-81e1-41a2a9f90e32-maas-api-service-ca\") pod \"f3542cfc-0c1a-45c1-81e1-41a2a9f90e32\" (UID: \"f3542cfc-0c1a-45c1-81e1-41a2a9f90e32\") " Jul 22 01:01:27.298227 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:01:27.298200 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3542cfc-0c1a-45c1-81e1-41a2a9f90e32-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f3542cfc-0c1a-45c1-81e1-41a2a9f90e32" (UID: "f3542cfc-0c1a-45c1-81e1-41a2a9f90e32"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 01:01:27.398863 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:01:27.398837 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3542cfc-0c1a-45c1-81e1-41a2a9f90e32-maas-api-service-ca\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 01:01:28.129511 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:01:28.129483 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5186cc87-gll4f" Jul 22 01:01:28.129701 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:01:28.129523 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5186cc87-gll4f" event={"ID":"f3542cfc-0c1a-45c1-81e1-41a2a9f90e32","Type":"ContainerDied","Data":"16246724bebc88ac07503b1a4e6c681bc5982cce3c421b5226fb47149b1ad392"} Jul 22 01:01:28.129701 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:01:28.129569 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16246724bebc88ac07503b1a4e6c681bc5982cce3c421b5226fb47149b1ad392" Jul 22 01:01:35.456522 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:01:35.456486 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5186cc87-5cf8b49df5-ml5wq"] Jul 22 01:01:35.456970 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:01:35.456731 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5186cc87-5cf8b49df5-ml5wq" podUID="3a23cdb5-16a2-4281-955c-aa2c008fed66" containerName="maas-api" containerID="cri-o://220b8d82deaa59b0d392909cfacc5562f1720997a861f2f1896e3d2123935f61" gracePeriod=30 Jul 22 01:01:35.698319 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:01:35.698293 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5186cc87-5cf8b49df5-ml5wq" Jul 22 01:01:35.775054 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:01:35.774958 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a23cdb5-16a2-4281-955c-aa2c008fed66-openshift-service-ca\") pod \"3a23cdb5-16a2-4281-955c-aa2c008fed66\" (UID: \"3a23cdb5-16a2-4281-955c-aa2c008fed66\") " Jul 22 01:01:35.775054 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:01:35.774999 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4xc5w\" (UniqueName: \"kubernetes.io/projected/3a23cdb5-16a2-4281-955c-aa2c008fed66-kube-api-access-4xc5w\") pod \"3a23cdb5-16a2-4281-955c-aa2c008fed66\" (UID: \"3a23cdb5-16a2-4281-955c-aa2c008fed66\") " Jul 22 01:01:35.775054 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:01:35.775027 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a23cdb5-16a2-4281-955c-aa2c008fed66-maas-api-tls\") pod \"3a23cdb5-16a2-4281-955c-aa2c008fed66\" (UID: \"3a23cdb5-16a2-4281-955c-aa2c008fed66\") " Jul 22 01:01:35.775405 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:01:35.775382 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a23cdb5-16a2-4281-955c-aa2c008fed66-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3a23cdb5-16a2-4281-955c-aa2c008fed66" (UID: "3a23cdb5-16a2-4281-955c-aa2c008fed66"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 01:01:35.777158 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:01:35.777123 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a23cdb5-16a2-4281-955c-aa2c008fed66-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3a23cdb5-16a2-4281-955c-aa2c008fed66" (UID: "3a23cdb5-16a2-4281-955c-aa2c008fed66"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 01:01:35.777158 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:01:35.777137 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a23cdb5-16a2-4281-955c-aa2c008fed66-kube-api-access-4xc5w" (OuterVolumeSpecName: "kube-api-access-4xc5w") pod "3a23cdb5-16a2-4281-955c-aa2c008fed66" (UID: "3a23cdb5-16a2-4281-955c-aa2c008fed66"). InnerVolumeSpecName "kube-api-access-4xc5w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:01:35.876198 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:01:35.876160 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a23cdb5-16a2-4281-955c-aa2c008fed66-openshift-service-ca\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 01:01:35.876198 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:01:35.876192 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xc5w\" (UniqueName: \"kubernetes.io/projected/3a23cdb5-16a2-4281-955c-aa2c008fed66-kube-api-access-4xc5w\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 01:01:35.876198 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:01:35.876202 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a23cdb5-16a2-4281-955c-aa2c008fed66-maas-api-tls\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 01:01:36.181072 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:01:36.181038 2574 generic.go:356] "Generic (PLEG): container finished" podID="3a23cdb5-16a2-4281-955c-aa2c008fed66" containerID="220b8d82deaa59b0d392909cfacc5562f1720997a861f2f1896e3d2123935f61" exitCode=0 Jul 22 01:01:36.181264 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:01:36.181094 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5186cc87-5cf8b49df5-ml5wq" Jul 22 01:01:36.181264 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:01:36.181129 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5186cc87-5cf8b49df5-ml5wq" event={"ID":"3a23cdb5-16a2-4281-955c-aa2c008fed66","Type":"ContainerDied","Data":"220b8d82deaa59b0d392909cfacc5562f1720997a861f2f1896e3d2123935f61"} Jul 22 01:01:36.181264 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:01:36.181208 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5186cc87-5cf8b49df5-ml5wq" event={"ID":"3a23cdb5-16a2-4281-955c-aa2c008fed66","Type":"ContainerDied","Data":"21f8bb265066755cf3b8cf00d8c3dc5bf5ac4a49c6fff4f78acf3bdf306b5313"} Jul 22 01:01:36.181264 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:01:36.181230 2574 scope.go:117] "RemoveContainer" containerID="220b8d82deaa59b0d392909cfacc5562f1720997a861f2f1896e3d2123935f61" Jul 22 01:01:36.190390 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:01:36.190371 2574 scope.go:117] "RemoveContainer" containerID="220b8d82deaa59b0d392909cfacc5562f1720997a861f2f1896e3d2123935f61" Jul 22 01:01:36.190682 ip-10-0-139-192 kubenswrapper[2574]: E0722 01:01:36.190661 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"220b8d82deaa59b0d392909cfacc5562f1720997a861f2f1896e3d2123935f61\": container with ID starting with 220b8d82deaa59b0d392909cfacc5562f1720997a861f2f1896e3d2123935f61 not found: ID does not exist" containerID="220b8d82deaa59b0d392909cfacc5562f1720997a861f2f1896e3d2123935f61" Jul 22 01:01:36.190742 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:01:36.190690 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"220b8d82deaa59b0d392909cfacc5562f1720997a861f2f1896e3d2123935f61"} err="failed to get container status \"220b8d82deaa59b0d392909cfacc5562f1720997a861f2f1896e3d2123935f61\": rpc error: code = NotFound desc = could not find container \"220b8d82deaa59b0d392909cfacc5562f1720997a861f2f1896e3d2123935f61\": container with ID starting with 220b8d82deaa59b0d392909cfacc5562f1720997a861f2f1896e3d2123935f61 not found: ID does not exist" Jul 22 01:01:36.205843 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:01:36.205816 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5186cc87-5cf8b49df5-ml5wq"] Jul 22 01:01:36.213549 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:01:36.213525 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5186cc87-5cf8b49df5-ml5wq"] Jul 22 01:01:37.088990 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:01:37.088960 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a23cdb5-16a2-4281-955c-aa2c008fed66" path="/var/lib/kubelet/pods/3a23cdb5-16a2-4281-955c-aa2c008fed66/volumes" Jul 22 01:01:39.913882 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:01:39.913848 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5186cc87-gll4f"] Jul 22 01:01:39.918146 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:01:39.918121 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5186cc87-gll4f"] Jul 22 01:01:41.089041 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:01:41.089008 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3542cfc-0c1a-45c1-81e1-41a2a9f90e32" path="/var/lib/kubelet/pods/f3542cfc-0c1a-45c1-81e1-41a2a9f90e32/volumes" Jul 22 01:01:55.160433 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:01:55.160402 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-hb69v_bf7b370a-c67c-4c3c-b3ba-7533131eace2/console-operator/1.log" Jul 22 01:01:55.164359 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:01:55.164338 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hr5fn_947d311a-ec4b-42e7-a0fb-ee5edc0caf8d/ovn-acl-logging/0.log" Jul 22 01:01:55.165275 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:01:55.165256 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-hb69v_bf7b370a-c67c-4c3c-b3ba-7533131eace2/console-operator/1.log" Jul 22 01:01:55.169264 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:01:55.169244 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hr5fn_947d311a-ec4b-42e7-a0fb-ee5edc0caf8d/ovn-acl-logging/0.log" Jul 22 01:03:26.399567 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:03:26.399527 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3a8f56da-6fc7948756-kgxfz"] Jul 22 01:03:26.400087 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:03:26.400072 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3542cfc-0c1a-45c1-81e1-41a2a9f90e32" containerName="revoke-keys" Jul 22 01:03:26.400135 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:03:26.400092 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3542cfc-0c1a-45c1-81e1-41a2a9f90e32" containerName="revoke-keys" Jul 22 01:03:26.400135 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:03:26.400114 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a23cdb5-16a2-4281-955c-aa2c008fed66" containerName="maas-api" Jul 22 01:03:26.400135 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:03:26.400123 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a23cdb5-16a2-4281-955c-aa2c008fed66" containerName="maas-api" Jul 22 01:03:26.400244 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:03:26.400216 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3542cfc-0c1a-45c1-81e1-41a2a9f90e32" containerName="revoke-keys" Jul 22 01:03:26.400244 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:03:26.400229 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a23cdb5-16a2-4281-955c-aa2c008fed66" containerName="maas-api" Jul 22 01:03:26.403739 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:03:26.403718 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3a8f56da-6fc7948756-kgxfz" Jul 22 01:03:26.406986 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:03:26.406961 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-3a8f56da-serving-cert\"" Jul 22 01:03:26.419948 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:03:26.419921 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3a8f56da-6fc7948756-kgxfz"] Jul 22 01:03:26.473834 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:03:26.473801 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f851bbc-14b8-4743-b208-aee6f03c2b6d-maas-api-tls\") pod \"maas-api-e2e-mt-3a8f56da-6fc7948756-kgxfz\" (UID: \"0f851bbc-14b8-4743-b208-aee6f03c2b6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3a8f56da-6fc7948756-kgxfz" Jul 22 01:03:26.474022 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:03:26.473850 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f851bbc-14b8-4743-b208-aee6f03c2b6d-openshift-service-ca\") pod \"maas-api-e2e-mt-3a8f56da-6fc7948756-kgxfz\" (UID: \"0f851bbc-14b8-4743-b208-aee6f03c2b6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3a8f56da-6fc7948756-kgxfz" Jul 22 01:03:26.474022 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:03:26.473917 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8kdv\" (UniqueName: \"kubernetes.io/projected/0f851bbc-14b8-4743-b208-aee6f03c2b6d-kube-api-access-x8kdv\") pod \"maas-api-e2e-mt-3a8f56da-6fc7948756-kgxfz\" (UID: \"0f851bbc-14b8-4743-b208-aee6f03c2b6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3a8f56da-6fc7948756-kgxfz" Jul 22 01:03:26.574393 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:03:26.574354 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f851bbc-14b8-4743-b208-aee6f03c2b6d-maas-api-tls\") pod \"maas-api-e2e-mt-3a8f56da-6fc7948756-kgxfz\" (UID: \"0f851bbc-14b8-4743-b208-aee6f03c2b6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3a8f56da-6fc7948756-kgxfz" Jul 22 01:03:26.574393 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:03:26.574400 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f851bbc-14b8-4743-b208-aee6f03c2b6d-openshift-service-ca\") pod \"maas-api-e2e-mt-3a8f56da-6fc7948756-kgxfz\" (UID: \"0f851bbc-14b8-4743-b208-aee6f03c2b6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3a8f56da-6fc7948756-kgxfz" Jul 22 01:03:26.574683 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:03:26.574420 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8kdv\" (UniqueName: \"kubernetes.io/projected/0f851bbc-14b8-4743-b208-aee6f03c2b6d-kube-api-access-x8kdv\") pod \"maas-api-e2e-mt-3a8f56da-6fc7948756-kgxfz\" (UID: \"0f851bbc-14b8-4743-b208-aee6f03c2b6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3a8f56da-6fc7948756-kgxfz" Jul 22 01:03:26.574683 ip-10-0-139-192 kubenswrapper[2574]: E0722 01:03:26.574505 2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-3a8f56da-serving-cert: secret "maas-api-e2e-mt-3a8f56da-serving-cert" not found Jul 22 01:03:26.574683 ip-10-0-139-192 kubenswrapper[2574]: E0722 01:03:26.574590 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f851bbc-14b8-4743-b208-aee6f03c2b6d-maas-api-tls podName:0f851bbc-14b8-4743-b208-aee6f03c2b6d nodeName:}" failed. No retries permitted until 2026-07-22 01:03:27.074574324 +0000 UTC m=+2192.431332179 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0f851bbc-14b8-4743-b208-aee6f03c2b6d-maas-api-tls") pod "maas-api-e2e-mt-3a8f56da-6fc7948756-kgxfz" (UID: "0f851bbc-14b8-4743-b208-aee6f03c2b6d") : secret "maas-api-e2e-mt-3a8f56da-serving-cert" not found Jul 22 01:03:26.575082 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:03:26.575065 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f851bbc-14b8-4743-b208-aee6f03c2b6d-openshift-service-ca\") pod \"maas-api-e2e-mt-3a8f56da-6fc7948756-kgxfz\" (UID: \"0f851bbc-14b8-4743-b208-aee6f03c2b6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3a8f56da-6fc7948756-kgxfz" Jul 22 01:03:26.583816 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:03:26.583785 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8kdv\" (UniqueName: \"kubernetes.io/projected/0f851bbc-14b8-4743-b208-aee6f03c2b6d-kube-api-access-x8kdv\") pod \"maas-api-e2e-mt-3a8f56da-6fc7948756-kgxfz\" (UID: \"0f851bbc-14b8-4743-b208-aee6f03c2b6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3a8f56da-6fc7948756-kgxfz" Jul 22 01:03:27.080088 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:03:27.080051 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f851bbc-14b8-4743-b208-aee6f03c2b6d-maas-api-tls\") pod \"maas-api-e2e-mt-3a8f56da-6fc7948756-kgxfz\" (UID: \"0f851bbc-14b8-4743-b208-aee6f03c2b6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3a8f56da-6fc7948756-kgxfz" Jul 22 01:03:27.082480 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:03:27.082456 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f851bbc-14b8-4743-b208-aee6f03c2b6d-maas-api-tls\") pod \"maas-api-e2e-mt-3a8f56da-6fc7948756-kgxfz\" (UID: \"0f851bbc-14b8-4743-b208-aee6f03c2b6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3a8f56da-6fc7948756-kgxfz" Jul 22 01:03:27.314859 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:03:27.314825 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3a8f56da-6fc7948756-kgxfz" Jul 22 01:03:27.674355 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:03:27.674325 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3a8f56da-6fc7948756-kgxfz"] Jul 22 01:03:27.676065 ip-10-0-139-192 kubenswrapper[2574]: W0722 01:03:27.676038 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f851bbc_14b8_4743_b208_aee6f03c2b6d.slice/crio-bfa2d65cc53f426a4392b88ed16c112a3c3738efc7ab28351e4b147491cc5e70 WatchSource:0}: Error finding container bfa2d65cc53f426a4392b88ed16c112a3c3738efc7ab28351e4b147491cc5e70: Status 404 returned error can't find the container with id bfa2d65cc53f426a4392b88ed16c112a3c3738efc7ab28351e4b147491cc5e70 Jul 22 01:03:27.792633 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:03:27.792584 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3a8f56da-6fc7948756-kgxfz" event={"ID":"0f851bbc-14b8-4743-b208-aee6f03c2b6d","Type":"ContainerStarted","Data":"bfa2d65cc53f426a4392b88ed16c112a3c3738efc7ab28351e4b147491cc5e70"} Jul 22 01:03:28.799178 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:03:28.799139 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3a8f56da-6fc7948756-kgxfz" event={"ID":"0f851bbc-14b8-4743-b208-aee6f03c2b6d","Type":"ContainerStarted","Data":"4906134e356bdc5a9f16cc84737d99d43844fef93a8990af7a2c37ea04013acc"} Jul 22 01:03:28.799589 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:03:28.799243 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3a8f56da-6fc7948756-kgxfz" Jul 22 01:03:28.829222 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:03:28.829156 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3a8f56da-6fc7948756-kgxfz" podStartSLOduration=2.4209589400000002 podStartE2EDuration="2.829136796s" totalImagesPullingTime="408.177856ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:03:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:03:28.82542091 +0000 UTC m=+2194.182178787" watchObservedRunningTime="2026-07-22 01:03:28.829136796 +0000 UTC m=+2194.185894674" Jul 22 01:03:34.810848 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:03:34.810817 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3a8f56da-6fc7948756-kgxfz" Jul 22 01:03:38.131130 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:03:38.131091 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3a8f56da-x6s2n"] Jul 22 01:03:38.134662 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:03:38.134640 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3a8f56da-x6s2n" Jul 22 01:03:38.137669 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:03:38.137646 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tmggk\"" Jul 22 01:03:38.144747 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:03:38.144718 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3a8f56da-x6s2n"] Jul 22 01:03:38.190089 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:03:38.190056 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8585272d-e3f2-4482-96a4-a37afbde147c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-3a8f56da-x6s2n\" (UID: \"8585272d-e3f2-4482-96a4-a37afbde147c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3a8f56da-x6s2n" Jul 22 01:03:38.291130 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:03:38.291095 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8585272d-e3f2-4482-96a4-a37afbde147c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-3a8f56da-x6s2n\" (UID: \"8585272d-e3f2-4482-96a4-a37afbde147c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3a8f56da-x6s2n" Jul 22 01:03:38.291776 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:03:38.291749 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8585272d-e3f2-4482-96a4-a37afbde147c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-3a8f56da-x6s2n\" (UID: \"8585272d-e3f2-4482-96a4-a37afbde147c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3a8f56da-x6s2n" Jul 22 01:03:38.446980 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:03:38.446890 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3a8f56da-x6s2n" Jul 22 01:03:38.597478 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:03:38.597441 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3a8f56da-x6s2n"] Jul 22 01:03:38.602711 ip-10-0-139-192 kubenswrapper[2574]: W0722 01:03:38.602685 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8585272d_e3f2_4482_96a4_a37afbde147c.slice/crio-d05e61d468fa2448d52e6d62e890616c383ca863e64c2fdabc69e655d7d2e5b2 WatchSource:0}: Error finding container d05e61d468fa2448d52e6d62e890616c383ca863e64c2fdabc69e655d7d2e5b2: Status 404 returned error can't find the container with id d05e61d468fa2448d52e6d62e890616c383ca863e64c2fdabc69e655d7d2e5b2 Jul 22 01:03:38.861146 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:03:38.861114 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3a8f56da-x6s2n" event={"ID":"8585272d-e3f2-4482-96a4-a37afbde147c","Type":"ContainerStarted","Data":"c0493910d6d5309943943a415b8e2d36eaf9326b979baa3a54a4468cacece44a"} Jul 22 01:03:38.861330 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:03:38.861166 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3a8f56da-x6s2n" event={"ID":"8585272d-e3f2-4482-96a4-a37afbde147c","Type":"ContainerStarted","Data":"d05e61d468fa2448d52e6d62e890616c383ca863e64c2fdabc69e655d7d2e5b2"} Jul 22 01:03:38.880090 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:03:38.880031 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3a8f56da-x6s2n" podStartSLOduration=0.880015649 podStartE2EDuration="880.015649ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:03:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:03:38.876879689 +0000 UTC m=+2204.233637592" watchObservedRunningTime="2026-07-22 01:03:38.880015649 +0000 UTC m=+2204.236773548" Jul 22 01:03:39.867975 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:03:39.867937 2574 generic.go:356] "Generic (PLEG): container finished" podID="8585272d-e3f2-4482-96a4-a37afbde147c" containerID="c0493910d6d5309943943a415b8e2d36eaf9326b979baa3a54a4468cacece44a" exitCode=0 Jul 22 01:03:39.868421 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:03:39.868007 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3a8f56da-x6s2n" event={"ID":"8585272d-e3f2-4482-96a4-a37afbde147c","Type":"ContainerDied","Data":"c0493910d6d5309943943a415b8e2d36eaf9326b979baa3a54a4468cacece44a"} Jul 22 01:03:41.008236 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:03:41.008208 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3a8f56da-x6s2n" Jul 22 01:03:41.012375 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:03:41.012350 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8585272d-e3f2-4482-96a4-a37afbde147c-maas-api-service-ca\") pod \"8585272d-e3f2-4482-96a4-a37afbde147c\" (UID: \"8585272d-e3f2-4482-96a4-a37afbde147c\") " Jul 22 01:03:41.012664 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:03:41.012643 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8585272d-e3f2-4482-96a4-a37afbde147c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8585272d-e3f2-4482-96a4-a37afbde147c" (UID: "8585272d-e3f2-4482-96a4-a37afbde147c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 01:03:41.113230 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:03:41.113196 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8585272d-e3f2-4482-96a4-a37afbde147c-maas-api-service-ca\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 01:03:41.880024 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:03:41.879991 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3a8f56da-x6s2n" Jul 22 01:03:41.880229 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:03:41.879993 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3a8f56da-x6s2n" event={"ID":"8585272d-e3f2-4482-96a4-a37afbde147c","Type":"ContainerDied","Data":"d05e61d468fa2448d52e6d62e890616c383ca863e64c2fdabc69e655d7d2e5b2"} Jul 22 01:03:41.880229 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:03:41.880111 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d05e61d468fa2448d52e6d62e890616c383ca863e64c2fdabc69e655d7d2e5b2" Jul 22 01:03:48.757111 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:03:48.757071 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3a8f56da-6fc7948756-kgxfz"] Jul 22 01:03:48.757743 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:03:48.757349 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3a8f56da-6fc7948756-kgxfz" podUID="0f851bbc-14b8-4743-b208-aee6f03c2b6d" containerName="maas-api" containerID="cri-o://4906134e356bdc5a9f16cc84737d99d43844fef93a8990af7a2c37ea04013acc" gracePeriod=30 Jul 22 01:03:48.919805 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:03:48.919772 2574 generic.go:356] "Generic (PLEG): container finished" podID="0f851bbc-14b8-4743-b208-aee6f03c2b6d" containerID="4906134e356bdc5a9f16cc84737d99d43844fef93a8990af7a2c37ea04013acc" exitCode=0 Jul 22 01:03:48.919973 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:03:48.919851 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3a8f56da-6fc7948756-kgxfz" event={"ID":"0f851bbc-14b8-4743-b208-aee6f03c2b6d","Type":"ContainerDied","Data":"4906134e356bdc5a9f16cc84737d99d43844fef93a8990af7a2c37ea04013acc"} Jul 22 01:03:49.008691 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:03:49.008629 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3a8f56da-6fc7948756-kgxfz" Jul 22 01:03:49.083298 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:03:49.083260 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f851bbc-14b8-4743-b208-aee6f03c2b6d-openshift-service-ca\") pod \"0f851bbc-14b8-4743-b208-aee6f03c2b6d\" (UID: \"0f851bbc-14b8-4743-b208-aee6f03c2b6d\") " Jul 22 01:03:49.083497 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:03:49.083341 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x8kdv\" (UniqueName: \"kubernetes.io/projected/0f851bbc-14b8-4743-b208-aee6f03c2b6d-kube-api-access-x8kdv\") pod \"0f851bbc-14b8-4743-b208-aee6f03c2b6d\" (UID: \"0f851bbc-14b8-4743-b208-aee6f03c2b6d\") " Jul 22 01:03:49.083497 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:03:49.083381 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f851bbc-14b8-4743-b208-aee6f03c2b6d-maas-api-tls\") pod \"0f851bbc-14b8-4743-b208-aee6f03c2b6d\" (UID: \"0f851bbc-14b8-4743-b208-aee6f03c2b6d\") " Jul 22 01:03:49.083749 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:03:49.083720 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f851bbc-14b8-4743-b208-aee6f03c2b6d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0f851bbc-14b8-4743-b208-aee6f03c2b6d" (UID: "0f851bbc-14b8-4743-b208-aee6f03c2b6d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 01:03:49.085419 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:03:49.085391 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f851bbc-14b8-4743-b208-aee6f03c2b6d-kube-api-access-x8kdv" (OuterVolumeSpecName: "kube-api-access-x8kdv") pod "0f851bbc-14b8-4743-b208-aee6f03c2b6d" (UID: "0f851bbc-14b8-4743-b208-aee6f03c2b6d"). InnerVolumeSpecName "kube-api-access-x8kdv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:03:49.085520 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:03:49.085426 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f851bbc-14b8-4743-b208-aee6f03c2b6d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0f851bbc-14b8-4743-b208-aee6f03c2b6d" (UID: "0f851bbc-14b8-4743-b208-aee6f03c2b6d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 01:03:49.185375 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:03:49.185326 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f851bbc-14b8-4743-b208-aee6f03c2b6d-maas-api-tls\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 01:03:49.185375 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:03:49.185367 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f851bbc-14b8-4743-b208-aee6f03c2b6d-openshift-service-ca\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 01:03:49.185375 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:03:49.185383 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8kdv\" (UniqueName: \"kubernetes.io/projected/0f851bbc-14b8-4743-b208-aee6f03c2b6d-kube-api-access-x8kdv\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 01:03:49.927808 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:03:49.927777 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3a8f56da-6fc7948756-kgxfz" Jul 22 01:03:49.927808 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:03:49.927799 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3a8f56da-6fc7948756-kgxfz" event={"ID":"0f851bbc-14b8-4743-b208-aee6f03c2b6d","Type":"ContainerDied","Data":"bfa2d65cc53f426a4392b88ed16c112a3c3738efc7ab28351e4b147491cc5e70"} Jul 22 01:03:49.928339 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:03:49.927870 2574 scope.go:117] "RemoveContainer" containerID="4906134e356bdc5a9f16cc84737d99d43844fef93a8990af7a2c37ea04013acc" Jul 22 01:03:49.948016 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:03:49.947989 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3a8f56da-6fc7948756-kgxfz"] Jul 22 01:03:49.950153 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:03:49.950134 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3a8f56da-6fc7948756-kgxfz"] Jul 22 01:03:51.088842 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:03:51.088805 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f851bbc-14b8-4743-b208-aee6f03c2b6d" path="/var/lib/kubelet/pods/0f851bbc-14b8-4743-b208-aee6f03c2b6d/volumes" Jul 22 01:03:53.212678 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:03:53.212644 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3a8f56da-x6s2n"] Jul 22 01:03:53.216085 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:03:53.216059 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3a8f56da-x6s2n"] Jul 22 01:03:55.088924 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:03:55.088891 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8585272d-e3f2-4482-96a4-a37afbde147c" path="/var/lib/kubelet/pods/8585272d-e3f2-4482-96a4-a37afbde147c/volumes" Jul 22 01:04:22.488437 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:04:22.488345 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c9532535-7c7d645b99-7gnw2"] Jul 22 01:04:22.490887 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:04:22.488905 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f851bbc-14b8-4743-b208-aee6f03c2b6d" containerName="maas-api" Jul 22 01:04:22.490887 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:04:22.488920 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f851bbc-14b8-4743-b208-aee6f03c2b6d" containerName="maas-api" Jul 22 01:04:22.490887 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:04:22.488945 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8585272d-e3f2-4482-96a4-a37afbde147c" containerName="revoke-keys" Jul 22 01:04:22.490887 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:04:22.488953 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8585272d-e3f2-4482-96a4-a37afbde147c" containerName="revoke-keys" Jul 22 01:04:22.490887 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:04:22.489040 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8585272d-e3f2-4482-96a4-a37afbde147c" containerName="revoke-keys" Jul 22 01:04:22.490887 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:04:22.489053 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f851bbc-14b8-4743-b208-aee6f03c2b6d" containerName="maas-api" Jul 22 01:04:22.492175 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:04:22.492156 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9532535-7c7d645b99-7gnw2" Jul 22 01:04:22.494945 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:04:22.494924 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-c9532535-serving-cert\"" Jul 22 01:04:22.502066 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:04:22.502045 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c9532535-7c7d645b99-7gnw2"] Jul 22 01:04:22.586071 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:04:22.586032 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pq598\" (UniqueName: \"kubernetes.io/projected/372085ba-64f6-4b91-9924-18f3316d9fbb-kube-api-access-pq598\") pod \"maas-api-e2e-mt-c9532535-7c7d645b99-7gnw2\" (UID: \"372085ba-64f6-4b91-9924-18f3316d9fbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9532535-7c7d645b99-7gnw2" Jul 22 01:04:22.586257 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:04:22.586084 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/372085ba-64f6-4b91-9924-18f3316d9fbb-openshift-service-ca\") pod \"maas-api-e2e-mt-c9532535-7c7d645b99-7gnw2\" (UID: \"372085ba-64f6-4b91-9924-18f3316d9fbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9532535-7c7d645b99-7gnw2" Jul 22 01:04:22.586257 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:04:22.586157 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/372085ba-64f6-4b91-9924-18f3316d9fbb-maas-api-tls\") pod \"maas-api-e2e-mt-c9532535-7c7d645b99-7gnw2\" (UID: \"372085ba-64f6-4b91-9924-18f3316d9fbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9532535-7c7d645b99-7gnw2" Jul 22 01:04:22.687244 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:04:22.687208 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq598\" (UniqueName: \"kubernetes.io/projected/372085ba-64f6-4b91-9924-18f3316d9fbb-kube-api-access-pq598\") pod \"maas-api-e2e-mt-c9532535-7c7d645b99-7gnw2\" (UID: \"372085ba-64f6-4b91-9924-18f3316d9fbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9532535-7c7d645b99-7gnw2" Jul 22 01:04:22.687244 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:04:22.687249 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/372085ba-64f6-4b91-9924-18f3316d9fbb-openshift-service-ca\") pod \"maas-api-e2e-mt-c9532535-7c7d645b99-7gnw2\" (UID: \"372085ba-64f6-4b91-9924-18f3316d9fbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9532535-7c7d645b99-7gnw2" Jul 22 01:04:22.687509 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:04:22.687279 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/372085ba-64f6-4b91-9924-18f3316d9fbb-maas-api-tls\") pod \"maas-api-e2e-mt-c9532535-7c7d645b99-7gnw2\" (UID: \"372085ba-64f6-4b91-9924-18f3316d9fbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9532535-7c7d645b99-7gnw2" Jul 22 01:04:22.687509 ip-10-0-139-192 kubenswrapper[2574]: E0722 01:04:22.687428 2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-c9532535-serving-cert: secret "maas-api-e2e-mt-c9532535-serving-cert" not found Jul 22 01:04:22.687509 ip-10-0-139-192 kubenswrapper[2574]: E0722 01:04:22.687487 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/372085ba-64f6-4b91-9924-18f3316d9fbb-maas-api-tls podName:372085ba-64f6-4b91-9924-18f3316d9fbb nodeName:}" failed. No retries permitted until 2026-07-22 01:04:23.187468146 +0000 UTC m=+2248.544226006 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/372085ba-64f6-4b91-9924-18f3316d9fbb-maas-api-tls") pod "maas-api-e2e-mt-c9532535-7c7d645b99-7gnw2" (UID: "372085ba-64f6-4b91-9924-18f3316d9fbb") : secret "maas-api-e2e-mt-c9532535-serving-cert" not found Jul 22 01:04:22.687879 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:04:22.687860 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/372085ba-64f6-4b91-9924-18f3316d9fbb-openshift-service-ca\") pod \"maas-api-e2e-mt-c9532535-7c7d645b99-7gnw2\" (UID: \"372085ba-64f6-4b91-9924-18f3316d9fbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9532535-7c7d645b99-7gnw2" Jul 22 01:04:22.698062 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:04:22.698037 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pq598\" (UniqueName: \"kubernetes.io/projected/372085ba-64f6-4b91-9924-18f3316d9fbb-kube-api-access-pq598\") pod \"maas-api-e2e-mt-c9532535-7c7d645b99-7gnw2\" (UID: \"372085ba-64f6-4b91-9924-18f3316d9fbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9532535-7c7d645b99-7gnw2" Jul 22 01:04:23.192516 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:04:23.192482 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/372085ba-64f6-4b91-9924-18f3316d9fbb-maas-api-tls\") pod \"maas-api-e2e-mt-c9532535-7c7d645b99-7gnw2\" (UID: \"372085ba-64f6-4b91-9924-18f3316d9fbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9532535-7c7d645b99-7gnw2" Jul 22 01:04:23.192735 ip-10-0-139-192 kubenswrapper[2574]: E0722 01:04:23.192656 2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-c9532535-serving-cert: secret "maas-api-e2e-mt-c9532535-serving-cert" not found Jul 22 01:04:23.192735 ip-10-0-139-192 kubenswrapper[2574]: E0722 01:04:23.192720 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/372085ba-64f6-4b91-9924-18f3316d9fbb-maas-api-tls podName:372085ba-64f6-4b91-9924-18f3316d9fbb nodeName:}" failed. No retries permitted until 2026-07-22 01:04:24.19270285 +0000 UTC m=+2249.549460709 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/372085ba-64f6-4b91-9924-18f3316d9fbb-maas-api-tls") pod "maas-api-e2e-mt-c9532535-7c7d645b99-7gnw2" (UID: "372085ba-64f6-4b91-9924-18f3316d9fbb") : secret "maas-api-e2e-mt-c9532535-serving-cert" not found Jul 22 01:04:24.203440 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:04:24.203402 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/372085ba-64f6-4b91-9924-18f3316d9fbb-maas-api-tls\") pod \"maas-api-e2e-mt-c9532535-7c7d645b99-7gnw2\" (UID: \"372085ba-64f6-4b91-9924-18f3316d9fbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9532535-7c7d645b99-7gnw2" Jul 22 01:04:24.206094 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:04:24.206067 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/372085ba-64f6-4b91-9924-18f3316d9fbb-maas-api-tls\") pod \"maas-api-e2e-mt-c9532535-7c7d645b99-7gnw2\" (UID: \"372085ba-64f6-4b91-9924-18f3316d9fbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9532535-7c7d645b99-7gnw2" Jul 22 01:04:24.304938 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:04:24.304905 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9532535-7c7d645b99-7gnw2" Jul 22 01:04:24.445863 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:04:24.445830 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c9532535-7c7d645b99-7gnw2"] Jul 22 01:04:24.447660 ip-10-0-139-192 kubenswrapper[2574]: W0722 01:04:24.447636 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod372085ba_64f6_4b91_9924_18f3316d9fbb.slice/crio-c542a6c1af8ea4c335deee61fc492871fb6d553fe88b6e2216a1f35c5a97e25d WatchSource:0}: Error finding container c542a6c1af8ea4c335deee61fc492871fb6d553fe88b6e2216a1f35c5a97e25d: Status 404 returned error can't find the container with id c542a6c1af8ea4c335deee61fc492871fb6d553fe88b6e2216a1f35c5a97e25d Jul 22 01:04:24.449411 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:04:24.449390 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 01:04:25.129690 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:04:25.129642 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9532535-7c7d645b99-7gnw2" event={"ID":"372085ba-64f6-4b91-9924-18f3316d9fbb","Type":"ContainerStarted","Data":"fb2f64a6ce96b971be5d5c289db69feb5b56e178a7d18d4b545f7590998d1801"} Jul 22 01:04:25.129878 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:04:25.129732 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9532535-7c7d645b99-7gnw2" Jul 22 01:04:25.129878 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:04:25.129758 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9532535-7c7d645b99-7gnw2" event={"ID":"372085ba-64f6-4b91-9924-18f3316d9fbb","Type":"ContainerStarted","Data":"c542a6c1af8ea4c335deee61fc492871fb6d553fe88b6e2216a1f35c5a97e25d"} Jul 22 01:04:25.146260 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:04:25.146175 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9532535-7c7d645b99-7gnw2" podStartSLOduration=2.648119677 podStartE2EDuration="3.146153282s" totalImagesPullingTime="498.033605ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:04:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:04:25.146064621 +0000 UTC m=+2250.502822497" watchObservedRunningTime="2026-07-22 01:04:25.146153282 +0000 UTC m=+2250.502911160" Jul 22 01:04:31.143250 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:04:31.143221 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9532535-7c7d645b99-7gnw2" Jul 22 01:04:55.097322 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:04:55.097288 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c9532535-wj79r"] Jul 22 01:04:55.101115 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:04:55.101099 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c9532535-wj79r" Jul 22 01:04:55.104178 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:04:55.104050 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tmggk\"" Jul 22 01:04:55.105419 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:04:55.105376 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c9532535-wj79r"] Jul 22 01:04:55.183310 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:04:55.183274 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1a05595-f290-4348-b4e3-cac6bb59f216-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c9532535-wj79r\" (UID: \"a1a05595-f290-4348-b4e3-cac6bb59f216\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c9532535-wj79r" Jul 22 01:04:55.284535 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:04:55.284502 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1a05595-f290-4348-b4e3-cac6bb59f216-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c9532535-wj79r\" (UID: \"a1a05595-f290-4348-b4e3-cac6bb59f216\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c9532535-wj79r" Jul 22 01:04:55.285159 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:04:55.285131 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1a05595-f290-4348-b4e3-cac6bb59f216-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c9532535-wj79r\" (UID: \"a1a05595-f290-4348-b4e3-cac6bb59f216\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c9532535-wj79r" Jul 22 01:04:55.413012 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:04:55.412982 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c9532535-wj79r" Jul 22 01:04:55.549963 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:04:55.549938 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c9532535-wj79r"] Jul 22 01:04:55.552016 ip-10-0-139-192 kubenswrapper[2574]: W0722 01:04:55.551984 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1a05595_f290_4348_b4e3_cac6bb59f216.slice/crio-aa833bb51488ec925799afad3421694c4e05fb7ab9059e6e68be723c30c6e31a WatchSource:0}: Error finding container aa833bb51488ec925799afad3421694c4e05fb7ab9059e6e68be723c30c6e31a: Status 404 returned error can't find the container with id aa833bb51488ec925799afad3421694c4e05fb7ab9059e6e68be723c30c6e31a Jul 22 01:04:56.314400 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:04:56.314366 2574 generic.go:356] "Generic (PLEG): container finished" podID="a1a05595-f290-4348-b4e3-cac6bb59f216" containerID="68a2d0ac28ef13f383661eeb5957f40ecea99e03008f7b65edfb10e8fb602797" exitCode=0 Jul 22 01:04:56.314866 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:04:56.314418 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c9532535-wj79r" event={"ID":"a1a05595-f290-4348-b4e3-cac6bb59f216","Type":"ContainerDied","Data":"68a2d0ac28ef13f383661eeb5957f40ecea99e03008f7b65edfb10e8fb602797"} Jul 22 01:04:56.314866 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:04:56.314484 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c9532535-wj79r" event={"ID":"a1a05595-f290-4348-b4e3-cac6bb59f216","Type":"ContainerStarted","Data":"aa833bb51488ec925799afad3421694c4e05fb7ab9059e6e68be723c30c6e31a"} Jul 22 01:04:57.461317 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:04:57.461290 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c9532535-wj79r" Jul 22 01:04:57.604944 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:04:57.604861 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1a05595-f290-4348-b4e3-cac6bb59f216-maas-api-service-ca\") pod \"a1a05595-f290-4348-b4e3-cac6bb59f216\" (UID: \"a1a05595-f290-4348-b4e3-cac6bb59f216\") " Jul 22 01:04:57.605255 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:04:57.605234 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1a05595-f290-4348-b4e3-cac6bb59f216-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a1a05595-f290-4348-b4e3-cac6bb59f216" (UID: "a1a05595-f290-4348-b4e3-cac6bb59f216"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 01:04:57.706003 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:04:57.705960 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1a05595-f290-4348-b4e3-cac6bb59f216-maas-api-service-ca\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 01:04:58.327086 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:04:58.327056 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c9532535-wj79r" Jul 22 01:04:58.327262 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:04:58.327100 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c9532535-wj79r" event={"ID":"a1a05595-f290-4348-b4e3-cac6bb59f216","Type":"ContainerDied","Data":"aa833bb51488ec925799afad3421694c4e05fb7ab9059e6e68be723c30c6e31a"} Jul 22 01:04:58.327262 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:04:58.327175 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa833bb51488ec925799afad3421694c4e05fb7ab9059e6e68be723c30c6e31a" Jul 22 01:05:10.502413 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:05:10.502375 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c9532535-7c7d645b99-7gnw2"] Jul 22 01:05:10.503017 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:05:10.502642 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9532535-7c7d645b99-7gnw2" podUID="372085ba-64f6-4b91-9924-18f3316d9fbb" containerName="maas-api" containerID="cri-o://fb2f64a6ce96b971be5d5c289db69feb5b56e178a7d18d4b545f7590998d1801" gracePeriod=30 Jul 22 01:05:10.765905 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:05:10.765843 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9532535-7c7d645b99-7gnw2" Jul 22 01:05:10.925186 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:05:10.925150 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/372085ba-64f6-4b91-9924-18f3316d9fbb-maas-api-tls\") pod \"372085ba-64f6-4b91-9924-18f3316d9fbb\" (UID: \"372085ba-64f6-4b91-9924-18f3316d9fbb\") " Jul 22 01:05:10.925366 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:05:10.925198 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/372085ba-64f6-4b91-9924-18f3316d9fbb-openshift-service-ca\") pod \"372085ba-64f6-4b91-9924-18f3316d9fbb\" (UID: \"372085ba-64f6-4b91-9924-18f3316d9fbb\") " Jul 22 01:05:10.925366 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:05:10.925321 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pq598\" (UniqueName: \"kubernetes.io/projected/372085ba-64f6-4b91-9924-18f3316d9fbb-kube-api-access-pq598\") pod \"372085ba-64f6-4b91-9924-18f3316d9fbb\" (UID: \"372085ba-64f6-4b91-9924-18f3316d9fbb\") " Jul 22 01:05:10.925674 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:05:10.925644 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/372085ba-64f6-4b91-9924-18f3316d9fbb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "372085ba-64f6-4b91-9924-18f3316d9fbb" (UID: "372085ba-64f6-4b91-9924-18f3316d9fbb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 01:05:10.927423 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:05:10.927394 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/372085ba-64f6-4b91-9924-18f3316d9fbb-kube-api-access-pq598" (OuterVolumeSpecName: "kube-api-access-pq598") pod "372085ba-64f6-4b91-9924-18f3316d9fbb" (UID: "372085ba-64f6-4b91-9924-18f3316d9fbb"). InnerVolumeSpecName "kube-api-access-pq598". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:05:10.927522 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:05:10.927427 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/372085ba-64f6-4b91-9924-18f3316d9fbb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "372085ba-64f6-4b91-9924-18f3316d9fbb" (UID: "372085ba-64f6-4b91-9924-18f3316d9fbb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 01:05:11.026776 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:05:11.026677 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pq598\" (UniqueName: \"kubernetes.io/projected/372085ba-64f6-4b91-9924-18f3316d9fbb-kube-api-access-pq598\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 01:05:11.026776 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:05:11.026721 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/372085ba-64f6-4b91-9924-18f3316d9fbb-maas-api-tls\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 01:05:11.026776 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:05:11.026733 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/372085ba-64f6-4b91-9924-18f3316d9fbb-openshift-service-ca\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 01:05:11.405454 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:05:11.405420 2574 generic.go:356] "Generic (PLEG): container finished" podID="372085ba-64f6-4b91-9924-18f3316d9fbb" containerID="fb2f64a6ce96b971be5d5c289db69feb5b56e178a7d18d4b545f7590998d1801" exitCode=0 Jul 22 01:05:11.405649 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:05:11.405466 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9532535-7c7d645b99-7gnw2" Jul 22 01:05:11.405649 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:05:11.405506 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9532535-7c7d645b99-7gnw2" event={"ID":"372085ba-64f6-4b91-9924-18f3316d9fbb","Type":"ContainerDied","Data":"fb2f64a6ce96b971be5d5c289db69feb5b56e178a7d18d4b545f7590998d1801"} Jul 22 01:05:11.405649 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:05:11.405567 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c9532535-7c7d645b99-7gnw2" event={"ID":"372085ba-64f6-4b91-9924-18f3316d9fbb","Type":"ContainerDied","Data":"c542a6c1af8ea4c335deee61fc492871fb6d553fe88b6e2216a1f35c5a97e25d"} Jul 22 01:05:11.405649 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:05:11.405596 2574 scope.go:117] "RemoveContainer" containerID="fb2f64a6ce96b971be5d5c289db69feb5b56e178a7d18d4b545f7590998d1801" Jul 22 01:05:11.414642 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:05:11.414623 2574 scope.go:117] "RemoveContainer" containerID="fb2f64a6ce96b971be5d5c289db69feb5b56e178a7d18d4b545f7590998d1801" Jul 22 01:05:11.414907 ip-10-0-139-192 kubenswrapper[2574]: E0722 01:05:11.414887 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb2f64a6ce96b971be5d5c289db69feb5b56e178a7d18d4b545f7590998d1801\": container with ID starting with fb2f64a6ce96b971be5d5c289db69feb5b56e178a7d18d4b545f7590998d1801 not found: ID does not exist" containerID="fb2f64a6ce96b971be5d5c289db69feb5b56e178a7d18d4b545f7590998d1801" Jul 22 01:05:11.414985 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:05:11.414918 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb2f64a6ce96b971be5d5c289db69feb5b56e178a7d18d4b545f7590998d1801"} err="failed to get container status \"fb2f64a6ce96b971be5d5c289db69feb5b56e178a7d18d4b545f7590998d1801\": rpc error: code = NotFound desc = could not find container \"fb2f64a6ce96b971be5d5c289db69feb5b56e178a7d18d4b545f7590998d1801\": container with ID starting with fb2f64a6ce96b971be5d5c289db69feb5b56e178a7d18d4b545f7590998d1801 not found: ID does not exist" Jul 22 01:05:11.423110 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:05:11.423089 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c9532535-7c7d645b99-7gnw2"] Jul 22 01:05:11.425292 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:05:11.425267 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c9532535-7c7d645b99-7gnw2"] Jul 22 01:05:13.089760 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:05:13.089727 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="372085ba-64f6-4b91-9924-18f3316d9fbb" path="/var/lib/kubelet/pods/372085ba-64f6-4b91-9924-18f3316d9fbb/volumes" Jul 22 01:05:15.186702 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:05:15.186653 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c9532535-wj79r"] Jul 22 01:05:15.188846 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:05:15.188824 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c9532535-wj79r"] Jul 22 01:05:17.089096 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:05:17.089061 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1a05595-f290-4348-b4e3-cac6bb59f216" path="/var/lib/kubelet/pods/a1a05595-f290-4348-b4e3-cac6bb59f216/volumes" Jul 22 01:05:55.287486 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:05:55.287408 2574 scope.go:117] "RemoveContainer" containerID="b8c3dcf762d15fec823fd45bbae3441ed91ec37c4cfb0e0e0d6e548e36557906" Jul 22 01:06:55.198293 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:06:55.198265 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-hb69v_bf7b370a-c67c-4c3c-b3ba-7533131eace2/console-operator/1.log" Jul 22 01:06:55.202456 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:06:55.202431 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hr5fn_947d311a-ec4b-42e7-a0fb-ee5edc0caf8d/ovn-acl-logging/0.log" Jul 22 01:06:55.205827 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:06:55.205808 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-hb69v_bf7b370a-c67c-4c3c-b3ba-7533131eace2/console-operator/1.log" Jul 22 01:06:55.209893 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:06:55.209875 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hr5fn_947d311a-ec4b-42e7-a0fb-ee5edc0caf8d/ovn-acl-logging/0.log" Jul 22 01:06:55.311998 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:06:55.311964 2574 scope.go:117] "RemoveContainer" containerID="ae74c371f06cec8c021387ae75b6b0025ea32a08205f43bee6ecc9f8259f8600" Jul 22 01:06:55.320794 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:06:55.320771 2574 scope.go:117] "RemoveContainer" containerID="53c1ebae67f5243c1bbf7df90d2536203655314698a39d5b68a5bdc4c27ea2a0" Jul 22 01:06:55.329089 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:06:55.329067 2574 scope.go:117] "RemoveContainer" containerID="8a3f379ec46ee1ba9f8eae794d4e83b46f42ebff004cf1f9a5f40b61fdf4e03d" Jul 22 01:07:03.380054 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:07:03.380020 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-805a5b-5fbc98b88c-kmqmj"] Jul 22 01:07:03.386249 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:07:03.383296 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1a05595-f290-4348-b4e3-cac6bb59f216" containerName="revoke-keys" Jul 22 01:07:03.386249 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:07:03.383328 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1a05595-f290-4348-b4e3-cac6bb59f216" containerName="revoke-keys" Jul 22 01:07:03.386249 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:07:03.383393 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="372085ba-64f6-4b91-9924-18f3316d9fbb" containerName="maas-api" Jul 22 01:07:03.386249 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:07:03.383402 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="372085ba-64f6-4b91-9924-18f3316d9fbb" containerName="maas-api" Jul 22 01:07:03.386249 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:07:03.383623 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1a05595-f290-4348-b4e3-cac6bb59f216" containerName="revoke-keys" Jul 22 01:07:03.386249 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:07:03.383649 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="372085ba-64f6-4b91-9924-18f3316d9fbb" containerName="maas-api" Jul 22 01:07:03.389350 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:07:03.389324 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-805a5b-5fbc98b88c-kmqmj" Jul 22 01:07:03.392075 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:07:03.392050 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-805a5b-serving-cert\"" Jul 22 01:07:03.394822 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:07:03.394790 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-805a5b-5fbc98b88c-kmqmj"] Jul 22 01:07:03.420852 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:07:03.420817 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e248da55-1c7c-4b6b-a203-c5da3ca3a63d-maas-api-tls\") pod \"maas-api-e2e-inf-a-805a5b-5fbc98b88c-kmqmj\" (UID: \"e248da55-1c7c-4b6b-a203-c5da3ca3a63d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-805a5b-5fbc98b88c-kmqmj" Jul 22 01:07:03.421006 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:07:03.420868 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2rmf\" (UniqueName: \"kubernetes.io/projected/e248da55-1c7c-4b6b-a203-c5da3ca3a63d-kube-api-access-z2rmf\") pod \"maas-api-e2e-inf-a-805a5b-5fbc98b88c-kmqmj\" (UID: \"e248da55-1c7c-4b6b-a203-c5da3ca3a63d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-805a5b-5fbc98b88c-kmqmj" Jul 22 01:07:03.421006 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:07:03.420947 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e248da55-1c7c-4b6b-a203-c5da3ca3a63d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-805a5b-5fbc98b88c-kmqmj\" (UID: \"e248da55-1c7c-4b6b-a203-c5da3ca3a63d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-805a5b-5fbc98b88c-kmqmj" Jul 22 01:07:03.521499 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:07:03.521468 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e248da55-1c7c-4b6b-a203-c5da3ca3a63d-maas-api-tls\") pod \"maas-api-e2e-inf-a-805a5b-5fbc98b88c-kmqmj\" (UID: \"e248da55-1c7c-4b6b-a203-c5da3ca3a63d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-805a5b-5fbc98b88c-kmqmj" Jul 22 01:07:03.521499 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:07:03.521503 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2rmf\" (UniqueName: \"kubernetes.io/projected/e248da55-1c7c-4b6b-a203-c5da3ca3a63d-kube-api-access-z2rmf\") pod \"maas-api-e2e-inf-a-805a5b-5fbc98b88c-kmqmj\" (UID: \"e248da55-1c7c-4b6b-a203-c5da3ca3a63d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-805a5b-5fbc98b88c-kmqmj" Jul 22 01:07:03.521726 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:07:03.521532 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e248da55-1c7c-4b6b-a203-c5da3ca3a63d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-805a5b-5fbc98b88c-kmqmj\" (UID: \"e248da55-1c7c-4b6b-a203-c5da3ca3a63d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-805a5b-5fbc98b88c-kmqmj" Jul 22 01:07:03.522219 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:07:03.522194 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e248da55-1c7c-4b6b-a203-c5da3ca3a63d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-805a5b-5fbc98b88c-kmqmj\" (UID: \"e248da55-1c7c-4b6b-a203-c5da3ca3a63d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-805a5b-5fbc98b88c-kmqmj" Jul 22 01:07:03.524028 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:07:03.524007 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e248da55-1c7c-4b6b-a203-c5da3ca3a63d-maas-api-tls\") pod \"maas-api-e2e-inf-a-805a5b-5fbc98b88c-kmqmj\" (UID: \"e248da55-1c7c-4b6b-a203-c5da3ca3a63d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-805a5b-5fbc98b88c-kmqmj" Jul 22 01:07:03.529763 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:07:03.529739 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2rmf\" (UniqueName: \"kubernetes.io/projected/e248da55-1c7c-4b6b-a203-c5da3ca3a63d-kube-api-access-z2rmf\") pod \"maas-api-e2e-inf-a-805a5b-5fbc98b88c-kmqmj\" (UID: \"e248da55-1c7c-4b6b-a203-c5da3ca3a63d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-805a5b-5fbc98b88c-kmqmj" Jul 22 01:07:03.702034 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:07:03.701936 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-805a5b-5fbc98b88c-kmqmj" Jul 22 01:07:03.845474 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:07:03.845449 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-805a5b-5fbc98b88c-kmqmj"] Jul 22 01:07:03.847571 ip-10-0-139-192 kubenswrapper[2574]: W0722 01:07:03.847519 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode248da55_1c7c_4b6b_a203_c5da3ca3a63d.slice/crio-dccfa93aea51111e6e50da5d2ffd473e74460f08a86763ccd0cb8c48c3d62ea9 WatchSource:0}: Error finding container dccfa93aea51111e6e50da5d2ffd473e74460f08a86763ccd0cb8c48c3d62ea9: Status 404 returned error can't find the container with id dccfa93aea51111e6e50da5d2ffd473e74460f08a86763ccd0cb8c48c3d62ea9 Jul 22 01:07:04.038167 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:07:04.038079 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-805a5b-5fbc98b88c-kmqmj" event={"ID":"e248da55-1c7c-4b6b-a203-c5da3ca3a63d","Type":"ContainerStarted","Data":"dccfa93aea51111e6e50da5d2ffd473e74460f08a86763ccd0cb8c48c3d62ea9"} Jul 22 01:07:05.045988 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:07:05.045888 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-805a5b-5fbc98b88c-kmqmj" event={"ID":"e248da55-1c7c-4b6b-a203-c5da3ca3a63d","Type":"ContainerStarted","Data":"4337a86e02ce25f697e169edc8e52787169e6f370c52f39badd7af54c3996e58"} Jul 22 01:07:05.046411 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:07:05.046092 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-805a5b-5fbc98b88c-kmqmj" Jul 22 01:07:05.064749 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:07:05.064696 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-805a5b-5fbc98b88c-kmqmj" podStartSLOduration=1.122375336 podStartE2EDuration="2.064680529s" totalImagesPullingTime="942.305193ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:07:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:07:05.061723159 +0000 UTC m=+2410.418481037" watchObservedRunningTime="2026-07-22 01:07:05.064680529 +0000 UTC m=+2410.421438416" Jul 22 01:07:11.057431 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:07:11.057401 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-805a5b-5fbc98b88c-kmqmj" Jul 22 01:07:11.276043 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:07:11.276004 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-75fe9f-859b994b55-qgsdg"] Jul 22 01:07:11.279864 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:07:11.279839 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75fe9f-859b994b55-qgsdg" Jul 22 01:07:11.282526 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:07:11.282470 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-75fe9f-serving-cert\"" Jul 22 01:07:11.286453 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:07:11.286430 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-75fe9f-859b994b55-qgsdg"] Jul 22 01:07:11.299101 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:07:11.299070 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f453aee-4f16-40a0-b86a-9f68815293f1-maas-api-tls\") pod \"maas-api-e2e-inf-b-75fe9f-859b994b55-qgsdg\" (UID: \"1f453aee-4f16-40a0-b86a-9f68815293f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75fe9f-859b994b55-qgsdg" Jul 22 01:07:11.299243 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:07:11.299192 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f453aee-4f16-40a0-b86a-9f68815293f1-openshift-service-ca\") pod \"maas-api-e2e-inf-b-75fe9f-859b994b55-qgsdg\" (UID: \"1f453aee-4f16-40a0-b86a-9f68815293f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75fe9f-859b994b55-qgsdg" Jul 22 01:07:11.299312 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:07:11.299241 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24pzq\" (UniqueName: \"kubernetes.io/projected/1f453aee-4f16-40a0-b86a-9f68815293f1-kube-api-access-24pzq\") pod \"maas-api-e2e-inf-b-75fe9f-859b994b55-qgsdg\" (UID: \"1f453aee-4f16-40a0-b86a-9f68815293f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75fe9f-859b994b55-qgsdg" Jul 22 01:07:11.400719 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:07:11.400680 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f453aee-4f16-40a0-b86a-9f68815293f1-openshift-service-ca\") pod \"maas-api-e2e-inf-b-75fe9f-859b994b55-qgsdg\" (UID: \"1f453aee-4f16-40a0-b86a-9f68815293f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75fe9f-859b994b55-qgsdg" Jul 22 01:07:11.400719 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:07:11.400723 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24pzq\" (UniqueName: \"kubernetes.io/projected/1f453aee-4f16-40a0-b86a-9f68815293f1-kube-api-access-24pzq\") pod \"maas-api-e2e-inf-b-75fe9f-859b994b55-qgsdg\" (UID: \"1f453aee-4f16-40a0-b86a-9f68815293f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75fe9f-859b994b55-qgsdg" Jul 22 01:07:11.400994 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:07:11.400743 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f453aee-4f16-40a0-b86a-9f68815293f1-maas-api-tls\") pod \"maas-api-e2e-inf-b-75fe9f-859b994b55-qgsdg\" (UID: \"1f453aee-4f16-40a0-b86a-9f68815293f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75fe9f-859b994b55-qgsdg" Jul 22 01:07:11.401338 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:07:11.401317 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f453aee-4f16-40a0-b86a-9f68815293f1-openshift-service-ca\") pod \"maas-api-e2e-inf-b-75fe9f-859b994b55-qgsdg\" (UID: \"1f453aee-4f16-40a0-b86a-9f68815293f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75fe9f-859b994b55-qgsdg" Jul 22 01:07:11.403206 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:07:11.403183 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f453aee-4f16-40a0-b86a-9f68815293f1-maas-api-tls\") pod \"maas-api-e2e-inf-b-75fe9f-859b994b55-qgsdg\" (UID: \"1f453aee-4f16-40a0-b86a-9f68815293f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75fe9f-859b994b55-qgsdg" Jul 22 01:07:11.416168 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:07:11.416141 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24pzq\" (UniqueName: \"kubernetes.io/projected/1f453aee-4f16-40a0-b86a-9f68815293f1-kube-api-access-24pzq\") pod \"maas-api-e2e-inf-b-75fe9f-859b994b55-qgsdg\" (UID: \"1f453aee-4f16-40a0-b86a-9f68815293f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75fe9f-859b994b55-qgsdg" Jul 22 01:07:11.594082 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:07:11.594045 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75fe9f-859b994b55-qgsdg" Jul 22 01:07:11.756955 ip-10-0-139-192 kubenswrapper[2574]: W0722 01:07:11.756922 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f453aee_4f16_40a0_b86a_9f68815293f1.slice/crio-fced2213398e4fd2983c00c97a9964c1f0edbef5f799ff669c93c56ca7842b2b WatchSource:0}: Error finding container fced2213398e4fd2983c00c97a9964c1f0edbef5f799ff669c93c56ca7842b2b: Status 404 returned error can't find the container with id fced2213398e4fd2983c00c97a9964c1f0edbef5f799ff669c93c56ca7842b2b Jul 22 01:07:11.757592 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:07:11.757568 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-75fe9f-859b994b55-qgsdg"] Jul 22 01:07:12.085141 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:07:12.085036 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75fe9f-859b994b55-qgsdg" event={"ID":"1f453aee-4f16-40a0-b86a-9f68815293f1","Type":"ContainerStarted","Data":"fced2213398e4fd2983c00c97a9964c1f0edbef5f799ff669c93c56ca7842b2b"} Jul 22 01:07:13.091824 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:07:13.091788 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75fe9f-859b994b55-qgsdg" event={"ID":"1f453aee-4f16-40a0-b86a-9f68815293f1","Type":"ContainerStarted","Data":"ef07c3bb288feeb5ce6ec6c50a99c41c2f40254710a34cdf353a4c48393219f0"} Jul 22 01:07:13.092209 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:07:13.091898 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75fe9f-859b994b55-qgsdg" Jul 22 01:07:13.112178 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:07:13.112126 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75fe9f-859b994b55-qgsdg" podStartSLOduration=1.503554076 podStartE2EDuration="2.112111805s" totalImagesPullingTime="608.557729ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:07:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:07:13.109805078 +0000 UTC m=+2418.466562977" watchObservedRunningTime="2026-07-22 01:07:13.112111805 +0000 UTC m=+2418.468869681" Jul 22 01:07:19.109112 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:07:19.109035 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75fe9f-859b994b55-qgsdg" Jul 22 01:07:55.358124 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:07:55.358093 2574 scope.go:117] "RemoveContainer" containerID="ac80f9576fe78ea69c8cf2cb2498bc15f5e783c3ea75daa55bb631040055059b" Jul 22 01:08:36.702628 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:08:36.702578 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-805a5b-bf75h"] Jul 22 01:08:36.706094 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:08:36.706075 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-805a5b-bf75h" Jul 22 01:08:36.708710 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:08:36.708686 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tmggk\"" Jul 22 01:08:36.713875 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:08:36.713851 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-805a5b-bf75h"] Jul 22 01:08:36.791798 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:08:36.791762 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74ae120b-d128-435f-8585-c47bd4dd6061-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-805a5b-bf75h\" (UID: \"74ae120b-d128-435f-8585-c47bd4dd6061\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-805a5b-bf75h" Jul 22 01:08:36.893046 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:08:36.893007 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74ae120b-d128-435f-8585-c47bd4dd6061-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-805a5b-bf75h\" (UID: \"74ae120b-d128-435f-8585-c47bd4dd6061\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-805a5b-bf75h" Jul 22 01:08:36.893664 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:08:36.893645 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74ae120b-d128-435f-8585-c47bd4dd6061-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-805a5b-bf75h\" (UID: \"74ae120b-d128-435f-8585-c47bd4dd6061\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-805a5b-bf75h" Jul 22 01:08:37.018900 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:08:37.018798 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-805a5b-bf75h" Jul 22 01:08:37.167070 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:08:37.167034 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-805a5b-bf75h"] Jul 22 01:08:37.169014 ip-10-0-139-192 kubenswrapper[2574]: W0722 01:08:37.168982 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74ae120b_d128_435f_8585_c47bd4dd6061.slice/crio-4b37f4ba429f0fdf99fea31dc8e6f33448c945d87ed1fba90900380a1245a806 WatchSource:0}: Error finding container 4b37f4ba429f0fdf99fea31dc8e6f33448c945d87ed1fba90900380a1245a806: Status 404 returned error can't find the container with id 4b37f4ba429f0fdf99fea31dc8e6f33448c945d87ed1fba90900380a1245a806 Jul 22 01:08:37.568950 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:08:37.568913 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-805a5b-bf75h" event={"ID":"74ae120b-d128-435f-8585-c47bd4dd6061","Type":"ContainerStarted","Data":"82749bee20a9b813f3ca9f0417bc15f80f82e73de88554b87abe053019e723c2"} Jul 22 01:08:37.569135 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:08:37.568970 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-805a5b-bf75h" event={"ID":"74ae120b-d128-435f-8585-c47bd4dd6061","Type":"ContainerStarted","Data":"4b37f4ba429f0fdf99fea31dc8e6f33448c945d87ed1fba90900380a1245a806"} Jul 22 01:08:37.589296 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:08:37.589200 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-805a5b-bf75h" podStartSLOduration=1.5891844370000001 podStartE2EDuration="1.589184437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:08:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:08:37.587636528 +0000 UTC m=+2502.944394405" watchObservedRunningTime="2026-07-22 01:08:37.589184437 +0000 UTC m=+2502.945942348" Jul 22 01:08:38.576294 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:08:38.576254 2574 generic.go:356] "Generic (PLEG): container finished" podID="74ae120b-d128-435f-8585-c47bd4dd6061" containerID="82749bee20a9b813f3ca9f0417bc15f80f82e73de88554b87abe053019e723c2" exitCode=0 Jul 22 01:08:38.576733 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:08:38.576340 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-805a5b-bf75h" event={"ID":"74ae120b-d128-435f-8585-c47bd4dd6061","Type":"ContainerDied","Data":"82749bee20a9b813f3ca9f0417bc15f80f82e73de88554b87abe053019e723c2"} Jul 22 01:08:39.716085 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:08:39.716061 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-805a5b-bf75h" Jul 22 01:08:39.825245 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:08:39.825208 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74ae120b-d128-435f-8585-c47bd4dd6061-maas-api-service-ca\") pod \"74ae120b-d128-435f-8585-c47bd4dd6061\" (UID: \"74ae120b-d128-435f-8585-c47bd4dd6061\") " Jul 22 01:08:39.825567 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:08:39.825541 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74ae120b-d128-435f-8585-c47bd4dd6061-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "74ae120b-d128-435f-8585-c47bd4dd6061" (UID: "74ae120b-d128-435f-8585-c47bd4dd6061"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 01:08:39.926792 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:08:39.926750 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74ae120b-d128-435f-8585-c47bd4dd6061-maas-api-service-ca\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 01:08:40.588785 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:08:40.588753 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-805a5b-bf75h" Jul 22 01:08:40.588785 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:08:40.588770 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-805a5b-bf75h" event={"ID":"74ae120b-d128-435f-8585-c47bd4dd6061","Type":"ContainerDied","Data":"4b37f4ba429f0fdf99fea31dc8e6f33448c945d87ed1fba90900380a1245a806"} Jul 22 01:08:40.588998 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:08:40.588831 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b37f4ba429f0fdf99fea31dc8e6f33448c945d87ed1fba90900380a1245a806" Jul 22 01:08:47.245716 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:08:47.245627 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-805a5b-5fbc98b88c-kmqmj"] Jul 22 01:08:47.246155 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:08:47.245889 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-805a5b-5fbc98b88c-kmqmj" podUID="e248da55-1c7c-4b6b-a203-c5da3ca3a63d" containerName="maas-api" containerID="cri-o://4337a86e02ce25f697e169edc8e52787169e6f370c52f39badd7af54c3996e58" gracePeriod=30 Jul 22 01:08:47.499272 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:08:47.499212 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-805a5b-5fbc98b88c-kmqmj" Jul 22 01:08:47.594767 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:08:47.594736 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e248da55-1c7c-4b6b-a203-c5da3ca3a63d-maas-api-tls\") pod \"e248da55-1c7c-4b6b-a203-c5da3ca3a63d\" (UID: \"e248da55-1c7c-4b6b-a203-c5da3ca3a63d\") " Jul 22 01:08:47.594767 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:08:47.594769 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z2rmf\" (UniqueName: \"kubernetes.io/projected/e248da55-1c7c-4b6b-a203-c5da3ca3a63d-kube-api-access-z2rmf\") pod \"e248da55-1c7c-4b6b-a203-c5da3ca3a63d\" (UID: \"e248da55-1c7c-4b6b-a203-c5da3ca3a63d\") " Jul 22 01:08:47.595014 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:08:47.594794 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e248da55-1c7c-4b6b-a203-c5da3ca3a63d-openshift-service-ca\") pod \"e248da55-1c7c-4b6b-a203-c5da3ca3a63d\" (UID: \"e248da55-1c7c-4b6b-a203-c5da3ca3a63d\") " Jul 22 01:08:47.595239 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:08:47.595201 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e248da55-1c7c-4b6b-a203-c5da3ca3a63d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e248da55-1c7c-4b6b-a203-c5da3ca3a63d" (UID: "e248da55-1c7c-4b6b-a203-c5da3ca3a63d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 01:08:47.596693 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:08:47.596667 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e248da55-1c7c-4b6b-a203-c5da3ca3a63d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e248da55-1c7c-4b6b-a203-c5da3ca3a63d" (UID: "e248da55-1c7c-4b6b-a203-c5da3ca3a63d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 01:08:47.596801 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:08:47.596748 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e248da55-1c7c-4b6b-a203-c5da3ca3a63d-kube-api-access-z2rmf" (OuterVolumeSpecName: "kube-api-access-z2rmf") pod "e248da55-1c7c-4b6b-a203-c5da3ca3a63d" (UID: "e248da55-1c7c-4b6b-a203-c5da3ca3a63d"). InnerVolumeSpecName "kube-api-access-z2rmf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:08:47.631886 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:08:47.631852 2574 generic.go:356] "Generic (PLEG): container finished" podID="e248da55-1c7c-4b6b-a203-c5da3ca3a63d" containerID="4337a86e02ce25f697e169edc8e52787169e6f370c52f39badd7af54c3996e58" exitCode=0 Jul 22 01:08:47.632035 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:08:47.631903 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-805a5b-5fbc98b88c-kmqmj" Jul 22 01:08:47.632035 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:08:47.631938 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-805a5b-5fbc98b88c-kmqmj" event={"ID":"e248da55-1c7c-4b6b-a203-c5da3ca3a63d","Type":"ContainerDied","Data":"4337a86e02ce25f697e169edc8e52787169e6f370c52f39badd7af54c3996e58"} Jul 22 01:08:47.632035 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:08:47.631994 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-805a5b-5fbc98b88c-kmqmj" event={"ID":"e248da55-1c7c-4b6b-a203-c5da3ca3a63d","Type":"ContainerDied","Data":"dccfa93aea51111e6e50da5d2ffd473e74460f08a86763ccd0cb8c48c3d62ea9"} Jul 22 01:08:47.632035 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:08:47.632016 2574 scope.go:117] "RemoveContainer" containerID="4337a86e02ce25f697e169edc8e52787169e6f370c52f39badd7af54c3996e58" Jul 22 01:08:47.641710 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:08:47.641696 2574 scope.go:117] "RemoveContainer" containerID="4337a86e02ce25f697e169edc8e52787169e6f370c52f39badd7af54c3996e58" Jul 22 01:08:47.641935 ip-10-0-139-192 kubenswrapper[2574]: E0722 01:08:47.641917 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4337a86e02ce25f697e169edc8e52787169e6f370c52f39badd7af54c3996e58\": container with ID starting with 4337a86e02ce25f697e169edc8e52787169e6f370c52f39badd7af54c3996e58 not found: ID does not exist" containerID="4337a86e02ce25f697e169edc8e52787169e6f370c52f39badd7af54c3996e58" Jul 22 01:08:47.641988 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:08:47.641942 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4337a86e02ce25f697e169edc8e52787169e6f370c52f39badd7af54c3996e58"} err="failed to get container status \"4337a86e02ce25f697e169edc8e52787169e6f370c52f39badd7af54c3996e58\": rpc error: code = NotFound desc = could not find container \"4337a86e02ce25f697e169edc8e52787169e6f370c52f39badd7af54c3996e58\": container with ID starting with 4337a86e02ce25f697e169edc8e52787169e6f370c52f39badd7af54c3996e58 not found: ID does not exist" Jul 22 01:08:47.656276 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:08:47.656256 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-805a5b-5fbc98b88c-kmqmj"] Jul 22 01:08:47.663360 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:08:47.663337 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-805a5b-5fbc98b88c-kmqmj"] Jul 22 01:08:47.696034 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:08:47.696011 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e248da55-1c7c-4b6b-a203-c5da3ca3a63d-maas-api-tls\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 01:08:47.696034 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:08:47.696032 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2rmf\" (UniqueName: \"kubernetes.io/projected/e248da55-1c7c-4b6b-a203-c5da3ca3a63d-kube-api-access-z2rmf\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 01:08:47.696148 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:08:47.696043 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e248da55-1c7c-4b6b-a203-c5da3ca3a63d-openshift-service-ca\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 01:08:49.088237 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:08:49.088200 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e248da55-1c7c-4b6b-a203-c5da3ca3a63d" path="/var/lib/kubelet/pods/e248da55-1c7c-4b6b-a203-c5da3ca3a63d/volumes" Jul 22 01:08:51.796741 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:08:51.796684 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-805a5b-bf75h"] Jul 22 01:08:51.798865 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:08:51.798844 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-805a5b-bf75h"] Jul 22 01:08:53.089138 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:08:53.089102 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74ae120b-d128-435f-8585-c47bd4dd6061" path="/var/lib/kubelet/pods/74ae120b-d128-435f-8585-c47bd4dd6061/volumes" Jul 22 01:08:58.805261 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:08:58.805227 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-75fe9f-zlgwl"] Jul 22 01:08:58.805654 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:08:58.805574 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74ae120b-d128-435f-8585-c47bd4dd6061" containerName="revoke-keys" Jul 22 01:08:58.805654 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:08:58.805586 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="74ae120b-d128-435f-8585-c47bd4dd6061" containerName="revoke-keys" Jul 22 01:08:58.805654 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:08:58.805615 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e248da55-1c7c-4b6b-a203-c5da3ca3a63d" containerName="maas-api" Jul 22 01:08:58.805654 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:08:58.805622 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e248da55-1c7c-4b6b-a203-c5da3ca3a63d" containerName="maas-api" Jul 22 01:08:58.805789 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:08:58.805684 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="74ae120b-d128-435f-8585-c47bd4dd6061" containerName="revoke-keys" Jul 22 01:08:58.805789 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:08:58.805691 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="e248da55-1c7c-4b6b-a203-c5da3ca3a63d" containerName="maas-api" Jul 22 01:08:58.810107 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:08:58.810090 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-75fe9f-zlgwl" Jul 22 01:08:58.813173 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:08:58.813146 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tmggk\"" Jul 22 01:08:58.816105 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:08:58.816079 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-75fe9f-zlgwl"] Jul 22 01:08:58.898365 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:08:58.898314 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86748d91-8116-466a-816b-5b456aa6c27c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-75fe9f-zlgwl\" (UID: \"86748d91-8116-466a-816b-5b456aa6c27c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-75fe9f-zlgwl" Jul 22 01:08:58.999059 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:08:58.999020 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86748d91-8116-466a-816b-5b456aa6c27c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-75fe9f-zlgwl\" (UID: \"86748d91-8116-466a-816b-5b456aa6c27c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-75fe9f-zlgwl" Jul 22 01:08:58.999696 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:08:58.999674 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86748d91-8116-466a-816b-5b456aa6c27c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-75fe9f-zlgwl\" (UID: \"86748d91-8116-466a-816b-5b456aa6c27c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-75fe9f-zlgwl" Jul 22 01:08:59.121952 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:08:59.121870 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-75fe9f-zlgwl" Jul 22 01:08:59.268620 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:08:59.268577 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-75fe9f-zlgwl"] Jul 22 01:08:59.269885 ip-10-0-139-192 kubenswrapper[2574]: W0722 01:08:59.269862 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86748d91_8116_466a_816b_5b456aa6c27c.slice/crio-791df8ac44344385cbddfd17078ce5fc3dbda85f7f01643ae7aa0505e8e96508 WatchSource:0}: Error finding container 791df8ac44344385cbddfd17078ce5fc3dbda85f7f01643ae7aa0505e8e96508: Status 404 returned error can't find the container with id 791df8ac44344385cbddfd17078ce5fc3dbda85f7f01643ae7aa0505e8e96508 Jul 22 01:08:59.700410 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:08:59.700322 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-75fe9f-zlgwl" event={"ID":"86748d91-8116-466a-816b-5b456aa6c27c","Type":"ContainerStarted","Data":"66d0e9a0ecc35a1c52e64aec6b4db4c0748cf2ea0a89d1cff4d14009aa0ef7fe"} Jul 22 01:08:59.700410 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:08:59.700375 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-75fe9f-zlgwl" event={"ID":"86748d91-8116-466a-816b-5b456aa6c27c","Type":"ContainerStarted","Data":"791df8ac44344385cbddfd17078ce5fc3dbda85f7f01643ae7aa0505e8e96508"} Jul 22 01:08:59.718272 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:08:59.718221 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-75fe9f-zlgwl" podStartSLOduration=1.71820444 podStartE2EDuration="1.71820444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:08:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:08:59.7157075 +0000 UTC m=+2525.072465376" watchObservedRunningTime="2026-07-22 01:08:59.71820444 +0000 UTC m=+2525.074962316" Jul 22 01:09:00.707203 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:00.707159 2574 generic.go:356] "Generic (PLEG): container finished" podID="86748d91-8116-466a-816b-5b456aa6c27c" containerID="66d0e9a0ecc35a1c52e64aec6b4db4c0748cf2ea0a89d1cff4d14009aa0ef7fe" exitCode=0 Jul 22 01:09:00.707666 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:00.707241 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-75fe9f-zlgwl" event={"ID":"86748d91-8116-466a-816b-5b456aa6c27c","Type":"ContainerDied","Data":"66d0e9a0ecc35a1c52e64aec6b4db4c0748cf2ea0a89d1cff4d14009aa0ef7fe"} Jul 22 01:09:01.848754 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:01.848729 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-75fe9f-zlgwl" Jul 22 01:09:01.927015 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:01.926982 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86748d91-8116-466a-816b-5b456aa6c27c-maas-api-service-ca\") pod \"86748d91-8116-466a-816b-5b456aa6c27c\" (UID: \"86748d91-8116-466a-816b-5b456aa6c27c\") " Jul 22 01:09:01.927382 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:01.927357 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86748d91-8116-466a-816b-5b456aa6c27c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "86748d91-8116-466a-816b-5b456aa6c27c" (UID: "86748d91-8116-466a-816b-5b456aa6c27c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 01:09:02.028450 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:02.028368 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86748d91-8116-466a-816b-5b456aa6c27c-maas-api-service-ca\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 01:09:02.721099 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:02.721058 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-75fe9f-zlgwl" event={"ID":"86748d91-8116-466a-816b-5b456aa6c27c","Type":"ContainerDied","Data":"791df8ac44344385cbddfd17078ce5fc3dbda85f7f01643ae7aa0505e8e96508"} Jul 22 01:09:02.721099 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:02.721083 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-75fe9f-zlgwl" Jul 22 01:09:02.721324 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:02.721143 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="791df8ac44344385cbddfd17078ce5fc3dbda85f7f01643ae7aa0505e8e96508" Jul 22 01:09:09.436371 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:09.436327 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-75fe9f-859b994b55-qgsdg"] Jul 22 01:09:09.436876 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:09.436540 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75fe9f-859b994b55-qgsdg" podUID="1f453aee-4f16-40a0-b86a-9f68815293f1" containerName="maas-api" containerID="cri-o://ef07c3bb288feeb5ce6ec6c50a99c41c2f40254710a34cdf353a4c48393219f0" gracePeriod=30 Jul 22 01:09:09.686018 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:09.685988 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75fe9f-859b994b55-qgsdg" Jul 22 01:09:09.759482 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:09.759399 2574 generic.go:356] "Generic (PLEG): container finished" podID="1f453aee-4f16-40a0-b86a-9f68815293f1" containerID="ef07c3bb288feeb5ce6ec6c50a99c41c2f40254710a34cdf353a4c48393219f0" exitCode=0 Jul 22 01:09:09.759482 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:09.759460 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75fe9f-859b994b55-qgsdg" Jul 22 01:09:09.759741 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:09.759461 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75fe9f-859b994b55-qgsdg" event={"ID":"1f453aee-4f16-40a0-b86a-9f68815293f1","Type":"ContainerDied","Data":"ef07c3bb288feeb5ce6ec6c50a99c41c2f40254710a34cdf353a4c48393219f0"} Jul 22 01:09:09.759741 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:09.759653 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75fe9f-859b994b55-qgsdg" event={"ID":"1f453aee-4f16-40a0-b86a-9f68815293f1","Type":"ContainerDied","Data":"fced2213398e4fd2983c00c97a9964c1f0edbef5f799ff669c93c56ca7842b2b"} Jul 22 01:09:09.759741 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:09.759694 2574 scope.go:117] "RemoveContainer" containerID="ef07c3bb288feeb5ce6ec6c50a99c41c2f40254710a34cdf353a4c48393219f0" Jul 22 01:09:09.768535 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:09.768518 2574 scope.go:117] "RemoveContainer" containerID="ef07c3bb288feeb5ce6ec6c50a99c41c2f40254710a34cdf353a4c48393219f0" Jul 22 01:09:09.768789 ip-10-0-139-192 kubenswrapper[2574]: E0722 01:09:09.768769 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef07c3bb288feeb5ce6ec6c50a99c41c2f40254710a34cdf353a4c48393219f0\": container with ID starting with ef07c3bb288feeb5ce6ec6c50a99c41c2f40254710a34cdf353a4c48393219f0 not found: ID does not exist" containerID="ef07c3bb288feeb5ce6ec6c50a99c41c2f40254710a34cdf353a4c48393219f0" Jul 22 01:09:09.768841 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:09.768797 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef07c3bb288feeb5ce6ec6c50a99c41c2f40254710a34cdf353a4c48393219f0"} err="failed to get container status \"ef07c3bb288feeb5ce6ec6c50a99c41c2f40254710a34cdf353a4c48393219f0\": rpc error: code = NotFound desc = could not find container \"ef07c3bb288feeb5ce6ec6c50a99c41c2f40254710a34cdf353a4c48393219f0\": container with ID starting with ef07c3bb288feeb5ce6ec6c50a99c41c2f40254710a34cdf353a4c48393219f0 not found: ID does not exist" Jul 22 01:09:09.800973 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:09.800933 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-24pzq\" (UniqueName: \"kubernetes.io/projected/1f453aee-4f16-40a0-b86a-9f68815293f1-kube-api-access-24pzq\") pod \"1f453aee-4f16-40a0-b86a-9f68815293f1\" (UID: \"1f453aee-4f16-40a0-b86a-9f68815293f1\") " Jul 22 01:09:09.801121 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:09.800992 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f453aee-4f16-40a0-b86a-9f68815293f1-maas-api-tls\") pod \"1f453aee-4f16-40a0-b86a-9f68815293f1\" (UID: \"1f453aee-4f16-40a0-b86a-9f68815293f1\") " Jul 22 01:09:09.801121 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:09.801066 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f453aee-4f16-40a0-b86a-9f68815293f1-openshift-service-ca\") pod \"1f453aee-4f16-40a0-b86a-9f68815293f1\" (UID: \"1f453aee-4f16-40a0-b86a-9f68815293f1\") " Jul 22 01:09:09.801428 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:09.801405 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f453aee-4f16-40a0-b86a-9f68815293f1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1f453aee-4f16-40a0-b86a-9f68815293f1" (UID: "1f453aee-4f16-40a0-b86a-9f68815293f1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 01:09:09.803103 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:09.803081 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f453aee-4f16-40a0-b86a-9f68815293f1-kube-api-access-24pzq" (OuterVolumeSpecName: "kube-api-access-24pzq") pod "1f453aee-4f16-40a0-b86a-9f68815293f1" (UID: "1f453aee-4f16-40a0-b86a-9f68815293f1"). InnerVolumeSpecName "kube-api-access-24pzq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:09:09.803198 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:09.803109 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f453aee-4f16-40a0-b86a-9f68815293f1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1f453aee-4f16-40a0-b86a-9f68815293f1" (UID: "1f453aee-4f16-40a0-b86a-9f68815293f1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 01:09:09.902584 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:09.902547 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24pzq\" (UniqueName: \"kubernetes.io/projected/1f453aee-4f16-40a0-b86a-9f68815293f1-kube-api-access-24pzq\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 01:09:09.902584 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:09.902579 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f453aee-4f16-40a0-b86a-9f68815293f1-maas-api-tls\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 01:09:09.902584 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:09.902588 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f453aee-4f16-40a0-b86a-9f68815293f1-openshift-service-ca\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 01:09:10.088876 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:10.088840 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-75fe9f-859b994b55-qgsdg"] Jul 22 01:09:10.092450 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:10.092423 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-75fe9f-859b994b55-qgsdg"] Jul 22 01:09:11.089737 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:11.089706 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f453aee-4f16-40a0-b86a-9f68815293f1" path="/var/lib/kubelet/pods/1f453aee-4f16-40a0-b86a-9f68815293f1/volumes" Jul 22 01:09:13.911846 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:13.911811 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-75fe9f-zlgwl"] Jul 22 01:09:13.918541 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:13.918513 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-75fe9f-zlgwl"] Jul 22 01:09:15.097791 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:15.097755 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86748d91-8116-466a-816b-5b456aa6c27c" path="/var/lib/kubelet/pods/86748d91-8116-466a-816b-5b456aa6c27c/volumes" Jul 22 01:09:25.091081 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:25.091047 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-659cfa-5b55489d47-glxgj"] Jul 22 01:09:25.093550 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:25.091548 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f453aee-4f16-40a0-b86a-9f68815293f1" containerName="maas-api" Jul 22 01:09:25.093550 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:25.091563 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f453aee-4f16-40a0-b86a-9f68815293f1" containerName="maas-api" Jul 22 01:09:25.093550 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:25.091593 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86748d91-8116-466a-816b-5b456aa6c27c" containerName="revoke-keys" Jul 22 01:09:25.093550 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:25.091616 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="86748d91-8116-466a-816b-5b456aa6c27c" containerName="revoke-keys" Jul 22 01:09:25.093550 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:25.091704 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="86748d91-8116-466a-816b-5b456aa6c27c" containerName="revoke-keys" Jul 22 01:09:25.093550 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:25.091723 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f453aee-4f16-40a0-b86a-9f68815293f1" containerName="maas-api" Jul 22 01:09:25.094984 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:25.094969 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-659cfa-5b55489d47-glxgj" Jul 22 01:09:25.099314 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:25.099282 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-659cfa-serving-cert\"" Jul 22 01:09:25.108396 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:25.108375 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-659cfa-5b55489d47-glxgj"] Jul 22 01:09:25.126233 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:25.126205 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-298jl\" (UniqueName: \"kubernetes.io/projected/42ccd28c-199d-494d-a597-622b21248b79-kube-api-access-298jl\") pod \"maas-api-e2e-api-a-659cfa-5b55489d47-glxgj\" (UID: \"42ccd28c-199d-494d-a597-622b21248b79\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-659cfa-5b55489d47-glxgj" Jul 22 01:09:25.126378 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:25.126260 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42ccd28c-199d-494d-a597-622b21248b79-maas-api-tls\") pod \"maas-api-e2e-api-a-659cfa-5b55489d47-glxgj\" (UID: \"42ccd28c-199d-494d-a597-622b21248b79\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-659cfa-5b55489d47-glxgj" Jul 22 01:09:25.126445 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:25.126400 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42ccd28c-199d-494d-a597-622b21248b79-openshift-service-ca\") pod \"maas-api-e2e-api-a-659cfa-5b55489d47-glxgj\" (UID: \"42ccd28c-199d-494d-a597-622b21248b79\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-659cfa-5b55489d47-glxgj" Jul 22 01:09:25.227869 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:25.227832 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42ccd28c-199d-494d-a597-622b21248b79-openshift-service-ca\") pod \"maas-api-e2e-api-a-659cfa-5b55489d47-glxgj\" (UID: \"42ccd28c-199d-494d-a597-622b21248b79\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-659cfa-5b55489d47-glxgj" Jul 22 01:09:25.228049 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:25.227937 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-298jl\" (UniqueName: \"kubernetes.io/projected/42ccd28c-199d-494d-a597-622b21248b79-kube-api-access-298jl\") pod \"maas-api-e2e-api-a-659cfa-5b55489d47-glxgj\" (UID: \"42ccd28c-199d-494d-a597-622b21248b79\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-659cfa-5b55489d47-glxgj" Jul 22 01:09:25.228049 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:25.227981 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42ccd28c-199d-494d-a597-622b21248b79-maas-api-tls\") pod \"maas-api-e2e-api-a-659cfa-5b55489d47-glxgj\" (UID: \"42ccd28c-199d-494d-a597-622b21248b79\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-659cfa-5b55489d47-glxgj" Jul 22 01:09:25.228177 ip-10-0-139-192 kubenswrapper[2574]: E0722 01:09:25.228102 2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-659cfa-serving-cert: secret "maas-api-e2e-api-a-659cfa-serving-cert" not found Jul 22 01:09:25.228244 ip-10-0-139-192 kubenswrapper[2574]: E0722 01:09:25.228178 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42ccd28c-199d-494d-a597-622b21248b79-maas-api-tls podName:42ccd28c-199d-494d-a597-622b21248b79 nodeName:}" failed. No retries permitted until 2026-07-22 01:09:25.728158058 +0000 UTC m=+2551.084915917 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/42ccd28c-199d-494d-a597-622b21248b79-maas-api-tls") pod "maas-api-e2e-api-a-659cfa-5b55489d47-glxgj" (UID: "42ccd28c-199d-494d-a597-622b21248b79") : secret "maas-api-e2e-api-a-659cfa-serving-cert" not found Jul 22 01:09:25.228706 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:25.228681 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42ccd28c-199d-494d-a597-622b21248b79-openshift-service-ca\") pod \"maas-api-e2e-api-a-659cfa-5b55489d47-glxgj\" (UID: \"42ccd28c-199d-494d-a597-622b21248b79\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-659cfa-5b55489d47-glxgj" Jul 22 01:09:25.246589 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:25.246560 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-298jl\" (UniqueName: \"kubernetes.io/projected/42ccd28c-199d-494d-a597-622b21248b79-kube-api-access-298jl\") pod \"maas-api-e2e-api-a-659cfa-5b55489d47-glxgj\" (UID: \"42ccd28c-199d-494d-a597-622b21248b79\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-659cfa-5b55489d47-glxgj" Jul 22 01:09:25.732970 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:25.732934 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42ccd28c-199d-494d-a597-622b21248b79-maas-api-tls\") pod \"maas-api-e2e-api-a-659cfa-5b55489d47-glxgj\" (UID: \"42ccd28c-199d-494d-a597-622b21248b79\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-659cfa-5b55489d47-glxgj" Jul 22 01:09:25.735337 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:25.735306 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42ccd28c-199d-494d-a597-622b21248b79-maas-api-tls\") pod \"maas-api-e2e-api-a-659cfa-5b55489d47-glxgj\" (UID: \"42ccd28c-199d-494d-a597-622b21248b79\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-659cfa-5b55489d47-glxgj" Jul 22 01:09:26.006469 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:26.006372 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-659cfa-5b55489d47-glxgj" Jul 22 01:09:26.160351 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:26.160324 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-659cfa-5b55489d47-glxgj"] Jul 22 01:09:26.161467 ip-10-0-139-192 kubenswrapper[2574]: W0722 01:09:26.161438 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42ccd28c_199d_494d_a597_622b21248b79.slice/crio-f27e959f4dd78a1169e50a3cfb800fa3fa392c7f6aa7097ef522c18e566de0f9 WatchSource:0}: Error finding container f27e959f4dd78a1169e50a3cfb800fa3fa392c7f6aa7097ef522c18e566de0f9: Status 404 returned error can't find the container with id f27e959f4dd78a1169e50a3cfb800fa3fa392c7f6aa7097ef522c18e566de0f9 Jul 22 01:09:26.162811 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:26.162786 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 01:09:26.853811 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:26.853776 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-659cfa-5b55489d47-glxgj" event={"ID":"42ccd28c-199d-494d-a597-622b21248b79","Type":"ContainerStarted","Data":"10db92ccc14f33da4a58a05b009a3223a1cc775376f5a7e40fb2085566280326"} Jul 22 01:09:26.853970 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:26.853841 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-659cfa-5b55489d47-glxgj" Jul 22 01:09:26.853970 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:26.853864 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-659cfa-5b55489d47-glxgj" event={"ID":"42ccd28c-199d-494d-a597-622b21248b79","Type":"ContainerStarted","Data":"f27e959f4dd78a1169e50a3cfb800fa3fa392c7f6aa7097ef522c18e566de0f9"} Jul 22 01:09:26.879575 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:26.879524 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-659cfa-5b55489d47-glxgj" podStartSLOduration=1.2964407900000001 podStartE2EDuration="1.879508266s" totalImagesPullingTime="583.067476ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:09:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:09:26.879170066 +0000 UTC m=+2552.235927962" watchObservedRunningTime="2026-07-22 01:09:26.879508266 +0000 UTC m=+2552.236266212" Jul 22 01:09:32.865885 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:32.865856 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-659cfa-5b55489d47-glxgj" Jul 22 01:09:33.060757 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:33.060719 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-956e89-78c869f968-ppch4"] Jul 22 01:09:33.064821 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:33.064800 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-956e89-78c869f968-ppch4" Jul 22 01:09:33.067384 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:33.067360 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-956e89-serving-cert\"" Jul 22 01:09:33.075501 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:33.075470 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-956e89-78c869f968-ppch4"] Jul 22 01:09:33.203200 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:33.203162 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/348ba016-e0b6-48e3-9bc9-099ad3ee0eb6-maas-api-tls\") pod \"maas-api-e2e-api-b-956e89-78c869f968-ppch4\" (UID: \"348ba016-e0b6-48e3-9bc9-099ad3ee0eb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-956e89-78c869f968-ppch4" Jul 22 01:09:33.203399 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:33.203237 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6c25\" (UniqueName: \"kubernetes.io/projected/348ba016-e0b6-48e3-9bc9-099ad3ee0eb6-kube-api-access-r6c25\") pod \"maas-api-e2e-api-b-956e89-78c869f968-ppch4\" (UID: \"348ba016-e0b6-48e3-9bc9-099ad3ee0eb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-956e89-78c869f968-ppch4" Jul 22 01:09:33.203399 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:33.203313 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/348ba016-e0b6-48e3-9bc9-099ad3ee0eb6-openshift-service-ca\") pod \"maas-api-e2e-api-b-956e89-78c869f968-ppch4\" (UID: \"348ba016-e0b6-48e3-9bc9-099ad3ee0eb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-956e89-78c869f968-ppch4" Jul 22 01:09:33.303827 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:33.303796 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/348ba016-e0b6-48e3-9bc9-099ad3ee0eb6-openshift-service-ca\") pod \"maas-api-e2e-api-b-956e89-78c869f968-ppch4\" (UID: \"348ba016-e0b6-48e3-9bc9-099ad3ee0eb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-956e89-78c869f968-ppch4" Jul 22 01:09:33.304007 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:33.303862 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/348ba016-e0b6-48e3-9bc9-099ad3ee0eb6-maas-api-tls\") pod \"maas-api-e2e-api-b-956e89-78c869f968-ppch4\" (UID: \"348ba016-e0b6-48e3-9bc9-099ad3ee0eb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-956e89-78c869f968-ppch4" Jul 22 01:09:33.304007 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:33.303896 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6c25\" (UniqueName: \"kubernetes.io/projected/348ba016-e0b6-48e3-9bc9-099ad3ee0eb6-kube-api-access-r6c25\") pod \"maas-api-e2e-api-b-956e89-78c869f968-ppch4\" (UID: \"348ba016-e0b6-48e3-9bc9-099ad3ee0eb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-956e89-78c869f968-ppch4" Jul 22 01:09:33.304007 ip-10-0-139-192 kubenswrapper[2574]: E0722 01:09:33.303991 2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-956e89-serving-cert: secret "maas-api-e2e-api-b-956e89-serving-cert" not found Jul 22 01:09:33.304107 ip-10-0-139-192 kubenswrapper[2574]: E0722 01:09:33.304083 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/348ba016-e0b6-48e3-9bc9-099ad3ee0eb6-maas-api-tls podName:348ba016-e0b6-48e3-9bc9-099ad3ee0eb6 nodeName:}" failed. No retries permitted until 2026-07-22 01:09:33.804045938 +0000 UTC m=+2559.160803799 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/348ba016-e0b6-48e3-9bc9-099ad3ee0eb6-maas-api-tls") pod "maas-api-e2e-api-b-956e89-78c869f968-ppch4" (UID: "348ba016-e0b6-48e3-9bc9-099ad3ee0eb6") : secret "maas-api-e2e-api-b-956e89-serving-cert" not found Jul 22 01:09:33.304576 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:33.304557 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/348ba016-e0b6-48e3-9bc9-099ad3ee0eb6-openshift-service-ca\") pod \"maas-api-e2e-api-b-956e89-78c869f968-ppch4\" (UID: \"348ba016-e0b6-48e3-9bc9-099ad3ee0eb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-956e89-78c869f968-ppch4" Jul 22 01:09:33.330797 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:33.330753 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6c25\" (UniqueName: \"kubernetes.io/projected/348ba016-e0b6-48e3-9bc9-099ad3ee0eb6-kube-api-access-r6c25\") pod \"maas-api-e2e-api-b-956e89-78c869f968-ppch4\" (UID: \"348ba016-e0b6-48e3-9bc9-099ad3ee0eb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-956e89-78c869f968-ppch4" Jul 22 01:09:33.808997 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:33.808961 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/348ba016-e0b6-48e3-9bc9-099ad3ee0eb6-maas-api-tls\") pod \"maas-api-e2e-api-b-956e89-78c869f968-ppch4\" (UID: \"348ba016-e0b6-48e3-9bc9-099ad3ee0eb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-956e89-78c869f968-ppch4" Jul 22 01:09:33.811575 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:33.811550 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/348ba016-e0b6-48e3-9bc9-099ad3ee0eb6-maas-api-tls\") pod \"maas-api-e2e-api-b-956e89-78c869f968-ppch4\" (UID: \"348ba016-e0b6-48e3-9bc9-099ad3ee0eb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-956e89-78c869f968-ppch4" Jul 22 01:09:33.978058 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:33.978014 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-956e89-78c869f968-ppch4" Jul 22 01:09:34.173314 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:34.169151 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-956e89-78c869f968-ppch4"] Jul 22 01:09:34.177949 ip-10-0-139-192 kubenswrapper[2574]: W0722 01:09:34.177919 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod348ba016_e0b6_48e3_9bc9_099ad3ee0eb6.slice/crio-65587ecfaf601de8643dc8366e99ab2e0b02f536b047fcaff9da25803b3e396d WatchSource:0}: Error finding container 65587ecfaf601de8643dc8366e99ab2e0b02f536b047fcaff9da25803b3e396d: Status 404 returned error can't find the container with id 65587ecfaf601de8643dc8366e99ab2e0b02f536b047fcaff9da25803b3e396d Jul 22 01:09:34.904122 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:34.904084 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-956e89-78c869f968-ppch4" event={"ID":"348ba016-e0b6-48e3-9bc9-099ad3ee0eb6","Type":"ContainerStarted","Data":"330f784701e402d4a37ae3a42fb1a10828108824ac98d4be54148d7c3b5f8734"} Jul 22 01:09:34.904324 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:34.904153 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-956e89-78c869f968-ppch4" Jul 22 01:09:34.904324 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:34.904177 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-956e89-78c869f968-ppch4" event={"ID":"348ba016-e0b6-48e3-9bc9-099ad3ee0eb6","Type":"ContainerStarted","Data":"65587ecfaf601de8643dc8366e99ab2e0b02f536b047fcaff9da25803b3e396d"} Jul 22 01:09:34.925343 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:34.925282 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-956e89-78c869f968-ppch4" podStartSLOduration=1.5030568610000001 podStartE2EDuration="1.925261917s" totalImagesPullingTime="422.205056ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:09:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:09:34.923925401 +0000 UTC m=+2560.280683307" watchObservedRunningTime="2026-07-22 01:09:34.925261917 +0000 UTC m=+2560.282019796" Jul 22 01:09:40.915049 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:40.915020 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-956e89-78c869f968-ppch4" Jul 22 01:09:45.553564 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:45.553521 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-659cfa-99898"] Jul 22 01:09:45.557427 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:45.557407 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-659cfa-99898" Jul 22 01:09:45.560272 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:45.560238 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tmggk\"" Jul 22 01:09:45.564200 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:45.564176 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-659cfa-99898"] Jul 22 01:09:45.722144 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:45.722106 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c729e2e-d3c2-47bc-9d94-b53d0cd84ddd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-659cfa-99898\" (UID: \"2c729e2e-d3c2-47bc-9d94-b53d0cd84ddd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-659cfa-99898" Jul 22 01:09:45.822839 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:45.822746 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c729e2e-d3c2-47bc-9d94-b53d0cd84ddd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-659cfa-99898\" (UID: \"2c729e2e-d3c2-47bc-9d94-b53d0cd84ddd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-659cfa-99898" Jul 22 01:09:45.823405 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:45.823384 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c729e2e-d3c2-47bc-9d94-b53d0cd84ddd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-659cfa-99898\" (UID: \"2c729e2e-d3c2-47bc-9d94-b53d0cd84ddd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-659cfa-99898" Jul 22 01:09:45.869567 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:45.869537 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-659cfa-99898" Jul 22 01:09:46.009991 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:46.009965 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-659cfa-99898"] Jul 22 01:09:46.011483 ip-10-0-139-192 kubenswrapper[2574]: W0722 01:09:46.011448 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c729e2e_d3c2_47bc_9d94_b53d0cd84ddd.slice/crio-07862eed71efe2a2aedd637147c8562bdb8ffe85ed1995385db9b098ebaa40bf WatchSource:0}: Error finding container 07862eed71efe2a2aedd637147c8562bdb8ffe85ed1995385db9b098ebaa40bf: Status 404 returned error can't find the container with id 07862eed71efe2a2aedd637147c8562bdb8ffe85ed1995385db9b098ebaa40bf Jul 22 01:09:46.977036 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:46.977000 2574 generic.go:356] "Generic (PLEG): container finished" podID="2c729e2e-d3c2-47bc-9d94-b53d0cd84ddd" containerID="5aeddda6ff27529755e211609253663e292de09600e50eb14ff0b68661dd0536" exitCode=0 Jul 22 01:09:46.977421 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:46.977040 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-659cfa-99898" event={"ID":"2c729e2e-d3c2-47bc-9d94-b53d0cd84ddd","Type":"ContainerDied","Data":"5aeddda6ff27529755e211609253663e292de09600e50eb14ff0b68661dd0536"} Jul 22 01:09:46.977421 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:46.977099 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-659cfa-99898" event={"ID":"2c729e2e-d3c2-47bc-9d94-b53d0cd84ddd","Type":"ContainerStarted","Data":"07862eed71efe2a2aedd637147c8562bdb8ffe85ed1995385db9b098ebaa40bf"} Jul 22 01:09:48.122491 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:48.122454 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-659cfa-99898" Jul 22 01:09:48.247266 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:48.247170 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c729e2e-d3c2-47bc-9d94-b53d0cd84ddd-maas-api-service-ca\") pod \"2c729e2e-d3c2-47bc-9d94-b53d0cd84ddd\" (UID: \"2c729e2e-d3c2-47bc-9d94-b53d0cd84ddd\") " Jul 22 01:09:48.247643 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:48.247591 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c729e2e-d3c2-47bc-9d94-b53d0cd84ddd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2c729e2e-d3c2-47bc-9d94-b53d0cd84ddd" (UID: "2c729e2e-d3c2-47bc-9d94-b53d0cd84ddd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 01:09:48.348037 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:48.347997 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c729e2e-d3c2-47bc-9d94-b53d0cd84ddd-maas-api-service-ca\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 01:09:48.989847 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:48.989813 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-659cfa-99898" event={"ID":"2c729e2e-d3c2-47bc-9d94-b53d0cd84ddd","Type":"ContainerDied","Data":"07862eed71efe2a2aedd637147c8562bdb8ffe85ed1995385db9b098ebaa40bf"} Jul 22 01:09:48.989847 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:48.989851 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-659cfa-99898" Jul 22 01:09:48.990063 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:48.989904 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07862eed71efe2a2aedd637147c8562bdb8ffe85ed1995385db9b098ebaa40bf" Jul 22 01:09:55.393504 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:09:55.393473 2574 scope.go:117] "RemoveContainer" containerID="c0493910d6d5309943943a415b8e2d36eaf9326b979baa3a54a4468cacece44a" Jul 22 01:10:01.746669 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:01.746634 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-659cfa-5b55489d47-glxgj"] Jul 22 01:10:01.747116 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:01.746885 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-659cfa-5b55489d47-glxgj" podUID="42ccd28c-199d-494d-a597-622b21248b79" containerName="maas-api" containerID="cri-o://10db92ccc14f33da4a58a05b009a3223a1cc775376f5a7e40fb2085566280326" gracePeriod=30 Jul 22 01:10:02.005997 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:02.005937 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-659cfa-5b55489d47-glxgj" Jul 22 01:10:02.062635 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:02.062572 2574 generic.go:356] "Generic (PLEG): container finished" podID="42ccd28c-199d-494d-a597-622b21248b79" containerID="10db92ccc14f33da4a58a05b009a3223a1cc775376f5a7e40fb2085566280326" exitCode=0 Jul 22 01:10:02.062635 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:02.062635 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-659cfa-5b55489d47-glxgj" Jul 22 01:10:02.062876 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:02.062635 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-659cfa-5b55489d47-glxgj" event={"ID":"42ccd28c-199d-494d-a597-622b21248b79","Type":"ContainerDied","Data":"10db92ccc14f33da4a58a05b009a3223a1cc775376f5a7e40fb2085566280326"} Jul 22 01:10:02.062876 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:02.062699 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-659cfa-5b55489d47-glxgj" event={"ID":"42ccd28c-199d-494d-a597-622b21248b79","Type":"ContainerDied","Data":"f27e959f4dd78a1169e50a3cfb800fa3fa392c7f6aa7097ef522c18e566de0f9"} Jul 22 01:10:02.062876 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:02.062729 2574 scope.go:117] "RemoveContainer" containerID="10db92ccc14f33da4a58a05b009a3223a1cc775376f5a7e40fb2085566280326" Jul 22 01:10:02.071738 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:02.071722 2574 scope.go:117] "RemoveContainer" containerID="10db92ccc14f33da4a58a05b009a3223a1cc775376f5a7e40fb2085566280326" Jul 22 01:10:02.071989 ip-10-0-139-192 kubenswrapper[2574]: E0722 01:10:02.071970 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10db92ccc14f33da4a58a05b009a3223a1cc775376f5a7e40fb2085566280326\": container with ID starting with 10db92ccc14f33da4a58a05b009a3223a1cc775376f5a7e40fb2085566280326 not found: ID does not exist" containerID="10db92ccc14f33da4a58a05b009a3223a1cc775376f5a7e40fb2085566280326" Jul 22 01:10:02.072032 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:02.071999 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10db92ccc14f33da4a58a05b009a3223a1cc775376f5a7e40fb2085566280326"} err="failed to get container status \"10db92ccc14f33da4a58a05b009a3223a1cc775376f5a7e40fb2085566280326\": rpc error: code = NotFound desc = could not find container \"10db92ccc14f33da4a58a05b009a3223a1cc775376f5a7e40fb2085566280326\": container with ID starting with 10db92ccc14f33da4a58a05b009a3223a1cc775376f5a7e40fb2085566280326 not found: ID does not exist" Jul 22 01:10:02.073208 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:02.073194 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-298jl\" (UniqueName: \"kubernetes.io/projected/42ccd28c-199d-494d-a597-622b21248b79-kube-api-access-298jl\") pod \"42ccd28c-199d-494d-a597-622b21248b79\" (UID: \"42ccd28c-199d-494d-a597-622b21248b79\") " Jul 22 01:10:02.073260 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:02.073222 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42ccd28c-199d-494d-a597-622b21248b79-openshift-service-ca\") pod \"42ccd28c-199d-494d-a597-622b21248b79\" (UID: \"42ccd28c-199d-494d-a597-622b21248b79\") " Jul 22 01:10:02.073330 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:02.073317 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42ccd28c-199d-494d-a597-622b21248b79-maas-api-tls\") pod \"42ccd28c-199d-494d-a597-622b21248b79\" (UID: \"42ccd28c-199d-494d-a597-622b21248b79\") " Jul 22 01:10:02.073659 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:02.073639 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42ccd28c-199d-494d-a597-622b21248b79-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "42ccd28c-199d-494d-a597-622b21248b79" (UID: "42ccd28c-199d-494d-a597-622b21248b79"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 01:10:02.075186 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:02.075161 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42ccd28c-199d-494d-a597-622b21248b79-kube-api-access-298jl" (OuterVolumeSpecName: "kube-api-access-298jl") pod "42ccd28c-199d-494d-a597-622b21248b79" (UID: "42ccd28c-199d-494d-a597-622b21248b79"). InnerVolumeSpecName "kube-api-access-298jl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:10:02.075271 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:02.075255 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42ccd28c-199d-494d-a597-622b21248b79-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "42ccd28c-199d-494d-a597-622b21248b79" (UID: "42ccd28c-199d-494d-a597-622b21248b79"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 01:10:02.174832 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:02.174785 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42ccd28c-199d-494d-a597-622b21248b79-maas-api-tls\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 01:10:02.174832 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:02.174826 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-298jl\" (UniqueName: \"kubernetes.io/projected/42ccd28c-199d-494d-a597-622b21248b79-kube-api-access-298jl\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 01:10:02.174832 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:02.174839 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42ccd28c-199d-494d-a597-622b21248b79-openshift-service-ca\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 01:10:02.386350 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:02.386319 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-659cfa-5b55489d47-glxgj"] Jul 22 01:10:02.390055 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:02.390028 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-659cfa-5b55489d47-glxgj"] Jul 22 01:10:03.088257 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:03.088222 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42ccd28c-199d-494d-a597-622b21248b79" path="/var/lib/kubelet/pods/42ccd28c-199d-494d-a597-622b21248b79/volumes" Jul 22 01:10:05.632329 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:05.632289 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-659cfa-99898"] Jul 22 01:10:05.634428 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:05.634406 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-659cfa-99898"] Jul 22 01:10:07.089362 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:07.089328 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c729e2e-d3c2-47bc-9d94-b53d0cd84ddd" path="/var/lib/kubelet/pods/2c729e2e-d3c2-47bc-9d94-b53d0cd84ddd/volumes" Jul 22 01:10:12.979764 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:12.979729 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-956e89-lv9hk"] Jul 22 01:10:12.980170 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:12.980106 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42ccd28c-199d-494d-a597-622b21248b79" containerName="maas-api" Jul 22 01:10:12.980170 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:12.980117 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="42ccd28c-199d-494d-a597-622b21248b79" containerName="maas-api" Jul 22 01:10:12.980170 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:12.980126 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c729e2e-d3c2-47bc-9d94-b53d0cd84ddd" containerName="revoke-keys" Jul 22 01:10:12.980170 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:12.980131 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c729e2e-d3c2-47bc-9d94-b53d0cd84ddd" containerName="revoke-keys" Jul 22 01:10:12.980307 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:12.980186 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c729e2e-d3c2-47bc-9d94-b53d0cd84ddd" containerName="revoke-keys" Jul 22 01:10:12.980307 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:12.980195 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="42ccd28c-199d-494d-a597-622b21248b79" containerName="maas-api" Jul 22 01:10:12.983243 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:12.983225 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-956e89-lv9hk" Jul 22 01:10:12.986011 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:12.985993 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tmggk\"" Jul 22 01:10:12.991314 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:12.990912 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-956e89-lv9hk"] Jul 22 01:10:13.076485 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:13.076448 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5cf40d1-1e46-4868-8a5e-c8d9255a96c1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-956e89-lv9hk\" (UID: \"b5cf40d1-1e46-4868-8a5e-c8d9255a96c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-956e89-lv9hk" Jul 22 01:10:13.177805 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:13.177769 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5cf40d1-1e46-4868-8a5e-c8d9255a96c1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-956e89-lv9hk\" (UID: \"b5cf40d1-1e46-4868-8a5e-c8d9255a96c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-956e89-lv9hk" Jul 22 01:10:13.178447 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:13.178428 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5cf40d1-1e46-4868-8a5e-c8d9255a96c1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-956e89-lv9hk\" (UID: \"b5cf40d1-1e46-4868-8a5e-c8d9255a96c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-956e89-lv9hk" Jul 22 01:10:13.295101 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:13.295022 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-956e89-lv9hk" Jul 22 01:10:13.441353 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:13.441328 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-956e89-lv9hk"] Jul 22 01:10:13.443550 ip-10-0-139-192 kubenswrapper[2574]: W0722 01:10:13.443527 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5cf40d1_1e46_4868_8a5e_c8d9255a96c1.slice/crio-9d78cd6a93dbb8b2e9e3fb047f8ad0535e820f798d5130dba88af7408fac0aa4 WatchSource:0}: Error finding container 9d78cd6a93dbb8b2e9e3fb047f8ad0535e820f798d5130dba88af7408fac0aa4: Status 404 returned error can't find the container with id 9d78cd6a93dbb8b2e9e3fb047f8ad0535e820f798d5130dba88af7408fac0aa4 Jul 22 01:10:14.130091 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:14.130057 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-956e89-lv9hk" event={"ID":"b5cf40d1-1e46-4868-8a5e-c8d9255a96c1","Type":"ContainerStarted","Data":"e92a3b49e0e5320393d02d70637aebd2559d6b558df7de6546df68611786f38f"} Jul 22 01:10:14.130565 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:14.130115 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-956e89-lv9hk" event={"ID":"b5cf40d1-1e46-4868-8a5e-c8d9255a96c1","Type":"ContainerStarted","Data":"9d78cd6a93dbb8b2e9e3fb047f8ad0535e820f798d5130dba88af7408fac0aa4"} Jul 22 01:10:14.148756 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:14.148704 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-956e89-lv9hk" podStartSLOduration=2.148688511 podStartE2EDuration="2.148688511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:10:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:10:14.146250704 +0000 UTC m=+2599.503008603" watchObservedRunningTime="2026-07-22 01:10:14.148688511 +0000 UTC m=+2599.505446419" Jul 22 01:10:15.136331 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:15.136295 2574 generic.go:356] "Generic (PLEG): container finished" podID="b5cf40d1-1e46-4868-8a5e-c8d9255a96c1" containerID="e92a3b49e0e5320393d02d70637aebd2559d6b558df7de6546df68611786f38f" exitCode=0 Jul 22 01:10:15.136744 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:15.136380 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-956e89-lv9hk" event={"ID":"b5cf40d1-1e46-4868-8a5e-c8d9255a96c1","Type":"ContainerDied","Data":"e92a3b49e0e5320393d02d70637aebd2559d6b558df7de6546df68611786f38f"} Jul 22 01:10:16.275847 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:16.275819 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-956e89-lv9hk" Jul 22 01:10:16.407805 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:16.407711 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5cf40d1-1e46-4868-8a5e-c8d9255a96c1-maas-api-service-ca\") pod \"b5cf40d1-1e46-4868-8a5e-c8d9255a96c1\" (UID: \"b5cf40d1-1e46-4868-8a5e-c8d9255a96c1\") " Jul 22 01:10:16.408058 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:16.408037 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5cf40d1-1e46-4868-8a5e-c8d9255a96c1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b5cf40d1-1e46-4868-8a5e-c8d9255a96c1" (UID: "b5cf40d1-1e46-4868-8a5e-c8d9255a96c1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 01:10:16.508374 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:16.508337 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5cf40d1-1e46-4868-8a5e-c8d9255a96c1-maas-api-service-ca\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 01:10:17.150647 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:17.150591 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-956e89-lv9hk" event={"ID":"b5cf40d1-1e46-4868-8a5e-c8d9255a96c1","Type":"ContainerDied","Data":"9d78cd6a93dbb8b2e9e3fb047f8ad0535e820f798d5130dba88af7408fac0aa4"} Jul 22 01:10:17.150647 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:17.150640 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-956e89-lv9hk" Jul 22 01:10:17.150872 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:17.150663 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d78cd6a93dbb8b2e9e3fb047f8ad0535e820f798d5130dba88af7408fac0aa4" Jul 22 01:10:28.424044 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:28.424008 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-956e89-78c869f968-ppch4"] Jul 22 01:10:28.424585 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:28.424218 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-956e89-78c869f968-ppch4" podUID="348ba016-e0b6-48e3-9bc9-099ad3ee0eb6" containerName="maas-api" containerID="cri-o://330f784701e402d4a37ae3a42fb1a10828108824ac98d4be54148d7c3b5f8734" gracePeriod=30 Jul 22 01:10:28.672665 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:28.672639 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-956e89-78c869f968-ppch4" Jul 22 01:10:28.823029 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:28.822939 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/348ba016-e0b6-48e3-9bc9-099ad3ee0eb6-maas-api-tls\") pod \"348ba016-e0b6-48e3-9bc9-099ad3ee0eb6\" (UID: \"348ba016-e0b6-48e3-9bc9-099ad3ee0eb6\") " Jul 22 01:10:28.823029 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:28.823002 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r6c25\" (UniqueName: \"kubernetes.io/projected/348ba016-e0b6-48e3-9bc9-099ad3ee0eb6-kube-api-access-r6c25\") pod \"348ba016-e0b6-48e3-9bc9-099ad3ee0eb6\" (UID: \"348ba016-e0b6-48e3-9bc9-099ad3ee0eb6\") " Jul 22 01:10:28.823233 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:28.823126 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/348ba016-e0b6-48e3-9bc9-099ad3ee0eb6-openshift-service-ca\") pod \"348ba016-e0b6-48e3-9bc9-099ad3ee0eb6\" (UID: \"348ba016-e0b6-48e3-9bc9-099ad3ee0eb6\") " Jul 22 01:10:28.823486 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:28.823456 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/348ba016-e0b6-48e3-9bc9-099ad3ee0eb6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "348ba016-e0b6-48e3-9bc9-099ad3ee0eb6" (UID: "348ba016-e0b6-48e3-9bc9-099ad3ee0eb6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 01:10:28.825076 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:28.825049 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/348ba016-e0b6-48e3-9bc9-099ad3ee0eb6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "348ba016-e0b6-48e3-9bc9-099ad3ee0eb6" (UID: "348ba016-e0b6-48e3-9bc9-099ad3ee0eb6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 01:10:28.825178 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:28.825123 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/348ba016-e0b6-48e3-9bc9-099ad3ee0eb6-kube-api-access-r6c25" (OuterVolumeSpecName: "kube-api-access-r6c25") pod "348ba016-e0b6-48e3-9bc9-099ad3ee0eb6" (UID: "348ba016-e0b6-48e3-9bc9-099ad3ee0eb6"). InnerVolumeSpecName "kube-api-access-r6c25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:10:28.924857 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:28.924820 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6c25\" (UniqueName: \"kubernetes.io/projected/348ba016-e0b6-48e3-9bc9-099ad3ee0eb6-kube-api-access-r6c25\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 01:10:28.924857 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:28.924851 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/348ba016-e0b6-48e3-9bc9-099ad3ee0eb6-openshift-service-ca\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 01:10:28.924857 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:28.924862 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/348ba016-e0b6-48e3-9bc9-099ad3ee0eb6-maas-api-tls\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 01:10:29.215015 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:29.214979 2574 generic.go:356] "Generic (PLEG): container finished" podID="348ba016-e0b6-48e3-9bc9-099ad3ee0eb6" containerID="330f784701e402d4a37ae3a42fb1a10828108824ac98d4be54148d7c3b5f8734" exitCode=0 Jul 22 01:10:29.215209 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:29.215033 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-956e89-78c869f968-ppch4" Jul 22 01:10:29.215209 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:29.215065 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-956e89-78c869f968-ppch4" event={"ID":"348ba016-e0b6-48e3-9bc9-099ad3ee0eb6","Type":"ContainerDied","Data":"330f784701e402d4a37ae3a42fb1a10828108824ac98d4be54148d7c3b5f8734"} Jul 22 01:10:29.215209 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:29.215123 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-956e89-78c869f968-ppch4" event={"ID":"348ba016-e0b6-48e3-9bc9-099ad3ee0eb6","Type":"ContainerDied","Data":"65587ecfaf601de8643dc8366e99ab2e0b02f536b047fcaff9da25803b3e396d"} Jul 22 01:10:29.215209 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:29.215146 2574 scope.go:117] "RemoveContainer" containerID="330f784701e402d4a37ae3a42fb1a10828108824ac98d4be54148d7c3b5f8734" Jul 22 01:10:29.224190 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:29.224169 2574 scope.go:117] "RemoveContainer" containerID="330f784701e402d4a37ae3a42fb1a10828108824ac98d4be54148d7c3b5f8734" Jul 22 01:10:29.224491 ip-10-0-139-192 kubenswrapper[2574]: E0722 01:10:29.224474 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"330f784701e402d4a37ae3a42fb1a10828108824ac98d4be54148d7c3b5f8734\": container with ID starting with 330f784701e402d4a37ae3a42fb1a10828108824ac98d4be54148d7c3b5f8734 not found: ID does not exist" containerID="330f784701e402d4a37ae3a42fb1a10828108824ac98d4be54148d7c3b5f8734" Jul 22 01:10:29.224559 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:29.224500 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"330f784701e402d4a37ae3a42fb1a10828108824ac98d4be54148d7c3b5f8734"} err="failed to get container status \"330f784701e402d4a37ae3a42fb1a10828108824ac98d4be54148d7c3b5f8734\": rpc error: code = NotFound desc = could not find container \"330f784701e402d4a37ae3a42fb1a10828108824ac98d4be54148d7c3b5f8734\": container with ID starting with 330f784701e402d4a37ae3a42fb1a10828108824ac98d4be54148d7c3b5f8734 not found: ID does not exist" Jul 22 01:10:29.235228 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:29.235201 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-956e89-78c869f968-ppch4"] Jul 22 01:10:29.241092 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:29.241066 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-956e89-78c869f968-ppch4"] Jul 22 01:10:31.089153 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:31.089119 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="348ba016-e0b6-48e3-9bc9-099ad3ee0eb6" path="/var/lib/kubelet/pods/348ba016-e0b6-48e3-9bc9-099ad3ee0eb6/volumes" Jul 22 01:10:33.054808 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:33.054772 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-956e89-lv9hk"] Jul 22 01:10:33.056841 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:33.056818 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-956e89-lv9hk"] Jul 22 01:10:33.093812 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:33.093780 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5cf40d1-1e46-4868-8a5e-c8d9255a96c1" path="/var/lib/kubelet/pods/b5cf40d1-1e46-4868-8a5e-c8d9255a96c1/volumes" Jul 22 01:10:43.484396 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:43.484337 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-e4b9ac-cdddb465b-ns8b9"] Jul 22 01:10:43.484957 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:43.484933 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5cf40d1-1e46-4868-8a5e-c8d9255a96c1" containerName="revoke-keys" Jul 22 01:10:43.484957 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:43.484959 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5cf40d1-1e46-4868-8a5e-c8d9255a96c1" containerName="revoke-keys" Jul 22 01:10:43.485096 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:43.484980 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="348ba016-e0b6-48e3-9bc9-099ad3ee0eb6" containerName="maas-api" Jul 22 01:10:43.485096 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:43.484990 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="348ba016-e0b6-48e3-9bc9-099ad3ee0eb6" containerName="maas-api" Jul 22 01:10:43.485167 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:43.485097 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5cf40d1-1e46-4868-8a5e-c8d9255a96c1" containerName="revoke-keys" Jul 22 01:10:43.485167 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:43.485112 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="348ba016-e0b6-48e3-9bc9-099ad3ee0eb6" containerName="maas-api" Jul 22 01:10:43.488521 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:43.488503 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e4b9ac-cdddb465b-ns8b9" Jul 22 01:10:43.491293 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:43.491267 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-e4b9ac-serving-cert\"" Jul 22 01:10:43.498245 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:43.498221 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-e4b9ac-cdddb465b-ns8b9"] Jul 22 01:10:43.551304 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:43.551273 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cc9aef8-4e20-431f-b3d6-86ec75711985-openshift-service-ca\") pod \"maas-api-e2e-shared-a-e4b9ac-cdddb465b-ns8b9\" (UID: \"8cc9aef8-4e20-431f-b3d6-86ec75711985\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e4b9ac-cdddb465b-ns8b9" Jul 22 01:10:43.551481 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:43.551323 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8cc9aef8-4e20-431f-b3d6-86ec75711985-maas-api-tls\") pod \"maas-api-e2e-shared-a-e4b9ac-cdddb465b-ns8b9\" (UID: \"8cc9aef8-4e20-431f-b3d6-86ec75711985\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e4b9ac-cdddb465b-ns8b9" Jul 22 01:10:43.551481 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:43.551413 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9ppk\" (UniqueName: \"kubernetes.io/projected/8cc9aef8-4e20-431f-b3d6-86ec75711985-kube-api-access-q9ppk\") pod \"maas-api-e2e-shared-a-e4b9ac-cdddb465b-ns8b9\" (UID: \"8cc9aef8-4e20-431f-b3d6-86ec75711985\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e4b9ac-cdddb465b-ns8b9" Jul 22 01:10:43.652216 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:43.652182 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cc9aef8-4e20-431f-b3d6-86ec75711985-openshift-service-ca\") pod \"maas-api-e2e-shared-a-e4b9ac-cdddb465b-ns8b9\" (UID: \"8cc9aef8-4e20-431f-b3d6-86ec75711985\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e4b9ac-cdddb465b-ns8b9" Jul 22 01:10:43.652429 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:43.652231 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8cc9aef8-4e20-431f-b3d6-86ec75711985-maas-api-tls\") pod \"maas-api-e2e-shared-a-e4b9ac-cdddb465b-ns8b9\" (UID: \"8cc9aef8-4e20-431f-b3d6-86ec75711985\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e4b9ac-cdddb465b-ns8b9" Jul 22 01:10:43.652429 ip-10-0-139-192 kubenswrapper[2574]: E0722 01:10:43.652348 2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-e4b9ac-serving-cert: secret "maas-api-e2e-shared-a-e4b9ac-serving-cert" not found Jul 22 01:10:43.652429 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:43.652358 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9ppk\" (UniqueName: \"kubernetes.io/projected/8cc9aef8-4e20-431f-b3d6-86ec75711985-kube-api-access-q9ppk\") pod \"maas-api-e2e-shared-a-e4b9ac-cdddb465b-ns8b9\" (UID: \"8cc9aef8-4e20-431f-b3d6-86ec75711985\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e4b9ac-cdddb465b-ns8b9" Jul 22 01:10:43.652429 ip-10-0-139-192 kubenswrapper[2574]: E0722 01:10:43.652408 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cc9aef8-4e20-431f-b3d6-86ec75711985-maas-api-tls podName:8cc9aef8-4e20-431f-b3d6-86ec75711985 nodeName:}" failed. No retries permitted until 2026-07-22 01:10:44.152393144 +0000 UTC m=+2629.509150999 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8cc9aef8-4e20-431f-b3d6-86ec75711985-maas-api-tls") pod "maas-api-e2e-shared-a-e4b9ac-cdddb465b-ns8b9" (UID: "8cc9aef8-4e20-431f-b3d6-86ec75711985") : secret "maas-api-e2e-shared-a-e4b9ac-serving-cert" not found Jul 22 01:10:43.652875 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:43.652856 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cc9aef8-4e20-431f-b3d6-86ec75711985-openshift-service-ca\") pod \"maas-api-e2e-shared-a-e4b9ac-cdddb465b-ns8b9\" (UID: \"8cc9aef8-4e20-431f-b3d6-86ec75711985\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e4b9ac-cdddb465b-ns8b9" Jul 22 01:10:43.663880 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:43.663849 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9ppk\" (UniqueName: \"kubernetes.io/projected/8cc9aef8-4e20-431f-b3d6-86ec75711985-kube-api-access-q9ppk\") pod \"maas-api-e2e-shared-a-e4b9ac-cdddb465b-ns8b9\" (UID: \"8cc9aef8-4e20-431f-b3d6-86ec75711985\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e4b9ac-cdddb465b-ns8b9" Jul 22 01:10:44.156899 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:44.156859 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8cc9aef8-4e20-431f-b3d6-86ec75711985-maas-api-tls\") pod \"maas-api-e2e-shared-a-e4b9ac-cdddb465b-ns8b9\" (UID: \"8cc9aef8-4e20-431f-b3d6-86ec75711985\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e4b9ac-cdddb465b-ns8b9" Jul 22 01:10:44.159343 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:44.159304 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8cc9aef8-4e20-431f-b3d6-86ec75711985-maas-api-tls\") pod \"maas-api-e2e-shared-a-e4b9ac-cdddb465b-ns8b9\" (UID: \"8cc9aef8-4e20-431f-b3d6-86ec75711985\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e4b9ac-cdddb465b-ns8b9" Jul 22 01:10:44.401553 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:44.401514 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e4b9ac-cdddb465b-ns8b9" Jul 22 01:10:44.541455 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:44.541426 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-e4b9ac-cdddb465b-ns8b9"] Jul 22 01:10:44.542864 ip-10-0-139-192 kubenswrapper[2574]: W0722 01:10:44.542836 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cc9aef8_4e20_431f_b3d6_86ec75711985.slice/crio-1fde6b64dd31de28bfd02f3088e76fec8aa5273a958ed432bf3fe5577024264d WatchSource:0}: Error finding container 1fde6b64dd31de28bfd02f3088e76fec8aa5273a958ed432bf3fe5577024264d: Status 404 returned error can't find the container with id 1fde6b64dd31de28bfd02f3088e76fec8aa5273a958ed432bf3fe5577024264d Jul 22 01:10:45.306173 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:45.306081 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e4b9ac-cdddb465b-ns8b9" event={"ID":"8cc9aef8-4e20-431f-b3d6-86ec75711985","Type":"ContainerStarted","Data":"12b535939e4da167b4836ca37326eadb8632fa4b60c42f679d04c55d3140b150"} Jul 22 01:10:45.306173 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:45.306141 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e4b9ac-cdddb465b-ns8b9" event={"ID":"8cc9aef8-4e20-431f-b3d6-86ec75711985","Type":"ContainerStarted","Data":"1fde6b64dd31de28bfd02f3088e76fec8aa5273a958ed432bf3fe5577024264d"} Jul 22 01:10:45.306364 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:45.306192 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e4b9ac-cdddb465b-ns8b9" Jul 22 01:10:45.328390 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:45.328339 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e4b9ac-cdddb465b-ns8b9" podStartSLOduration=1.941229533 podStartE2EDuration="2.328323101s" totalImagesPullingTime="387.093568ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:10:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:10:45.32609552 +0000 UTC m=+2630.682853399" watchObservedRunningTime="2026-07-22 01:10:45.328323101 +0000 UTC m=+2630.685080977" Jul 22 01:10:51.317480 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:51.317446 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e4b9ac-cdddb465b-ns8b9" Jul 22 01:10:51.423105 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:51.423071 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-4f35c1-864d798c5c-7c6kz"] Jul 22 01:10:51.427074 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:51.427053 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4f35c1-864d798c5c-7c6kz" Jul 22 01:10:51.429730 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:51.429707 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-4f35c1-serving-cert\"" Jul 22 01:10:51.438242 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:51.438216 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-4f35c1-864d798c5c-7c6kz"] Jul 22 01:10:51.522470 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:51.522431 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/906b0c5b-f651-4f3a-9247-fdd4e8a0000a-maas-api-tls\") pod \"maas-api-e2e-shared-b-4f35c1-864d798c5c-7c6kz\" (UID: \"906b0c5b-f651-4f3a-9247-fdd4e8a0000a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4f35c1-864d798c5c-7c6kz" Jul 22 01:10:51.522698 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:51.522482 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/906b0c5b-f651-4f3a-9247-fdd4e8a0000a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-4f35c1-864d798c5c-7c6kz\" (UID: \"906b0c5b-f651-4f3a-9247-fdd4e8a0000a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4f35c1-864d798c5c-7c6kz" Jul 22 01:10:51.522698 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:51.522509 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8z4d6\" (UniqueName: \"kubernetes.io/projected/906b0c5b-f651-4f3a-9247-fdd4e8a0000a-kube-api-access-8z4d6\") pod \"maas-api-e2e-shared-b-4f35c1-864d798c5c-7c6kz\" (UID: \"906b0c5b-f651-4f3a-9247-fdd4e8a0000a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4f35c1-864d798c5c-7c6kz" Jul 22 01:10:51.624117 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:51.624010 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/906b0c5b-f651-4f3a-9247-fdd4e8a0000a-maas-api-tls\") pod \"maas-api-e2e-shared-b-4f35c1-864d798c5c-7c6kz\" (UID: \"906b0c5b-f651-4f3a-9247-fdd4e8a0000a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4f35c1-864d798c5c-7c6kz" Jul 22 01:10:51.624117 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:51.624059 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/906b0c5b-f651-4f3a-9247-fdd4e8a0000a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-4f35c1-864d798c5c-7c6kz\" (UID: \"906b0c5b-f651-4f3a-9247-fdd4e8a0000a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4f35c1-864d798c5c-7c6kz" Jul 22 01:10:51.624117 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:51.624077 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z4d6\" (UniqueName: \"kubernetes.io/projected/906b0c5b-f651-4f3a-9247-fdd4e8a0000a-kube-api-access-8z4d6\") pod \"maas-api-e2e-shared-b-4f35c1-864d798c5c-7c6kz\" (UID: \"906b0c5b-f651-4f3a-9247-fdd4e8a0000a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4f35c1-864d798c5c-7c6kz" Jul 22 01:10:51.624412 ip-10-0-139-192 kubenswrapper[2574]: E0722 01:10:51.624202 2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-4f35c1-serving-cert: secret "maas-api-e2e-shared-b-4f35c1-serving-cert" not found Jul 22 01:10:51.624412 ip-10-0-139-192 kubenswrapper[2574]: E0722 01:10:51.624279 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/906b0c5b-f651-4f3a-9247-fdd4e8a0000a-maas-api-tls podName:906b0c5b-f651-4f3a-9247-fdd4e8a0000a nodeName:}" failed. No retries permitted until 2026-07-22 01:10:52.124257894 +0000 UTC m=+2637.481015761 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/906b0c5b-f651-4f3a-9247-fdd4e8a0000a-maas-api-tls") pod "maas-api-e2e-shared-b-4f35c1-864d798c5c-7c6kz" (UID: "906b0c5b-f651-4f3a-9247-fdd4e8a0000a") : secret "maas-api-e2e-shared-b-4f35c1-serving-cert" not found Jul 22 01:10:51.624905 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:51.624884 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/906b0c5b-f651-4f3a-9247-fdd4e8a0000a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-4f35c1-864d798c5c-7c6kz\" (UID: \"906b0c5b-f651-4f3a-9247-fdd4e8a0000a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4f35c1-864d798c5c-7c6kz" Jul 22 01:10:51.635360 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:51.635330 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z4d6\" (UniqueName: \"kubernetes.io/projected/906b0c5b-f651-4f3a-9247-fdd4e8a0000a-kube-api-access-8z4d6\") pod \"maas-api-e2e-shared-b-4f35c1-864d798c5c-7c6kz\" (UID: \"906b0c5b-f651-4f3a-9247-fdd4e8a0000a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4f35c1-864d798c5c-7c6kz" Jul 22 01:10:52.129510 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:52.129469 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/906b0c5b-f651-4f3a-9247-fdd4e8a0000a-maas-api-tls\") pod \"maas-api-e2e-shared-b-4f35c1-864d798c5c-7c6kz\" (UID: \"906b0c5b-f651-4f3a-9247-fdd4e8a0000a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4f35c1-864d798c5c-7c6kz" Jul 22 01:10:52.132014 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:52.131992 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/906b0c5b-f651-4f3a-9247-fdd4e8a0000a-maas-api-tls\") pod \"maas-api-e2e-shared-b-4f35c1-864d798c5c-7c6kz\" (UID: \"906b0c5b-f651-4f3a-9247-fdd4e8a0000a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4f35c1-864d798c5c-7c6kz" Jul 22 01:10:52.338305 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:52.338269 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4f35c1-864d798c5c-7c6kz" Jul 22 01:10:52.479446 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:52.479418 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-4f35c1-864d798c5c-7c6kz"] Jul 22 01:10:52.480468 ip-10-0-139-192 kubenswrapper[2574]: W0722 01:10:52.480442 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod906b0c5b_f651_4f3a_9247_fdd4e8a0000a.slice/crio-941eabbdd14b9bf34f3c44390df2341c9762d683889fe6439273aedb62260794 WatchSource:0}: Error finding container 941eabbdd14b9bf34f3c44390df2341c9762d683889fe6439273aedb62260794: Status 404 returned error can't find the container with id 941eabbdd14b9bf34f3c44390df2341c9762d683889fe6439273aedb62260794 Jul 22 01:10:53.371405 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:53.371334 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4f35c1-864d798c5c-7c6kz" event={"ID":"906b0c5b-f651-4f3a-9247-fdd4e8a0000a","Type":"ContainerStarted","Data":"941eabbdd14b9bf34f3c44390df2341c9762d683889fe6439273aedb62260794"} Jul 22 01:10:54.378824 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:54.378778 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4f35c1-864d798c5c-7c6kz" event={"ID":"906b0c5b-f651-4f3a-9247-fdd4e8a0000a","Type":"ContainerStarted","Data":"649808778f844980a49c661e6173f3cb132d746ee600c2f1ec6498ebe948ab72"} Jul 22 01:10:54.379208 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:54.378859 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4f35c1-864d798c5c-7c6kz" Jul 22 01:10:54.401485 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:10:54.401436 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4f35c1-864d798c5c-7c6kz" podStartSLOduration=2.317356283 podStartE2EDuration="3.401421487s" totalImagesPullingTime="1.084065204s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:10:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:10:54.399472166 +0000 UTC m=+2639.756230045" watchObservedRunningTime="2026-07-22 01:10:54.401421487 +0000 UTC m=+2639.758179364" Jul 22 01:11:00.391824 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:11:00.391791 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4f35c1-864d798c5c-7c6kz" Jul 22 01:11:55.235774 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:11:55.235701 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-hb69v_bf7b370a-c67c-4c3c-b3ba-7533131eace2/console-operator/1.log" Jul 22 01:11:55.239450 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:11:55.239426 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hr5fn_947d311a-ec4b-42e7-a0fb-ee5edc0caf8d/ovn-acl-logging/0.log" Jul 22 01:11:55.243593 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:11:55.243572 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-hb69v_bf7b370a-c67c-4c3c-b3ba-7533131eace2/console-operator/1.log" Jul 22 01:11:55.247161 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:11:55.247143 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hr5fn_947d311a-ec4b-42e7-a0fb-ee5edc0caf8d/ovn-acl-logging/0.log" Jul 22 01:11:55.440295 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:11:55.440266 2574 scope.go:117] "RemoveContainer" containerID="68a2d0ac28ef13f383661eeb5957f40ecea99e03008f7b65edfb10e8fb602797" Jul 22 01:14:55.477645 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:14:55.477554 2574 scope.go:117] "RemoveContainer" containerID="82749bee20a9b813f3ca9f0417bc15f80f82e73de88554b87abe053019e723c2" Jul 22 01:15:00.139193 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:15:00.139155 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29744715-lbvxc"] Jul 22 01:15:00.144023 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:15:00.144000 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-4f35c1-29744715-sdj6z"] Jul 22 01:15:00.144183 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:15:00.144130 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744715-lbvxc" Jul 22 01:15:00.146843 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:15:00.146824 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tmggk\"" Jul 22 01:15:00.147425 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:15:00.147407 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-e4b9ac-29744715-dj7xw"] Jul 22 01:15:00.147520 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:15:00.147508 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-4f35c1-29744715-sdj6z" Jul 22 01:15:00.150907 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:15:00.150884 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-e4b9ac-29744715-dj7xw" Jul 22 01:15:00.152255 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:15:00.152175 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29744715-lbvxc"] Jul 22 01:15:00.154793 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:15:00.154764 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-4f35c1-29744715-sdj6z"] Jul 22 01:15:00.157336 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:15:00.157314 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744715-lbvxc" Jul 22 01:15:00.157744 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:15:00.157724 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-e4b9ac-29744715-dj7xw"] Jul 22 01:15:00.166975 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:15:00.166952 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-4f35c1-29744715-sdj6z" Jul 22 01:15:00.205830 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:15:00.205804 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-e4b9ac-29744715-dj7xw" Jul 22 01:15:00.322424 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:15:00.322381 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29744715-lbvxc"] Jul 22 01:15:00.327742 ip-10-0-139-192 kubenswrapper[2574]: W0722 01:15:00.327713 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded5d6e1a_5dd8_4898_850e_cc38d385bff7.slice/crio-86ad3850b6abf84fa67ba9a4f280837d3ab667ed8ad077f324101aa76b72b18b WatchSource:0}: Error finding container 86ad3850b6abf84fa67ba9a4f280837d3ab667ed8ad077f324101aa76b72b18b: Status 404 returned error can't find the container with id 86ad3850b6abf84fa67ba9a4f280837d3ab667ed8ad077f324101aa76b72b18b Jul 22 01:15:00.358552 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:15:00.358525 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-4f35c1-29744715-sdj6z"] Jul 22 01:15:00.360594 ip-10-0-139-192 kubenswrapper[2574]: W0722 01:15:00.360567 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91ea4b49_867f_4e27_ad7d_ecbafd505295.slice/crio-feade6770eeb81151556fa6950cbf04ac30068834cf16c6015c3868f885a7628 WatchSource:0}: Error finding container feade6770eeb81151556fa6950cbf04ac30068834cf16c6015c3868f885a7628: Status 404 returned error can't find the container with id feade6770eeb81151556fa6950cbf04ac30068834cf16c6015c3868f885a7628 Jul 22 01:15:00.381719 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:15:00.381698 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-e4b9ac-29744715-dj7xw"] Jul 22 01:15:00.383117 ip-10-0-139-192 kubenswrapper[2574]: W0722 01:15:00.383093 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3023efc7_fe01_47aa_9dd5_a1a03e71668d.slice/crio-27c50b3bc793be7d099463d50f7f72d28c9fbc51c96a3970482af39ca52781ec WatchSource:0}: Error finding container 27c50b3bc793be7d099463d50f7f72d28c9fbc51c96a3970482af39ca52781ec: Status 404 returned error can't find the container with id 27c50b3bc793be7d099463d50f7f72d28c9fbc51c96a3970482af39ca52781ec Jul 22 01:15:00.752490 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:15:00.752456 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-e4b9ac-29744715-dj7xw" event={"ID":"3023efc7-fe01-47aa-9dd5-a1a03e71668d","Type":"ContainerStarted","Data":"f1c9f71a8fba3dd1b420c04351f9fe19879e55ce91ee8109279f16c3b43e6475"} Jul 22 01:15:00.752648 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:15:00.752520 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-e4b9ac-29744715-dj7xw" event={"ID":"3023efc7-fe01-47aa-9dd5-a1a03e71668d","Type":"ContainerStarted","Data":"27c50b3bc793be7d099463d50f7f72d28c9fbc51c96a3970482af39ca52781ec"} Jul 22 01:15:00.754041 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:15:00.754013 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-4f35c1-29744715-sdj6z" event={"ID":"91ea4b49-867f-4e27-ad7d-ecbafd505295","Type":"ContainerStarted","Data":"d4ab90be2ad64e254ea87bff6134b4b4626d2c8f9c55eca0edcc8d44b70044ce"} Jul 22 01:15:00.754148 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:15:00.754061 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-4f35c1-29744715-sdj6z" event={"ID":"91ea4b49-867f-4e27-ad7d-ecbafd505295","Type":"ContainerStarted","Data":"feade6770eeb81151556fa6950cbf04ac30068834cf16c6015c3868f885a7628"} Jul 22 01:15:00.755446 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:15:00.755427 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744715-lbvxc" event={"ID":"ed5d6e1a-5dd8-4898-850e-cc38d385bff7","Type":"ContainerStarted","Data":"7bc459884e01464d544b0362538e186cbbf067a897f7a39572190eecd445751b"} Jul 22 01:15:00.755531 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:15:00.755466 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744715-lbvxc" event={"ID":"ed5d6e1a-5dd8-4898-850e-cc38d385bff7","Type":"ContainerStarted","Data":"86ad3850b6abf84fa67ba9a4f280837d3ab667ed8ad077f324101aa76b72b18b"} Jul 22 01:15:00.769397 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:15:00.769353 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-e4b9ac-29744715-dj7xw" podStartSLOduration=0.769341117 podStartE2EDuration="769.341117ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:15:00.767221459 +0000 UTC m=+2886.123979337" watchObservedRunningTime="2026-07-22 01:15:00.769341117 +0000 UTC m=+2886.126098972" Jul 22 01:15:00.783248 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:15:00.783205 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-4f35c1-29744715-sdj6z" podStartSLOduration=0.783192544 podStartE2EDuration="783.192544ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:15:00.780811492 +0000 UTC m=+2886.137569395" watchObservedRunningTime="2026-07-22 01:15:00.783192544 +0000 UTC m=+2886.139950420" Jul 22 01:15:00.794133 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:15:00.794087 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744715-lbvxc" podStartSLOduration=0.794074688 podStartE2EDuration="794.074688ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:15:00.794007712 +0000 UTC m=+2886.150765613" watchObservedRunningTime="2026-07-22 01:15:00.794074688 +0000 UTC m=+2886.150832564" Jul 22 01:15:01.762990 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:15:01.762953 2574 generic.go:356] "Generic (PLEG): container finished" podID="3023efc7-fe01-47aa-9dd5-a1a03e71668d" containerID="f1c9f71a8fba3dd1b420c04351f9fe19879e55ce91ee8109279f16c3b43e6475" exitCode=0 Jul 22 01:15:01.763434 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:15:01.763033 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-e4b9ac-29744715-dj7xw" event={"ID":"3023efc7-fe01-47aa-9dd5-a1a03e71668d","Type":"ContainerDied","Data":"f1c9f71a8fba3dd1b420c04351f9fe19879e55ce91ee8109279f16c3b43e6475"} Jul 22 01:15:01.764685 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:15:01.764648 2574 generic.go:356] "Generic (PLEG): container finished" podID="91ea4b49-867f-4e27-ad7d-ecbafd505295" containerID="d4ab90be2ad64e254ea87bff6134b4b4626d2c8f9c55eca0edcc8d44b70044ce" exitCode=0 Jul 22 01:15:01.764826 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:15:01.764676 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-4f35c1-29744715-sdj6z" event={"ID":"91ea4b49-867f-4e27-ad7d-ecbafd505295","Type":"ContainerDied","Data":"d4ab90be2ad64e254ea87bff6134b4b4626d2c8f9c55eca0edcc8d44b70044ce"} Jul 22 01:15:01.766130 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:15:01.766110 2574 generic.go:356] "Generic (PLEG): container finished" podID="ed5d6e1a-5dd8-4898-850e-cc38d385bff7" containerID="7bc459884e01464d544b0362538e186cbbf067a897f7a39572190eecd445751b" exitCode=0 Jul 22 01:15:01.766258 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:15:01.766173 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744715-lbvxc" event={"ID":"ed5d6e1a-5dd8-4898-850e-cc38d385bff7","Type":"ContainerDied","Data":"7bc459884e01464d544b0362538e186cbbf067a897f7a39572190eecd445751b"} Jul 22 01:15:02.903795 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:15:02.903765 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-e4b9ac-29744715-dj7xw" Jul 22 01:15:02.946126 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:15:02.946099 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744715-lbvxc" Jul 22 01:15:02.984863 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:15:02.984838 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-4f35c1-29744715-sdj6z" Jul 22 01:15:03.779924 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:15:03.779891 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-4f35c1-29744715-sdj6z" Jul 22 01:15:03.780135 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:15:03.779890 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-4f35c1-29744715-sdj6z" event={"ID":"91ea4b49-867f-4e27-ad7d-ecbafd505295","Type":"ContainerDied","Data":"feade6770eeb81151556fa6950cbf04ac30068834cf16c6015c3868f885a7628"} Jul 22 01:15:03.780135 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:15:03.780046 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="feade6770eeb81151556fa6950cbf04ac30068834cf16c6015c3868f885a7628" Jul 22 01:15:03.781224 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:15:03.781204 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744715-lbvxc" Jul 22 01:15:03.781332 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:15:03.781252 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744715-lbvxc" event={"ID":"ed5d6e1a-5dd8-4898-850e-cc38d385bff7","Type":"ContainerDied","Data":"86ad3850b6abf84fa67ba9a4f280837d3ab667ed8ad077f324101aa76b72b18b"} Jul 22 01:15:03.781332 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:15:03.781308 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86ad3850b6abf84fa67ba9a4f280837d3ab667ed8ad077f324101aa76b72b18b" Jul 22 01:15:03.782730 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:15:03.782713 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-e4b9ac-29744715-dj7xw" Jul 22 01:15:03.782831 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:15:03.782717 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-e4b9ac-29744715-dj7xw" event={"ID":"3023efc7-fe01-47aa-9dd5-a1a03e71668d","Type":"ContainerDied","Data":"27c50b3bc793be7d099463d50f7f72d28c9fbc51c96a3970482af39ca52781ec"} Jul 22 01:15:03.782831 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:15:03.782826 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="27c50b3bc793be7d099463d50f7f72d28c9fbc51c96a3970482af39ca52781ec" Jul 22 01:15:55.498713 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:15:55.498683 2574 scope.go:117] "RemoveContainer" containerID="5aeddda6ff27529755e211609253663e292de09600e50eb14ff0b68661dd0536" Jul 22 01:15:55.507620 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:15:55.507578 2574 scope.go:117] "RemoveContainer" containerID="66d0e9a0ecc35a1c52e64aec6b4db4c0748cf2ea0a89d1cff4d14009aa0ef7fe" Jul 22 01:16:55.271435 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:16:55.271398 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-hb69v_bf7b370a-c67c-4c3c-b3ba-7533131eace2/console-operator/1.log" Jul 22 01:16:55.275297 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:16:55.275278 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hr5fn_947d311a-ec4b-42e7-a0fb-ee5edc0caf8d/ovn-acl-logging/0.log" Jul 22 01:16:55.279666 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:16:55.279646 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-hb69v_bf7b370a-c67c-4c3c-b3ba-7533131eace2/console-operator/1.log" Jul 22 01:16:55.283333 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:16:55.283316 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hr5fn_947d311a-ec4b-42e7-a0fb-ee5edc0caf8d/ovn-acl-logging/0.log" Jul 22 01:16:55.533900 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:16:55.533806 2574 scope.go:117] "RemoveContainer" containerID="e92a3b49e0e5320393d02d70637aebd2559d6b558df7de6546df68611786f38f" Jul 22 01:17:14.001205 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:14.001172 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-e4b9ac-dj78z"] Jul 22 01:17:14.001660 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:14.001547 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91ea4b49-867f-4e27-ad7d-ecbafd505295" containerName="cleanup" Jul 22 01:17:14.001660 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:14.001558 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="91ea4b49-867f-4e27-ad7d-ecbafd505295" containerName="cleanup" Jul 22 01:17:14.001660 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:14.001573 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3023efc7-fe01-47aa-9dd5-a1a03e71668d" containerName="cleanup" Jul 22 01:17:14.001660 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:14.001578 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="3023efc7-fe01-47aa-9dd5-a1a03e71668d" containerName="cleanup" Jul 22 01:17:14.001660 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:14.001595 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed5d6e1a-5dd8-4898-850e-cc38d385bff7" containerName="cleanup" Jul 22 01:17:14.001660 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:14.001616 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed5d6e1a-5dd8-4898-850e-cc38d385bff7" containerName="cleanup" Jul 22 01:17:14.001852 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:14.001676 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="91ea4b49-867f-4e27-ad7d-ecbafd505295" containerName="cleanup" Jul 22 01:17:14.001852 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:14.001686 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed5d6e1a-5dd8-4898-850e-cc38d385bff7" containerName="cleanup" Jul 22 01:17:14.001852 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:14.001693 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="3023efc7-fe01-47aa-9dd5-a1a03e71668d" containerName="cleanup" Jul 22 01:17:14.004796 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:14.004772 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-e4b9ac-dj78z" Jul 22 01:17:14.007741 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:14.007719 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tmggk\"" Jul 22 01:17:14.017146 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:14.017123 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-e4b9ac-dj78z"] Jul 22 01:17:14.152465 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:14.152433 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fb9513b-21fb-4371-a97d-cbf36c24d09c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-e4b9ac-dj78z\" (UID: \"4fb9513b-21fb-4371-a97d-cbf36c24d09c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-e4b9ac-dj78z" Jul 22 01:17:14.254013 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:14.253932 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fb9513b-21fb-4371-a97d-cbf36c24d09c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-e4b9ac-dj78z\" (UID: \"4fb9513b-21fb-4371-a97d-cbf36c24d09c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-e4b9ac-dj78z" Jul 22 01:17:14.254547 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:14.254529 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fb9513b-21fb-4371-a97d-cbf36c24d09c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-e4b9ac-dj78z\" (UID: \"4fb9513b-21fb-4371-a97d-cbf36c24d09c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-e4b9ac-dj78z" Jul 22 01:17:14.315928 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:14.315895 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-e4b9ac-dj78z" Jul 22 01:17:14.476786 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:14.476760 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-e4b9ac-dj78z"] Jul 22 01:17:14.478657 ip-10-0-139-192 kubenswrapper[2574]: W0722 01:17:14.478581 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fb9513b_21fb_4371_a97d_cbf36c24d09c.slice/crio-409741b80216950910a87172bb1f80be386637c0927a375ab38c1521f5c37018 WatchSource:0}: Error finding container 409741b80216950910a87172bb1f80be386637c0927a375ab38c1521f5c37018: Status 404 returned error can't find the container with id 409741b80216950910a87172bb1f80be386637c0927a375ab38c1521f5c37018 Jul 22 01:17:14.518025 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:14.517999 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-e4b9ac-dj78z" event={"ID":"4fb9513b-21fb-4371-a97d-cbf36c24d09c","Type":"ContainerStarted","Data":"409741b80216950910a87172bb1f80be386637c0927a375ab38c1521f5c37018"} Jul 22 01:17:15.525640 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:15.525528 2574 generic.go:356] "Generic (PLEG): container finished" podID="4fb9513b-21fb-4371-a97d-cbf36c24d09c" containerID="453d4e0616f6bf7e5cce7ac4906936cc9be1e03d33e28c92b8371d707bcbc1dc" exitCode=0 Jul 22 01:17:15.526004 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:15.525636 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-e4b9ac-dj78z" event={"ID":"4fb9513b-21fb-4371-a97d-cbf36c24d09c","Type":"ContainerDied","Data":"453d4e0616f6bf7e5cce7ac4906936cc9be1e03d33e28c92b8371d707bcbc1dc"} Jul 22 01:17:16.665926 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:16.665898 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-e4b9ac-dj78z" Jul 22 01:17:16.776491 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:16.776456 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fb9513b-21fb-4371-a97d-cbf36c24d09c-maas-api-service-ca\") pod \"4fb9513b-21fb-4371-a97d-cbf36c24d09c\" (UID: \"4fb9513b-21fb-4371-a97d-cbf36c24d09c\") " Jul 22 01:17:16.776819 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:16.776798 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fb9513b-21fb-4371-a97d-cbf36c24d09c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4fb9513b-21fb-4371-a97d-cbf36c24d09c" (UID: "4fb9513b-21fb-4371-a97d-cbf36c24d09c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 01:17:16.877110 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:16.877080 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fb9513b-21fb-4371-a97d-cbf36c24d09c-maas-api-service-ca\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 01:17:17.537840 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:17.537813 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-e4b9ac-dj78z" Jul 22 01:17:17.538042 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:17.537809 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-e4b9ac-dj78z" event={"ID":"4fb9513b-21fb-4371-a97d-cbf36c24d09c","Type":"ContainerDied","Data":"409741b80216950910a87172bb1f80be386637c0927a375ab38c1521f5c37018"} Jul 22 01:17:17.538042 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:17.537943 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="409741b80216950910a87172bb1f80be386637c0927a375ab38c1521f5c37018" Jul 22 01:17:24.420873 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:24.420836 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-e4b9ac-cdddb465b-ns8b9"] Jul 22 01:17:24.421344 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:24.421095 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e4b9ac-cdddb465b-ns8b9" podUID="8cc9aef8-4e20-431f-b3d6-86ec75711985" containerName="maas-api" containerID="cri-o://12b535939e4da167b4836ca37326eadb8632fa4b60c42f679d04c55d3140b150" gracePeriod=30 Jul 22 01:17:24.455898 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:24.455864 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-e4b9ac-29744715-dj7xw"] Jul 22 01:17:24.459566 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:24.459537 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-e4b9ac-29744715-dj7xw"] Jul 22 01:17:24.880247 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:24.880223 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e4b9ac-cdddb465b-ns8b9" Jul 22 01:17:25.053858 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:25.053751 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cc9aef8-4e20-431f-b3d6-86ec75711985-openshift-service-ca\") pod \"8cc9aef8-4e20-431f-b3d6-86ec75711985\" (UID: \"8cc9aef8-4e20-431f-b3d6-86ec75711985\") " Jul 22 01:17:25.053858 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:25.053809 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8cc9aef8-4e20-431f-b3d6-86ec75711985-maas-api-tls\") pod \"8cc9aef8-4e20-431f-b3d6-86ec75711985\" (UID: \"8cc9aef8-4e20-431f-b3d6-86ec75711985\") " Jul 22 01:17:25.054096 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:25.053885 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q9ppk\" (UniqueName: \"kubernetes.io/projected/8cc9aef8-4e20-431f-b3d6-86ec75711985-kube-api-access-q9ppk\") pod \"8cc9aef8-4e20-431f-b3d6-86ec75711985\" (UID: \"8cc9aef8-4e20-431f-b3d6-86ec75711985\") " Jul 22 01:17:25.054223 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:25.054196 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cc9aef8-4e20-431f-b3d6-86ec75711985-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8cc9aef8-4e20-431f-b3d6-86ec75711985" (UID: "8cc9aef8-4e20-431f-b3d6-86ec75711985"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 01:17:25.055921 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:25.055899 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cc9aef8-4e20-431f-b3d6-86ec75711985-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8cc9aef8-4e20-431f-b3d6-86ec75711985" (UID: "8cc9aef8-4e20-431f-b3d6-86ec75711985"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 01:17:25.056024 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:25.056003 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cc9aef8-4e20-431f-b3d6-86ec75711985-kube-api-access-q9ppk" (OuterVolumeSpecName: "kube-api-access-q9ppk") pod "8cc9aef8-4e20-431f-b3d6-86ec75711985" (UID: "8cc9aef8-4e20-431f-b3d6-86ec75711985"). InnerVolumeSpecName "kube-api-access-q9ppk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:17:25.089185 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:25.089150 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3023efc7-fe01-47aa-9dd5-a1a03e71668d" path="/var/lib/kubelet/pods/3023efc7-fe01-47aa-9dd5-a1a03e71668d/volumes" Jul 22 01:17:25.154964 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:25.154927 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cc9aef8-4e20-431f-b3d6-86ec75711985-openshift-service-ca\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 01:17:25.154964 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:25.154957 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8cc9aef8-4e20-431f-b3d6-86ec75711985-maas-api-tls\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 01:17:25.154964 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:25.154966 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9ppk\" (UniqueName: \"kubernetes.io/projected/8cc9aef8-4e20-431f-b3d6-86ec75711985-kube-api-access-q9ppk\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 01:17:25.581563 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:25.581519 2574 generic.go:356] "Generic (PLEG): container finished" podID="8cc9aef8-4e20-431f-b3d6-86ec75711985" containerID="12b535939e4da167b4836ca37326eadb8632fa4b60c42f679d04c55d3140b150" exitCode=0 Jul 22 01:17:25.581563 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:25.581553 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e4b9ac-cdddb465b-ns8b9" event={"ID":"8cc9aef8-4e20-431f-b3d6-86ec75711985","Type":"ContainerDied","Data":"12b535939e4da167b4836ca37326eadb8632fa4b60c42f679d04c55d3140b150"} Jul 22 01:17:25.582143 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:25.581634 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e4b9ac-cdddb465b-ns8b9" event={"ID":"8cc9aef8-4e20-431f-b3d6-86ec75711985","Type":"ContainerDied","Data":"1fde6b64dd31de28bfd02f3088e76fec8aa5273a958ed432bf3fe5577024264d"} Jul 22 01:17:25.582143 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:25.581565 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e4b9ac-cdddb465b-ns8b9" Jul 22 01:17:25.582143 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:25.581662 2574 scope.go:117] "RemoveContainer" containerID="12b535939e4da167b4836ca37326eadb8632fa4b60c42f679d04c55d3140b150" Jul 22 01:17:25.590540 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:25.590518 2574 scope.go:117] "RemoveContainer" containerID="12b535939e4da167b4836ca37326eadb8632fa4b60c42f679d04c55d3140b150" Jul 22 01:17:25.590846 ip-10-0-139-192 kubenswrapper[2574]: E0722 01:17:25.590823 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12b535939e4da167b4836ca37326eadb8632fa4b60c42f679d04c55d3140b150\": container with ID starting with 12b535939e4da167b4836ca37326eadb8632fa4b60c42f679d04c55d3140b150 not found: ID does not exist" containerID="12b535939e4da167b4836ca37326eadb8632fa4b60c42f679d04c55d3140b150" Jul 22 01:17:25.590923 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:25.590853 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12b535939e4da167b4836ca37326eadb8632fa4b60c42f679d04c55d3140b150"} err="failed to get container status \"12b535939e4da167b4836ca37326eadb8632fa4b60c42f679d04c55d3140b150\": rpc error: code = NotFound desc = could not find container \"12b535939e4da167b4836ca37326eadb8632fa4b60c42f679d04c55d3140b150\": container with ID starting with 12b535939e4da167b4836ca37326eadb8632fa4b60c42f679d04c55d3140b150 not found: ID does not exist" Jul 22 01:17:25.603397 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:25.603376 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-e4b9ac-cdddb465b-ns8b9"] Jul 22 01:17:25.607981 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:25.607959 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-e4b9ac-cdddb465b-ns8b9"] Jul 22 01:17:27.088729 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:27.088695 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8cc9aef8-4e20-431f-b3d6-86ec75711985" path="/var/lib/kubelet/pods/8cc9aef8-4e20-431f-b3d6-86ec75711985/volumes" Jul 22 01:17:29.089869 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:29.089839 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-e4b9ac-dj78z"] Jul 22 01:17:29.091752 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:29.091728 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-e4b9ac-dj78z"] Jul 22 01:17:31.088547 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:31.088516 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4fb9513b-21fb-4371-a97d-cbf36c24d09c" path="/var/lib/kubelet/pods/4fb9513b-21fb-4371-a97d-cbf36c24d09c/volumes" Jul 22 01:17:36.387449 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:36.387412 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4f35c1-j8k8r"] Jul 22 01:17:36.387971 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:36.387790 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cc9aef8-4e20-431f-b3d6-86ec75711985" containerName="maas-api" Jul 22 01:17:36.387971 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:36.387801 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cc9aef8-4e20-431f-b3d6-86ec75711985" containerName="maas-api" Jul 22 01:17:36.387971 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:36.387827 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fb9513b-21fb-4371-a97d-cbf36c24d09c" containerName="revoke-keys" Jul 22 01:17:36.387971 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:36.387833 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fb9513b-21fb-4371-a97d-cbf36c24d09c" containerName="revoke-keys" Jul 22 01:17:36.387971 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:36.387886 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fb9513b-21fb-4371-a97d-cbf36c24d09c" containerName="revoke-keys" Jul 22 01:17:36.387971 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:36.387894 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8cc9aef8-4e20-431f-b3d6-86ec75711985" containerName="maas-api" Jul 22 01:17:36.390937 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:36.390916 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4f35c1-j8k8r" Jul 22 01:17:36.393512 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:36.393490 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tmggk\"" Jul 22 01:17:36.397800 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:36.397776 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4f35c1-j8k8r"] Jul 22 01:17:36.456132 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:36.456100 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/213219f4-101f-43f0-bffb-086eeeecac01-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-4f35c1-j8k8r\" (UID: \"213219f4-101f-43f0-bffb-086eeeecac01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4f35c1-j8k8r" Jul 22 01:17:36.556944 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:36.556907 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/213219f4-101f-43f0-bffb-086eeeecac01-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-4f35c1-j8k8r\" (UID: \"213219f4-101f-43f0-bffb-086eeeecac01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4f35c1-j8k8r" Jul 22 01:17:36.557569 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:36.557549 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/213219f4-101f-43f0-bffb-086eeeecac01-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-4f35c1-j8k8r\" (UID: \"213219f4-101f-43f0-bffb-086eeeecac01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4f35c1-j8k8r" Jul 22 01:17:36.703722 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:36.703629 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4f35c1-j8k8r" Jul 22 01:17:36.838747 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:36.838717 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4f35c1-j8k8r"] Jul 22 01:17:36.839087 ip-10-0-139-192 kubenswrapper[2574]: W0722 01:17:36.839061 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod213219f4_101f_43f0_bffb_086eeeecac01.slice/crio-b0da0348675c998441c3bc4619c074f53f676e50f9942941d442e78f3ff31c9e WatchSource:0}: Error finding container b0da0348675c998441c3bc4619c074f53f676e50f9942941d442e78f3ff31c9e: Status 404 returned error can't find the container with id b0da0348675c998441c3bc4619c074f53f676e50f9942941d442e78f3ff31c9e Jul 22 01:17:37.650677 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:37.650636 2574 generic.go:356] "Generic (PLEG): container finished" podID="213219f4-101f-43f0-bffb-086eeeecac01" containerID="49bcb835917675068564aaac0a02239d0bcf76381b01a575f1af57cd00fc5fbd" exitCode=0 Jul 22 01:17:37.651060 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:37.650720 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4f35c1-j8k8r" event={"ID":"213219f4-101f-43f0-bffb-086eeeecac01","Type":"ContainerDied","Data":"49bcb835917675068564aaac0a02239d0bcf76381b01a575f1af57cd00fc5fbd"} Jul 22 01:17:37.651060 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:37.650773 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4f35c1-j8k8r" event={"ID":"213219f4-101f-43f0-bffb-086eeeecac01","Type":"ContainerStarted","Data":"b0da0348675c998441c3bc4619c074f53f676e50f9942941d442e78f3ff31c9e"} Jul 22 01:17:38.788274 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:38.788251 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4f35c1-j8k8r" Jul 22 01:17:38.875810 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:38.875774 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/213219f4-101f-43f0-bffb-086eeeecac01-maas-api-service-ca\") pod \"213219f4-101f-43f0-bffb-086eeeecac01\" (UID: \"213219f4-101f-43f0-bffb-086eeeecac01\") " Jul 22 01:17:38.876125 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:38.876101 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/213219f4-101f-43f0-bffb-086eeeecac01-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "213219f4-101f-43f0-bffb-086eeeecac01" (UID: "213219f4-101f-43f0-bffb-086eeeecac01"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 01:17:38.977215 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:38.977130 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/213219f4-101f-43f0-bffb-086eeeecac01-maas-api-service-ca\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 01:17:39.663460 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:39.663430 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4f35c1-j8k8r" Jul 22 01:17:39.663460 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:39.663441 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4f35c1-j8k8r" event={"ID":"213219f4-101f-43f0-bffb-086eeeecac01","Type":"ContainerDied","Data":"b0da0348675c998441c3bc4619c074f53f676e50f9942941d442e78f3ff31c9e"} Jul 22 01:17:39.663693 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:39.663489 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b0da0348675c998441c3bc4619c074f53f676e50f9942941d442e78f3ff31c9e" Jul 22 01:17:46.828364 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:46.828287 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-4f35c1-864d798c5c-7c6kz"] Jul 22 01:17:46.828813 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:46.828549 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4f35c1-864d798c5c-7c6kz" podUID="906b0c5b-f651-4f3a-9247-fdd4e8a0000a" containerName="maas-api" containerID="cri-o://649808778f844980a49c661e6173f3cb132d746ee600c2f1ec6498ebe948ab72" gracePeriod=30 Jul 22 01:17:46.869511 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:46.869480 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-4f35c1-29744715-sdj6z"] Jul 22 01:17:46.873651 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:46.873625 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-4f35c1-29744715-sdj6z"] Jul 22 01:17:47.080379 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:47.080317 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4f35c1-864d798c5c-7c6kz" Jul 22 01:17:47.090126 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:47.090099 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91ea4b49-867f-4e27-ad7d-ecbafd505295" path="/var/lib/kubelet/pods/91ea4b49-867f-4e27-ad7d-ecbafd505295/volumes" Jul 22 01:17:47.153716 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:47.153682 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/906b0c5b-f651-4f3a-9247-fdd4e8a0000a-maas-api-tls\") pod \"906b0c5b-f651-4f3a-9247-fdd4e8a0000a\" (UID: \"906b0c5b-f651-4f3a-9247-fdd4e8a0000a\") " Jul 22 01:17:47.153716 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:47.153718 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/906b0c5b-f651-4f3a-9247-fdd4e8a0000a-openshift-service-ca\") pod \"906b0c5b-f651-4f3a-9247-fdd4e8a0000a\" (UID: \"906b0c5b-f651-4f3a-9247-fdd4e8a0000a\") " Jul 22 01:17:47.153961 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:47.153800 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8z4d6\" (UniqueName: \"kubernetes.io/projected/906b0c5b-f651-4f3a-9247-fdd4e8a0000a-kube-api-access-8z4d6\") pod \"906b0c5b-f651-4f3a-9247-fdd4e8a0000a\" (UID: \"906b0c5b-f651-4f3a-9247-fdd4e8a0000a\") " Jul 22 01:17:47.154127 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:47.154101 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/906b0c5b-f651-4f3a-9247-fdd4e8a0000a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "906b0c5b-f651-4f3a-9247-fdd4e8a0000a" (UID: "906b0c5b-f651-4f3a-9247-fdd4e8a0000a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 01:17:47.155891 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:47.155865 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/906b0c5b-f651-4f3a-9247-fdd4e8a0000a-kube-api-access-8z4d6" (OuterVolumeSpecName: "kube-api-access-8z4d6") pod "906b0c5b-f651-4f3a-9247-fdd4e8a0000a" (UID: "906b0c5b-f651-4f3a-9247-fdd4e8a0000a"). InnerVolumeSpecName "kube-api-access-8z4d6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 01:17:47.156010 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:47.155861 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/906b0c5b-f651-4f3a-9247-fdd4e8a0000a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "906b0c5b-f651-4f3a-9247-fdd4e8a0000a" (UID: "906b0c5b-f651-4f3a-9247-fdd4e8a0000a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 01:17:47.255284 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:47.255251 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8z4d6\" (UniqueName: \"kubernetes.io/projected/906b0c5b-f651-4f3a-9247-fdd4e8a0000a-kube-api-access-8z4d6\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 01:17:47.255284 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:47.255283 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/906b0c5b-f651-4f3a-9247-fdd4e8a0000a-maas-api-tls\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 01:17:47.255489 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:47.255299 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/906b0c5b-f651-4f3a-9247-fdd4e8a0000a-openshift-service-ca\") on node \"ip-10-0-139-192.ec2.internal\" DevicePath \"\"" Jul 22 01:17:47.705952 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:47.705920 2574 generic.go:356] "Generic (PLEG): container finished" podID="906b0c5b-f651-4f3a-9247-fdd4e8a0000a" containerID="649808778f844980a49c661e6173f3cb132d746ee600c2f1ec6498ebe948ab72" exitCode=0 Jul 22 01:17:47.706252 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:47.705962 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4f35c1-864d798c5c-7c6kz" Jul 22 01:17:47.706252 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:47.706002 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4f35c1-864d798c5c-7c6kz" event={"ID":"906b0c5b-f651-4f3a-9247-fdd4e8a0000a","Type":"ContainerDied","Data":"649808778f844980a49c661e6173f3cb132d746ee600c2f1ec6498ebe948ab72"} Jul 22 01:17:47.706252 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:47.706054 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4f35c1-864d798c5c-7c6kz" event={"ID":"906b0c5b-f651-4f3a-9247-fdd4e8a0000a","Type":"ContainerDied","Data":"941eabbdd14b9bf34f3c44390df2341c9762d683889fe6439273aedb62260794"} Jul 22 01:17:47.706252 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:47.706078 2574 scope.go:117] "RemoveContainer" containerID="649808778f844980a49c661e6173f3cb132d746ee600c2f1ec6498ebe948ab72" Jul 22 01:17:47.715351 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:47.715334 2574 scope.go:117] "RemoveContainer" containerID="649808778f844980a49c661e6173f3cb132d746ee600c2f1ec6498ebe948ab72" Jul 22 01:17:47.715595 ip-10-0-139-192 kubenswrapper[2574]: E0722 01:17:47.715574 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"649808778f844980a49c661e6173f3cb132d746ee600c2f1ec6498ebe948ab72\": container with ID starting with 649808778f844980a49c661e6173f3cb132d746ee600c2f1ec6498ebe948ab72 not found: ID does not exist" containerID="649808778f844980a49c661e6173f3cb132d746ee600c2f1ec6498ebe948ab72" Jul 22 01:17:47.715680 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:47.715620 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"649808778f844980a49c661e6173f3cb132d746ee600c2f1ec6498ebe948ab72"} err="failed to get container status \"649808778f844980a49c661e6173f3cb132d746ee600c2f1ec6498ebe948ab72\": rpc error: code = NotFound desc = could not find container \"649808778f844980a49c661e6173f3cb132d746ee600c2f1ec6498ebe948ab72\": container with ID starting with 649808778f844980a49c661e6173f3cb132d746ee600c2f1ec6498ebe948ab72 not found: ID does not exist" Jul 22 01:17:47.728230 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:47.728206 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-4f35c1-864d798c5c-7c6kz"] Jul 22 01:17:47.730380 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:47.730362 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-4f35c1-864d798c5c-7c6kz"] Jul 22 01:17:49.089676 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:49.089640 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="906b0c5b-f651-4f3a-9247-fdd4e8a0000a" path="/var/lib/kubelet/pods/906b0c5b-f651-4f3a-9247-fdd4e8a0000a/volumes" Jul 22 01:17:51.543212 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:51.543169 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4f35c1-j8k8r"] Jul 22 01:17:51.547901 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:51.547866 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-4f35c1-j8k8r"] Jul 22 01:17:53.089219 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:17:53.089182 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="213219f4-101f-43f0-bffb-086eeeecac01" path="/var/lib/kubelet/pods/213219f4-101f-43f0-bffb-086eeeecac01/volumes" Jul 22 01:18:04.570662 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:04.570628 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-84cdf579f4-c6prf_a5124289-201e-4af1-bc93-4bd7b9a8895f/manager/0.log" Jul 22 01:18:04.694546 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:04.694521 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-hqj4z_02566740-8d72-417f-be20-b16b2dc78276/server/0.log" Jul 22 01:18:04.817034 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:04.817000 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-zzw9z_fcc3095a-033a-43f1-8ed0-1f42ed66dcd7/manager/0.log" Jul 22 01:18:05.072701 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:05.072676 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6bb84b476f-t469v_62a8d5be-9c0f-4ace-a6f3-2be1c2090e86/copy-manifests/0.log" Jul 22 01:18:05.080186 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:05.080157 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6bb84b476f-t469v_62a8d5be-9c0f-4ace-a6f3-2be1c2090e86/manager/0.log" Jul 22 01:18:05.575963 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:05.575938 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5664d797bd-rrrgw_0d2e465e-d76a-43f4-b4e1-003d89b0bb53/maas-api/0.log" Jul 22 01:18:05.695947 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:05.695918 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29744685-59rzf_0f978879-8672-49a1-9982-93fee590cc5a/cleanup/0.log" Jul 22 01:18:05.809964 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:05.809930 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29744700-7t9f4_9c94d0ef-0181-4658-9920-73a1f0c0d3d1/cleanup/0.log" Jul 22 01:18:05.928761 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:05.928727 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29744715-lbvxc_ed5d6e1a-5dd8-4898-850e-cc38d385bff7/cleanup/0.log" Jul 22 01:18:06.043027 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:06.042994 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-tk6gw_c8fa4ab2-d239-432a-97c9-cd4eb1281cd0/postgres/0.log" Jul 22 01:18:06.853917 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:06.853890 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q4vzc_4f0c0663-abac-4777-a42e-00bb0cf90c9f/util/0.log" Jul 22 01:18:06.861522 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:06.861502 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q4vzc_4f0c0663-abac-4777-a42e-00bb0cf90c9f/pull/0.log" Jul 22 01:18:06.869474 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:06.869453 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q4vzc_4f0c0663-abac-4777-a42e-00bb0cf90c9f/extract/0.log" Jul 22 01:18:06.989359 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:06.989326 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mfn4p_6ce6a5b6-b585-45b1-99ee-ee0c9826c982/util/0.log" Jul 22 01:18:06.998921 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:06.998898 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mfn4p_6ce6a5b6-b585-45b1-99ee-ee0c9826c982/pull/0.log" Jul 22 01:18:07.008131 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:07.008110 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mfn4p_6ce6a5b6-b585-45b1-99ee-ee0c9826c982/extract/0.log" Jul 22 01:18:07.122886 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:07.122803 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738frn4_8bb8e779-075b-4123-a24a-be5ecae1a32f/util/0.log" Jul 22 01:18:07.129405 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:07.129382 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738frn4_8bb8e779-075b-4123-a24a-be5ecae1a32f/pull/0.log" Jul 22 01:18:07.136138 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:07.136115 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738frn4_8bb8e779-075b-4123-a24a-be5ecae1a32f/extract/0.log" Jul 22 01:18:07.253641 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:07.253594 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef148wqw_1e1e65cd-f7b5-439f-ab31-cad60691ce9a/pull/0.log" Jul 22 01:18:07.261710 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:07.261685 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef148wqw_1e1e65cd-f7b5-439f-ab31-cad60691ce9a/extract/0.log" Jul 22 01:18:07.268233 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:07.268212 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef148wqw_1e1e65cd-f7b5-439f-ab31-cad60691ce9a/util/0.log" Jul 22 01:18:07.864852 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:07.864825 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-kgcvm_e05a821e-c5ae-46ba-a304-2704b9c0bb4c/registry-server/0.log" Jul 22 01:18:07.990165 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:07.990131 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-ngvsl_93ed6faf-7f88-4c7a-bc55-f4f710f6e050/manager/0.log" Jul 22 01:18:08.231384 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:08.231354 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-nrvbn_0b5eeae9-2497-4dc4-b0ef-25a654135223/manager/0.log" Jul 22 01:18:08.688264 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:08.688235 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-jfssq_78bbc2cc-2bf2-44e5-a0ca-910e31f6ed20/discovery/0.log" Jul 22 01:18:08.801689 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:08.801659 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6467ff7b5c-lj8jm_017333c1-6450-4714-8b46-cb84b6a4b65b/kube-auth-proxy/0.log" Jul 22 01:18:09.780139 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:09.780110 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-ph8m4_034f4343-633a-45ff-9fee-7eccd8cf2e15/main/0.log" Jul 22 01:18:10.152772 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:10.152750 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fng9jf_d5eed5fe-1c85-4646-a373-f4fb80d65542/main/0.log" Jul 22 01:18:18.996412 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:18.996381 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-sl6b2_08b6ba5c-7b7f-4cbd-a6d9-584bc9a829fd/global-pull-secret-syncer/0.log" Jul 22 01:18:19.195108 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:19.195078 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-zk4rp_d2fac953-0d0d-48cd-be69-240d4d16dd49/konnectivity-agent/0.log" Jul 22 01:18:19.284245 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:19.284170 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-192.ec2.internal_5c5847e7e75764b70e99faa3e06c32f2/haproxy/0.log" Jul 22 01:18:24.122899 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:24.122865 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q4vzc_4f0c0663-abac-4777-a42e-00bb0cf90c9f/extract/0.log" Jul 22 01:18:24.147036 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:24.147009 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q4vzc_4f0c0663-abac-4777-a42e-00bb0cf90c9f/util/0.log" Jul 22 01:18:24.172708 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:24.172682 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q4vzc_4f0c0663-abac-4777-a42e-00bb0cf90c9f/pull/0.log" Jul 22 01:18:24.206761 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:24.206731 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mfn4p_6ce6a5b6-b585-45b1-99ee-ee0c9826c982/extract/0.log" Jul 22 01:18:24.232287 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:24.232262 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mfn4p_6ce6a5b6-b585-45b1-99ee-ee0c9826c982/util/0.log" Jul 22 01:18:24.255225 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:24.255203 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mfn4p_6ce6a5b6-b585-45b1-99ee-ee0c9826c982/pull/0.log" Jul 22 01:18:24.285951 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:24.285928 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738frn4_8bb8e779-075b-4123-a24a-be5ecae1a32f/extract/0.log" Jul 22 01:18:24.315338 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:24.315311 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738frn4_8bb8e779-075b-4123-a24a-be5ecae1a32f/util/0.log" Jul 22 01:18:24.342224 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:24.342200 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed738frn4_8bb8e779-075b-4123-a24a-be5ecae1a32f/pull/0.log" Jul 22 01:18:24.374684 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:24.374618 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef148wqw_1e1e65cd-f7b5-439f-ab31-cad60691ce9a/extract/0.log" Jul 22 01:18:24.399022 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:24.398996 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef148wqw_1e1e65cd-f7b5-439f-ab31-cad60691ce9a/util/0.log" Jul 22 01:18:24.422746 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:24.422728 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef148wqw_1e1e65cd-f7b5-439f-ab31-cad60691ce9a/pull/0.log" Jul 22 01:18:25.155550 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:25.155516 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-kgcvm_e05a821e-c5ae-46ba-a304-2704b9c0bb4c/registry-server/0.log" Jul 22 01:18:25.299821 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:25.299791 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-ngvsl_93ed6faf-7f88-4c7a-bc55-f4f710f6e050/manager/0.log" Jul 22 01:18:25.395161 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:25.395128 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-nrvbn_0b5eeae9-2497-4dc4-b0ef-25a654135223/manager/0.log" Jul 22 01:18:27.287441 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:27.287416 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-fsnlx_65e8b186-5032-46e9-b65c-5fbc08f5d68d/kube-state-metrics/0.log" Jul 22 01:18:27.317200 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:27.317173 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-fsnlx_65e8b186-5032-46e9-b65c-5fbc08f5d68d/kube-rbac-proxy-main/0.log" Jul 22 01:18:27.341301 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:27.341272 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-fsnlx_65e8b186-5032-46e9-b65c-5fbc08f5d68d/kube-rbac-proxy-self/0.log" Jul 22 01:18:27.615347 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:27.615271 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xvq7l_4d7dbfcd-8818-437c-b363-6c8abd0c52b6/node-exporter/0.log" Jul 22 01:18:27.643454 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:27.643430 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xvq7l_4d7dbfcd-8818-437c-b363-6c8abd0c52b6/kube-rbac-proxy/0.log" Jul 22 01:18:27.666977 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:27.666958 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xvq7l_4d7dbfcd-8818-437c-b363-6c8abd0c52b6/init-textfile/0.log" Jul 22 01:18:27.965311 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:27.965282 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-hvttw_2bc91e32-84b8-4e76-b19f-d0de3c4bddcc/prometheus-operator/0.log" Jul 22 01:18:27.988027 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:27.988000 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-hvttw_2bc91e32-84b8-4e76-b19f-d0de3c4bddcc/kube-rbac-proxy/0.log" Jul 22 01:18:29.409524 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:29.406802 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-sfz9r/perf-node-gather-daemonset-5zm9z"] Jul 22 01:18:29.409524 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:29.407577 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="213219f4-101f-43f0-bffb-086eeeecac01" containerName="revoke-keys" Jul 22 01:18:29.409524 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:29.407618 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="213219f4-101f-43f0-bffb-086eeeecac01" containerName="revoke-keys" Jul 22 01:18:29.409524 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:29.407695 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="906b0c5b-f651-4f3a-9247-fdd4e8a0000a" containerName="maas-api" Jul 22 01:18:29.409524 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:29.407704 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="906b0c5b-f651-4f3a-9247-fdd4e8a0000a" containerName="maas-api" Jul 22 01:18:29.409524 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:29.407855 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="213219f4-101f-43f0-bffb-086eeeecac01" containerName="revoke-keys" Jul 22 01:18:29.409524 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:29.407870 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="906b0c5b-f651-4f3a-9247-fdd4e8a0000a" containerName="maas-api" Jul 22 01:18:29.412234 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:29.412200 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sfz9r/perf-node-gather-daemonset-5zm9z" Jul 22 01:18:29.415028 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:29.415003 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-sfz9r\"/\"openshift-service-ca.crt\"" Jul 22 01:18:29.415285 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:29.415266 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-sfz9r\"/\"default-dockercfg-8wxxf\"" Jul 22 01:18:29.415390 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:29.415367 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-sfz9r\"/\"kube-root-ca.crt\"" Jul 22 01:18:29.421648 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:29.421626 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sfz9r/perf-node-gather-daemonset-5zm9z"] Jul 22 01:18:29.525351 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:29.525313 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/390cc812-4e13-4061-8ed3-142f6c8ff3e4-podres\") pod \"perf-node-gather-daemonset-5zm9z\" (UID: \"390cc812-4e13-4061-8ed3-142f6c8ff3e4\") " pod="openshift-must-gather-sfz9r/perf-node-gather-daemonset-5zm9z" Jul 22 01:18:29.525351 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:29.525349 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/390cc812-4e13-4061-8ed3-142f6c8ff3e4-sys\") pod \"perf-node-gather-daemonset-5zm9z\" (UID: \"390cc812-4e13-4061-8ed3-142f6c8ff3e4\") " pod="openshift-must-gather-sfz9r/perf-node-gather-daemonset-5zm9z" Jul 22 01:18:29.525595 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:29.525363 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/390cc812-4e13-4061-8ed3-142f6c8ff3e4-proc\") pod \"perf-node-gather-daemonset-5zm9z\" (UID: \"390cc812-4e13-4061-8ed3-142f6c8ff3e4\") " pod="openshift-must-gather-sfz9r/perf-node-gather-daemonset-5zm9z" Jul 22 01:18:29.525595 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:29.525380 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/390cc812-4e13-4061-8ed3-142f6c8ff3e4-lib-modules\") pod \"perf-node-gather-daemonset-5zm9z\" (UID: \"390cc812-4e13-4061-8ed3-142f6c8ff3e4\") " pod="openshift-must-gather-sfz9r/perf-node-gather-daemonset-5zm9z" Jul 22 01:18:29.525595 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:29.525497 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzqvf\" (UniqueName: \"kubernetes.io/projected/390cc812-4e13-4061-8ed3-142f6c8ff3e4-kube-api-access-dzqvf\") pod \"perf-node-gather-daemonset-5zm9z\" (UID: \"390cc812-4e13-4061-8ed3-142f6c8ff3e4\") " pod="openshift-must-gather-sfz9r/perf-node-gather-daemonset-5zm9z" Jul 22 01:18:29.626962 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:29.626923 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzqvf\" (UniqueName: \"kubernetes.io/projected/390cc812-4e13-4061-8ed3-142f6c8ff3e4-kube-api-access-dzqvf\") pod \"perf-node-gather-daemonset-5zm9z\" (UID: \"390cc812-4e13-4061-8ed3-142f6c8ff3e4\") " pod="openshift-must-gather-sfz9r/perf-node-gather-daemonset-5zm9z" Jul 22 01:18:29.627138 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:29.627014 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/390cc812-4e13-4061-8ed3-142f6c8ff3e4-podres\") pod \"perf-node-gather-daemonset-5zm9z\" (UID: \"390cc812-4e13-4061-8ed3-142f6c8ff3e4\") " pod="openshift-must-gather-sfz9r/perf-node-gather-daemonset-5zm9z" Jul 22 01:18:29.627138 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:29.627032 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/390cc812-4e13-4061-8ed3-142f6c8ff3e4-sys\") pod \"perf-node-gather-daemonset-5zm9z\" (UID: \"390cc812-4e13-4061-8ed3-142f6c8ff3e4\") " pod="openshift-must-gather-sfz9r/perf-node-gather-daemonset-5zm9z" Jul 22 01:18:29.627138 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:29.627049 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/390cc812-4e13-4061-8ed3-142f6c8ff3e4-proc\") pod \"perf-node-gather-daemonset-5zm9z\" (UID: \"390cc812-4e13-4061-8ed3-142f6c8ff3e4\") " pod="openshift-must-gather-sfz9r/perf-node-gather-daemonset-5zm9z" Jul 22 01:18:29.627138 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:29.627063 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/390cc812-4e13-4061-8ed3-142f6c8ff3e4-lib-modules\") pod \"perf-node-gather-daemonset-5zm9z\" (UID: \"390cc812-4e13-4061-8ed3-142f6c8ff3e4\") " pod="openshift-must-gather-sfz9r/perf-node-gather-daemonset-5zm9z" Jul 22 01:18:29.627282 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:29.627138 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/390cc812-4e13-4061-8ed3-142f6c8ff3e4-sys\") pod \"perf-node-gather-daemonset-5zm9z\" (UID: \"390cc812-4e13-4061-8ed3-142f6c8ff3e4\") " pod="openshift-must-gather-sfz9r/perf-node-gather-daemonset-5zm9z" Jul 22 01:18:29.627282 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:29.627156 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/390cc812-4e13-4061-8ed3-142f6c8ff3e4-proc\") pod \"perf-node-gather-daemonset-5zm9z\" (UID: \"390cc812-4e13-4061-8ed3-142f6c8ff3e4\") " pod="openshift-must-gather-sfz9r/perf-node-gather-daemonset-5zm9z" Jul 22 01:18:29.627282 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:29.627211 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/390cc812-4e13-4061-8ed3-142f6c8ff3e4-podres\") pod \"perf-node-gather-daemonset-5zm9z\" (UID: \"390cc812-4e13-4061-8ed3-142f6c8ff3e4\") " pod="openshift-must-gather-sfz9r/perf-node-gather-daemonset-5zm9z" Jul 22 01:18:29.627282 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:29.627250 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/390cc812-4e13-4061-8ed3-142f6c8ff3e4-lib-modules\") pod \"perf-node-gather-daemonset-5zm9z\" (UID: \"390cc812-4e13-4061-8ed3-142f6c8ff3e4\") " pod="openshift-must-gather-sfz9r/perf-node-gather-daemonset-5zm9z" Jul 22 01:18:29.636048 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:29.636021 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzqvf\" (UniqueName: \"kubernetes.io/projected/390cc812-4e13-4061-8ed3-142f6c8ff3e4-kube-api-access-dzqvf\") pod \"perf-node-gather-daemonset-5zm9z\" (UID: \"390cc812-4e13-4061-8ed3-142f6c8ff3e4\") " pod="openshift-must-gather-sfz9r/perf-node-gather-daemonset-5zm9z" Jul 22 01:18:29.724290 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:29.724206 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sfz9r/perf-node-gather-daemonset-5zm9z" Jul 22 01:18:29.871170 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:29.871144 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sfz9r/perf-node-gather-daemonset-5zm9z"] Jul 22 01:18:29.872353 ip-10-0-139-192 kubenswrapper[2574]: W0722 01:18:29.872329 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod390cc812_4e13_4061_8ed3_142f6c8ff3e4.slice/crio-61ae209a34920fb0a17f30b27e8bcb1a1ddf880fb1513c47aa454a0607ab366b WatchSource:0}: Error finding container 61ae209a34920fb0a17f30b27e8bcb1a1ddf880fb1513c47aa454a0607ab366b: Status 404 returned error can't find the container with id 61ae209a34920fb0a17f30b27e8bcb1a1ddf880fb1513c47aa454a0607ab366b Jul 22 01:18:29.949612 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:29.949572 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sfz9r/perf-node-gather-daemonset-5zm9z" event={"ID":"390cc812-4e13-4061-8ed3-142f6c8ff3e4","Type":"ContainerStarted","Data":"7508fceb34eae0129385bf0f2b973598a660310eec34c29a6e427aa519c19275"} Jul 22 01:18:29.949743 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:29.949650 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sfz9r/perf-node-gather-daemonset-5zm9z" event={"ID":"390cc812-4e13-4061-8ed3-142f6c8ff3e4","Type":"ContainerStarted","Data":"61ae209a34920fb0a17f30b27e8bcb1a1ddf880fb1513c47aa454a0607ab366b"} Jul 22 01:18:29.949785 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:29.949747 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-sfz9r/perf-node-gather-daemonset-5zm9z" Jul 22 01:18:29.999379 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:29.999275 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-sfz9r/perf-node-gather-daemonset-5zm9z" podStartSLOduration=0.999259072 podStartE2EDuration="999.259072ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 01:18:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:18:29.996961603 +0000 UTC m=+3095.353719480" watchObservedRunningTime="2026-07-22 01:18:29.999259072 +0000 UTC m=+3095.356016948" Jul 22 01:18:30.090230 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:30.090202 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-hb69v_bf7b370a-c67c-4c3c-b3ba-7533131eace2/console-operator/1.log" Jul 22 01:18:30.095772 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:30.095750 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-hb69v_bf7b370a-c67c-4c3c-b3ba-7533131eace2/console-operator/2.log" Jul 22 01:18:30.531974 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:30.531940 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-59878fc9f5-pwpl6_3b3d22fe-fe28-4de2-93f3-fab1548afd1b/console/0.log" Jul 22 01:18:32.282584 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:32.282547 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9lcsl_646c9f6d-62e7-4a07-8383-554a3c6b8b63/dns/0.log" Jul 22 01:18:32.304854 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:32.304827 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9lcsl_646c9f6d-62e7-4a07-8383-554a3c6b8b63/kube-rbac-proxy/0.log" Jul 22 01:18:32.426925 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:32.426895 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zqvbx_35037e44-306a-4ec1-9937-d3f4b3b7891a/dns-node-resolver/0.log" Jul 22 01:18:32.998075 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:32.998032 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-779f6b595d-ml5n9_22bd50aa-4014-4172-b5f1-90cdcbe3cfb0/registry/0.log" Jul 22 01:18:33.025102 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:33.025073 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7wkvg_44c9d8f9-4c85-4dfa-8572-2feb0f653720/node-ca/0.log" Jul 22 01:18:34.189911 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:34.189877 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-jfssq_78bbc2cc-2bf2-44e5-a0ca-910e31f6ed20/discovery/0.log" Jul 22 01:18:34.212659 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:34.212628 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6467ff7b5c-lj8jm_017333c1-6450-4714-8b46-cb84b6a4b65b/kube-auth-proxy/0.log" Jul 22 01:18:35.104932 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:35.104898 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hsxnj_2b90aaae-368a-4ba1-ba82-7e66aa1e518b/serve-healthcheck-canary/0.log" Jul 22 01:18:35.811378 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:35.811342 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-844qz_cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f/kube-rbac-proxy/0.log" Jul 22 01:18:35.833017 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:35.832997 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-844qz_cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f/exporter/0.log" Jul 22 01:18:35.856460 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:35.856431 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-844qz_cce653d8-6ab6-49b0-9a93-b03a8cf6ec0f/extractor/0.log" Jul 22 01:18:35.965021 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:35.964994 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-sfz9r/perf-node-gather-daemonset-5zm9z" Jul 22 01:18:38.550286 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:38.550252 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-84cdf579f4-c6prf_a5124289-201e-4af1-bc93-4bd7b9a8895f/manager/0.log" Jul 22 01:18:38.572528 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:38.572502 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-hqj4z_02566740-8d72-417f-be20-b16b2dc78276/server/0.log" Jul 22 01:18:38.647709 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:38.647677 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-zzw9z_fcc3095a-033a-43f1-8ed0-1f42ed66dcd7/manager/0.log" Jul 22 01:18:38.747346 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:38.747318 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6bb84b476f-t469v_62a8d5be-9c0f-4ace-a6f3-2be1c2090e86/manager/0.log" Jul 22 01:18:38.767484 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:38.767459 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6bb84b476f-t469v_62a8d5be-9c0f-4ace-a6f3-2be1c2090e86/copy-manifests/0.log" Jul 22 01:18:39.926487 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:39.926451 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-jfssq_78bbc2cc-2bf2-44e5-a0ca-910e31f6ed20/discovery/0.log" Jul 22 01:18:39.941777 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:39.941753 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6467ff7b5c-lj8jm_017333c1-6450-4714-8b46-cb84b6a4b65b/kube-auth-proxy/0.log" Jul 22 01:18:40.351141 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:40.351058 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-6jqcr_f61ea847-05db-4dfb-86d3-3e84726bcc91/openshift-lws-operator/0.log" Jul 22 01:18:41.415559 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:41.415527 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-jfssq_78bbc2cc-2bf2-44e5-a0ca-910e31f6ed20/discovery/0.log" Jul 22 01:18:41.438298 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:41.438273 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6467ff7b5c-lj8jm_017333c1-6450-4714-8b46-cb84b6a4b65b/kube-auth-proxy/0.log" Jul 22 01:18:47.090038 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:47.090012 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-v2ksf_077d5538-1eab-48a5-97f9-3a623a4b63c9/kube-multus-additional-cni-plugins/0.log" Jul 22 01:18:47.114751 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:47.114723 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-v2ksf_077d5538-1eab-48a5-97f9-3a623a4b63c9/egress-router-binary-copy/0.log" Jul 22 01:18:47.137755 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:47.137728 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-v2ksf_077d5538-1eab-48a5-97f9-3a623a4b63c9/cni-plugins/0.log" Jul 22 01:18:47.165918 ip-10-0-139-192 kubenswrapper[2574]: I0722 01:18:47.165884 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-v2ksf_077d5538-1eab-48a5-97f9-3a623a4b63c9/bond-cni-plugin/0.log"