Jul 23 11:35:31.620300 ip-10-0-134-236 systemd[1]: Starting Kubernetes Kubelet... Jul 23 11:35:32.040451 ip-10-0-134-236 kubenswrapper[2561]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 23 11:35:32.040451 ip-10-0-134-236 kubenswrapper[2561]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 23 11:35:32.040451 ip-10-0-134-236 kubenswrapper[2561]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 23 11:35:32.040451 ip-10-0-134-236 kubenswrapper[2561]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 23 11:35:32.040451 ip-10-0-134-236 kubenswrapper[2561]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 23 11:35:32.041412 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.041330 2561 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 23 11:35:32.047890 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.047875 2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 11:35:32.047890 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.047891 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 11:35:32.047952 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.047895 2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 11:35:32.047952 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.047898 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 11:35:32.047952 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.047901 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 11:35:32.047952 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.047904 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 11:35:32.047952 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.047907 2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 11:35:32.047952 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.047909 2561 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 11:35:32.047952 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.047912 2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 11:35:32.047952 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.047914 2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 11:35:32.047952 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.047917 2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 11:35:32.047952 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.047920 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 11:35:32.047952 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.047922 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 11:35:32.047952 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.047924 2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 11:35:32.047952 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.047927 2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 11:35:32.047952 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.047930 2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 11:35:32.047952 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.047932 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 11:35:32.047952 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.047938 2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 11:35:32.047952 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.047941 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 11:35:32.047952 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.047944 2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 11:35:32.047952 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.047946 2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 11:35:32.047952 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.047949 2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 11:35:32.048417 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.047952 2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 11:35:32.048417 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.047954 2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 11:35:32.048417 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.047957 2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 11:35:32.048417 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.047961 2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 11:35:32.048417 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.047965 2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 11:35:32.048417 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.047968 2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 11:35:32.048417 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.047971 2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 11:35:32.048417 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.047974 2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 11:35:32.048417 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.047976 2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 11:35:32.048417 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.047979 2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 11:35:32.048417 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.047983 2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 11:35:32.048417 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.047986 2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 11:35:32.048417 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.047989 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 11:35:32.048417 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.047992 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 11:35:32.048417 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.047995 2561 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 11:35:32.048417 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.047997 2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 11:35:32.048417 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048005 2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 11:35:32.048417 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048008 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 11:35:32.048417 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048010 2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 11:35:32.048860 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048012 2561 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 11:35:32.048860 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048015 2561 feature_gate.go:328] unrecognized feature gate: Example Jul 23 11:35:32.048860 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048018 2561 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 11:35:32.048860 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048020 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 11:35:32.048860 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048023 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 11:35:32.048860 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048025 2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 11:35:32.048860 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048028 2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 11:35:32.048860 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048030 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 11:35:32.048860 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048033 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 11:35:32.048860 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048036 2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 11:35:32.048860 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048038 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 11:35:32.048860 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048040 2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 11:35:32.048860 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048043 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 11:35:32.048860 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048046 2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 11:35:32.048860 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048060 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 11:35:32.048860 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048063 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 11:35:32.048860 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048066 2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 11:35:32.048860 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048068 2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 11:35:32.048860 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048071 2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 11:35:32.048860 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048073 2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 11:35:32.049344 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048076 2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 11:35:32.049344 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048078 2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 11:35:32.049344 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048081 2561 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 11:35:32.049344 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048083 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 11:35:32.049344 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048085 2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 11:35:32.049344 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048088 2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 11:35:32.049344 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048090 2561 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 11:35:32.049344 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048092 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 11:35:32.049344 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048095 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 11:35:32.049344 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048097 2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 11:35:32.049344 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048100 2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 11:35:32.049344 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048102 2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 11:35:32.049344 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048104 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 11:35:32.049344 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048108 2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 11:35:32.049344 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048111 2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 11:35:32.049344 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048114 2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 11:35:32.049344 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048117 2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 11:35:32.049344 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048120 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 11:35:32.049344 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048122 2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 11:35:32.049770 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048124 2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 11:35:32.049770 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048127 2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 11:35:32.049770 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048129 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 11:35:32.049770 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048131 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 11:35:32.049770 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048133 2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 11:35:32.049770 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048137 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 11:35:32.049770 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048139 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 11:35:32.049770 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048142 2561 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 11:35:32.049770 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048145 2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 11:35:32.049770 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048147 2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 11:35:32.049770 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048149 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 11:35:32.049770 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048151 2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 11:35:32.049770 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048154 2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 11:35:32.049770 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048156 2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 11:35:32.049770 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048177 2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 11:35:32.049770 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048181 2561 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 11:35:32.049770 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048184 2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 11:35:32.049770 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048186 2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 11:35:32.049770 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048189 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 11:35:32.049770 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048868 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 11:35:32.050264 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048875 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 11:35:32.050264 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048878 2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 11:35:32.050264 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048880 2561 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 11:35:32.050264 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048883 2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 11:35:32.050264 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048887 2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 11:35:32.050264 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048889 2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 11:35:32.050264 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048892 2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 11:35:32.050264 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048895 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 11:35:32.050264 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048899 2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 11:35:32.050264 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048902 2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 11:35:32.050264 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048905 2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 11:35:32.050264 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048908 2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 11:35:32.050264 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048911 2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 11:35:32.050264 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048914 2561 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 11:35:32.050264 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048916 2561 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 11:35:32.050264 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048920 2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 11:35:32.050264 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048922 2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 11:35:32.050264 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048925 2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 11:35:32.050264 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048928 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 11:35:32.050264 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048930 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 11:35:32.050742 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048933 2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 11:35:32.050742 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048935 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 11:35:32.050742 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048938 2561 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 11:35:32.050742 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048941 2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 11:35:32.050742 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048943 2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 11:35:32.050742 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048945 2561 feature_gate.go:328] unrecognized feature gate: Example Jul 23 11:35:32.050742 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048948 2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 11:35:32.050742 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048950 2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 11:35:32.050742 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048953 2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 11:35:32.050742 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048955 2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 11:35:32.050742 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048958 2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 11:35:32.050742 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048960 2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 11:35:32.050742 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048962 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 11:35:32.050742 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048965 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 11:35:32.050742 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048967 2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 11:35:32.050742 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048969 2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 11:35:32.050742 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048972 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 11:35:32.050742 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048974 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 11:35:32.050742 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048976 2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 11:35:32.050742 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048979 2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 11:35:32.051234 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048981 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 11:35:32.051234 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048984 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 11:35:32.051234 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048987 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 11:35:32.051234 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048989 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 11:35:32.051234 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048992 2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 11:35:32.051234 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048995 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 11:35:32.051234 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.048998 2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 11:35:32.051234 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049001 2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 11:35:32.051234 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049004 2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 11:35:32.051234 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049008 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 11:35:32.051234 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049011 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 11:35:32.051234 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049014 2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 11:35:32.051234 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049017 2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 11:35:32.051234 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049020 2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 11:35:32.051234 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049022 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 11:35:32.051234 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049025 2561 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 11:35:32.051234 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049028 2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 11:35:32.051234 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049031 2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 11:35:32.051234 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049035 2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 11:35:32.051234 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049037 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 11:35:32.051694 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049040 2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 11:35:32.051694 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049042 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 11:35:32.051694 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049045 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 11:35:32.051694 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049063 2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 11:35:32.051694 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049068 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 11:35:32.051694 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049072 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 11:35:32.051694 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049075 2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 11:35:32.051694 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049080 2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 11:35:32.051694 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049082 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 11:35:32.051694 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049085 2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 11:35:32.051694 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049088 2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 11:35:32.051694 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049090 2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 11:35:32.051694 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049092 2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 11:35:32.051694 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049095 2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 11:35:32.051694 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049097 2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 11:35:32.051694 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049100 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 11:35:32.051694 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049102 2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 11:35:32.051694 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049106 2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 11:35:32.051694 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049108 2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 11:35:32.052147 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049111 2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 11:35:32.052147 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049114 2561 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 11:35:32.052147 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049116 2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 11:35:32.052147 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049119 2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 11:35:32.052147 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049121 2561 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 11:35:32.052147 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049124 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 11:35:32.052147 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049126 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 11:35:32.052147 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049129 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 11:35:32.052147 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049131 2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 11:35:32.052147 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049133 2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 11:35:32.052147 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049136 2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 11:35:32.052147 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049138 2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 11:35:32.052147 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049141 2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 11:35:32.052147 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049143 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 11:35:32.052147 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049145 2561 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 11:35:32.052147 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049148 2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 11:35:32.052147 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049150 2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 11:35:32.052147 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049152 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 11:35:32.052147 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049154 2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 11:35:32.052147 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049501 2561 feature_gate.go:328] unrecognized feature gate: Example Jul 23 11:35:32.052614 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049505 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 11:35:32.052614 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049508 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 11:35:32.052614 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049510 2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 11:35:32.052614 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049513 2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 11:35:32.052614 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049515 2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 11:35:32.052614 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049518 2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 11:35:32.052614 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049520 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 11:35:32.052614 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049523 2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 11:35:32.052614 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049527 2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 11:35:32.052614 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049531 2561 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 11:35:32.052614 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049534 2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 11:35:32.052614 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049537 2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 11:35:32.052614 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049540 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 11:35:32.052614 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049542 2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 11:35:32.052614 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049545 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 11:35:32.052614 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049548 2561 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 11:35:32.052614 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049551 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 11:35:32.052614 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049554 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 11:35:32.052614 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049556 2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 11:35:32.053093 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049559 2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 11:35:32.053093 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049561 2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 11:35:32.053093 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049563 2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 11:35:32.053093 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049566 2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 11:35:32.053093 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049569 2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 11:35:32.053093 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049571 2561 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 11:35:32.053093 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049573 2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 11:35:32.053093 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049576 2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 11:35:32.053093 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049578 2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 11:35:32.053093 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049581 2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 11:35:32.053093 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049583 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 11:35:32.053093 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049585 2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 11:35:32.053093 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049588 2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 11:35:32.053093 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049590 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 11:35:32.053093 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049592 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 11:35:32.053093 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049595 2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 11:35:32.053093 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049597 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 11:35:32.053093 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049600 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 11:35:32.053093 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049602 2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 11:35:32.053093 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049604 2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 11:35:32.053541 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049607 2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 11:35:32.053541 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049609 2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 11:35:32.053541 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049611 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 11:35:32.053541 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049614 2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 11:35:32.053541 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049616 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 11:35:32.053541 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049619 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 11:35:32.053541 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049621 2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 11:35:32.053541 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049624 2561 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 11:35:32.053541 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049626 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 11:35:32.053541 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049629 2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 11:35:32.053541 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049631 2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 11:35:32.053541 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049634 2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 11:35:32.053541 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049637 2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 11:35:32.053541 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049640 2561 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 11:35:32.053541 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049642 2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 11:35:32.053541 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049644 2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 11:35:32.053541 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049647 2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 11:35:32.053541 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049649 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 11:35:32.053541 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049652 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 11:35:32.053541 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049654 2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 11:35:32.054030 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049656 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 11:35:32.054030 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049659 2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 11:35:32.054030 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049661 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 11:35:32.054030 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049664 2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 11:35:32.054030 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049667 2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 11:35:32.054030 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049669 2561 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 11:35:32.054030 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049671 2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 11:35:32.054030 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049674 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 11:35:32.054030 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049676 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 11:35:32.054030 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049678 2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 11:35:32.054030 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049680 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 11:35:32.054030 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049684 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 11:35:32.054030 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049686 2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 11:35:32.054030 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049688 2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 11:35:32.054030 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049690 2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 11:35:32.054030 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049693 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 11:35:32.054030 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049695 2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 11:35:32.054030 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049698 2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 11:35:32.054030 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049700 2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 11:35:32.054030 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049703 2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 11:35:32.054545 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049706 2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 11:35:32.054545 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049709 2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 11:35:32.054545 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049712 2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 11:35:32.054545 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049714 2561 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 11:35:32.054545 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049717 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 11:35:32.054545 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049719 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 11:35:32.054545 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049721 2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 11:35:32.054545 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049724 2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 11:35:32.054545 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049727 2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 11:35:32.054545 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049731 2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 11:35:32.054545 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049733 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 11:35:32.054545 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049736 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 11:35:32.054545 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049738 2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 11:35:32.054545 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049740 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 11:35:32.054545 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049743 2561 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 11:35:32.054545 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049745 2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 11:35:32.054545 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049748 2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 11:35:32.054545 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049750 2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 11:35:32.054545 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.049753 2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 11:35:32.054545 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.049817 2561 flags.go:64] FLAG: --address="0.0.0.0" Jul 23 11:35:32.055003 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.049824 2561 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 23 11:35:32.055003 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.049832 2561 flags.go:64] FLAG: --anonymous-auth="true" Jul 23 11:35:32.055003 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.049838 2561 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 23 11:35:32.055003 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.049844 2561 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 23 11:35:32.055003 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.049849 2561 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 23 11:35:32.055003 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.049855 2561 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 23 11:35:32.055003 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.049861 2561 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 23 11:35:32.055003 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.049865 2561 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 23 11:35:32.055003 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.049867 2561 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 23 11:35:32.055003 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.049874 2561 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 23 11:35:32.055003 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.049878 2561 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 23 11:35:32.055003 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.049881 2561 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 23 11:35:32.055003 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.049884 2561 flags.go:64] FLAG: --cgroup-root="" Jul 23 11:35:32.055003 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.049886 2561 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 23 11:35:32.055003 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.049889 2561 flags.go:64] FLAG: --client-ca-file="" Jul 23 11:35:32.055003 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.049892 2561 flags.go:64] FLAG: --cloud-provider="external" Jul 23 11:35:32.055003 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.049894 2561 flags.go:64] FLAG: --cluster-dns="[]" Jul 23 11:35:32.055003 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.049899 2561 flags.go:64] FLAG: --cluster-domain="" Jul 23 11:35:32.055003 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.049901 2561 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 23 11:35:32.055003 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.049904 2561 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 23 11:35:32.055003 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.049907 2561 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 23 11:35:32.055003 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.049910 2561 flags.go:64] FLAG: --container-log-max-files="5" Jul 23 11:35:32.055003 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.049913 2561 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 23 11:35:32.055003 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.049916 2561 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 23 11:35:32.055613 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.049919 2561 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 23 11:35:32.055613 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.049922 2561 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 23 11:35:32.055613 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.049928 2561 flags.go:64] FLAG: --contention-profiling="false" Jul 23 11:35:32.055613 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.049930 2561 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 23 11:35:32.055613 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.049933 2561 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 23 11:35:32.055613 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.049936 2561 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 23 11:35:32.055613 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.049938 2561 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 23 11:35:32.055613 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.049942 2561 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 23 11:35:32.055613 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.049945 2561 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 23 11:35:32.055613 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.049948 2561 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 23 11:35:32.055613 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.049951 2561 flags.go:64] FLAG: --enable-load-reader="false" Jul 23 11:35:32.055613 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.049953 2561 flags.go:64] FLAG: --enable-server="true" Jul 23 11:35:32.055613 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.049956 2561 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 23 11:35:32.055613 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.049959 2561 flags.go:64] FLAG: --event-burst="100" Jul 23 11:35:32.055613 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.049963 2561 flags.go:64] FLAG: --event-qps="50" Jul 23 11:35:32.055613 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.049965 2561 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 23 11:35:32.055613 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.049968 2561 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 23 11:35:32.055613 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.049972 2561 flags.go:64] FLAG: --eviction-hard="" Jul 23 11:35:32.055613 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.049975 2561 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 23 11:35:32.055613 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.049978 2561 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 23 11:35:32.055613 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.049981 2561 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 23 11:35:32.055613 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.049984 2561 flags.go:64] FLAG: --eviction-soft="" Jul 23 11:35:32.055613 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.049987 2561 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 23 11:35:32.055613 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.049989 2561 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 23 11:35:32.055613 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.049992 2561 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 23 11:35:32.056233 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.049994 2561 flags.go:64] FLAG: --experimental-mounter-path="" Jul 23 11:35:32.056233 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.049997 2561 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 23 11:35:32.056233 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.049999 2561 flags.go:64] FLAG: --fail-swap-on="true" Jul 23 11:35:32.056233 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050002 2561 flags.go:64] FLAG: --feature-gates="" Jul 23 11:35:32.056233 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050005 2561 flags.go:64] FLAG: --file-check-frequency="20s" Jul 23 11:35:32.056233 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050008 2561 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 23 11:35:32.056233 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050010 2561 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 23 11:35:32.056233 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050013 2561 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 23 11:35:32.056233 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050016 2561 flags.go:64] FLAG: --healthz-port="10248" Jul 23 11:35:32.056233 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050020 2561 flags.go:64] FLAG: --help="false" Jul 23 11:35:32.056233 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050023 2561 flags.go:64] FLAG: --hostname-override="ip-10-0-134-236.ec2.internal" Jul 23 11:35:32.056233 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050026 2561 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 23 11:35:32.056233 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050028 2561 flags.go:64] FLAG: --http-check-frequency="20s" Jul 23 11:35:32.056233 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050031 2561 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 23 11:35:32.056233 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050034 2561 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 23 11:35:32.056233 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050037 2561 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 23 11:35:32.056233 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050040 2561 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 23 11:35:32.056233 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050043 2561 flags.go:64] FLAG: --image-service-endpoint="" Jul 23 11:35:32.056233 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050045 2561 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 23 11:35:32.056233 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050063 2561 flags.go:64] FLAG: --kube-api-burst="100" Jul 23 11:35:32.056233 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050066 2561 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 23 11:35:32.056233 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050069 2561 flags.go:64] FLAG: --kube-api-qps="50" Jul 23 11:35:32.056233 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050071 2561 flags.go:64] FLAG: --kube-reserved="" Jul 23 11:35:32.056233 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050074 2561 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 23 11:35:32.056816 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050078 2561 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 23 11:35:32.056816 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050081 2561 flags.go:64] FLAG: --kubelet-cgroups="" Jul 23 11:35:32.056816 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050084 2561 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 23 11:35:32.056816 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050088 2561 flags.go:64] FLAG: --lock-file="" Jul 23 11:35:32.056816 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050090 2561 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 23 11:35:32.056816 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050093 2561 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 23 11:35:32.056816 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050096 2561 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 23 11:35:32.056816 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050101 2561 flags.go:64] FLAG: --log-json-split-stream="false" Jul 23 11:35:32.056816 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050103 2561 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 23 11:35:32.056816 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050106 2561 flags.go:64] FLAG: --log-text-split-stream="false" Jul 23 11:35:32.056816 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050109 2561 flags.go:64] FLAG: --logging-format="text" Jul 23 11:35:32.056816 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050111 2561 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 23 11:35:32.056816 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050114 2561 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 23 11:35:32.056816 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050117 2561 flags.go:64] FLAG: --manifest-url="" Jul 23 11:35:32.056816 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050119 2561 flags.go:64] FLAG: --manifest-url-header="" Jul 23 11:35:32.056816 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050123 2561 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 23 11:35:32.056816 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050126 2561 flags.go:64] FLAG: --max-open-files="1000000" Jul 23 11:35:32.056816 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050131 2561 flags.go:64] FLAG: --max-pods="110" Jul 23 11:35:32.056816 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050133 2561 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 23 11:35:32.056816 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050136 2561 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 23 11:35:32.056816 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050139 2561 flags.go:64] FLAG: --memory-manager-policy="None" Jul 23 11:35:32.056816 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050141 2561 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 23 11:35:32.056816 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050144 2561 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 23 11:35:32.056816 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050147 2561 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 23 11:35:32.056816 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050149 2561 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 23 11:35:32.057416 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050161 2561 flags.go:64] FLAG: --node-status-max-images="50" Jul 23 11:35:32.057416 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050164 2561 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 23 11:35:32.057416 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050166 2561 flags.go:64] FLAG: --oom-score-adj="-999" Jul 23 11:35:32.057416 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050169 2561 flags.go:64] FLAG: --pod-cidr="" Jul 23 11:35:32.057416 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050172 2561 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 23 11:35:32.057416 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050177 2561 flags.go:64] FLAG: --pod-manifest-path="" Jul 23 11:35:32.057416 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050180 2561 flags.go:64] FLAG: --pod-max-pids="-1" Jul 23 11:35:32.057416 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050183 2561 flags.go:64] FLAG: --pods-per-core="0" Jul 23 11:35:32.057416 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050187 2561 flags.go:64] FLAG: --port="10250" Jul 23 11:35:32.057416 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050190 2561 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 23 11:35:32.057416 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050192 2561 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-02a77420667d0c816" Jul 23 11:35:32.057416 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050196 2561 flags.go:64] FLAG: --qos-reserved="" Jul 23 11:35:32.057416 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050198 2561 flags.go:64] FLAG: --read-only-port="10255" Jul 23 11:35:32.057416 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050201 2561 flags.go:64] FLAG: --register-node="true" Jul 23 11:35:32.057416 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050204 2561 flags.go:64] FLAG: --register-with-taints="" Jul 23 11:35:32.057416 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050207 2561 flags.go:64] FLAG: --registry-burst="10" Jul 23 11:35:32.057416 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050209 2561 flags.go:64] FLAG: --registry-qps="5" Jul 23 11:35:32.057416 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050212 2561 flags.go:64] FLAG: --reserved-cpus="" Jul 23 11:35:32.057416 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050214 2561 flags.go:64] FLAG: --reserved-memory="" Jul 23 11:35:32.057416 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050219 2561 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 23 11:35:32.057416 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050222 2561 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 23 11:35:32.057416 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050224 2561 flags.go:64] FLAG: --rotate-certificates="false" Jul 23 11:35:32.057416 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050237 2561 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 23 11:35:32.057416 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050240 2561 flags.go:64] FLAG: --runonce="false" Jul 23 11:35:32.057416 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050244 2561 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 23 11:35:32.058008 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050247 2561 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 23 11:35:32.058008 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050250 2561 flags.go:64] FLAG: --seccomp-default="false" Jul 23 11:35:32.058008 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050253 2561 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 23 11:35:32.058008 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050255 2561 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 23 11:35:32.058008 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050258 2561 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 23 11:35:32.058008 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050261 2561 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 23 11:35:32.058008 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050263 2561 flags.go:64] FLAG: --storage-driver-password="root" Jul 23 11:35:32.058008 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050266 2561 flags.go:64] FLAG: --storage-driver-secure="false" Jul 23 11:35:32.058008 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050269 2561 flags.go:64] FLAG: --storage-driver-table="stats" Jul 23 11:35:32.058008 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050272 2561 flags.go:64] FLAG: --storage-driver-user="root" Jul 23 11:35:32.058008 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050274 2561 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 23 11:35:32.058008 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050277 2561 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 23 11:35:32.058008 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050280 2561 flags.go:64] FLAG: --system-cgroups="" Jul 23 11:35:32.058008 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050283 2561 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 23 11:35:32.058008 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050288 2561 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 23 11:35:32.058008 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050291 2561 flags.go:64] FLAG: --tls-cert-file="" Jul 23 11:35:32.058008 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050294 2561 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 23 11:35:32.058008 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050298 2561 flags.go:64] FLAG: --tls-min-version="" Jul 23 11:35:32.058008 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050300 2561 flags.go:64] FLAG: --tls-private-key-file="" Jul 23 11:35:32.058008 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050303 2561 flags.go:64] FLAG: --topology-manager-policy="none" Jul 23 11:35:32.058008 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050306 2561 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 23 11:35:32.058008 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050309 2561 flags.go:64] FLAG: --topology-manager-scope="container" Jul 23 11:35:32.058008 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050311 2561 flags.go:64] FLAG: --v="2" Jul 23 11:35:32.058008 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050315 2561 flags.go:64] FLAG: --version="false" Jul 23 11:35:32.058008 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050318 2561 flags.go:64] FLAG: --vmodule="" Jul 23 11:35:32.058949 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050322 2561 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 23 11:35:32.058949 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050325 2561 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 23 11:35:32.058949 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050434 2561 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 11:35:32.058949 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050437 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 11:35:32.058949 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050440 2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 11:35:32.058949 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050443 2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 11:35:32.058949 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050446 2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 11:35:32.058949 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050449 2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 11:35:32.058949 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050452 2561 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 11:35:32.058949 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050454 2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 11:35:32.058949 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050457 2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 11:35:32.058949 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050459 2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 11:35:32.058949 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050462 2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 11:35:32.058949 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050464 2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 11:35:32.058949 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050466 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 11:35:32.058949 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050469 2561 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 11:35:32.058949 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050471 2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 11:35:32.058949 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050474 2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 11:35:32.058949 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050476 2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 11:35:32.058949 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050479 2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 11:35:32.059629 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050481 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 11:35:32.059629 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050483 2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 11:35:32.059629 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050487 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 11:35:32.059629 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050489 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 11:35:32.059629 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050492 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 11:35:32.059629 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050495 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 11:35:32.059629 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050497 2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 11:35:32.059629 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050500 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 11:35:32.059629 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050502 2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 11:35:32.059629 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050505 2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 11:35:32.059629 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050507 2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 11:35:32.059629 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050510 2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 11:35:32.059629 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050512 2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 11:35:32.059629 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050514 2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 11:35:32.059629 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050517 2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 11:35:32.059629 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050519 2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 11:35:32.059629 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050521 2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 11:35:32.059629 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050523 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 11:35:32.059629 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050527 2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 11:35:32.059629 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050529 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 11:35:32.060124 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050531 2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 11:35:32.060124 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050534 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 11:35:32.060124 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050536 2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 11:35:32.060124 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050539 2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 11:35:32.060124 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050541 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 11:35:32.060124 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050544 2561 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 11:35:32.060124 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050546 2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 11:35:32.060124 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050548 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 11:35:32.060124 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050551 2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 11:35:32.060124 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050555 2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 11:35:32.060124 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050558 2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 11:35:32.060124 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050560 2561 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 11:35:32.060124 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050563 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 11:35:32.060124 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050565 2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 11:35:32.060124 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050569 2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 11:35:32.060124 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050571 2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 11:35:32.060124 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050574 2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 11:35:32.060124 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050576 2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 11:35:32.060124 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050578 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 11:35:32.060566 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050581 2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 11:35:32.060566 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050584 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 11:35:32.060566 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050587 2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 11:35:32.060566 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050589 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 11:35:32.060566 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050591 2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 11:35:32.060566 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050594 2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 11:35:32.060566 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050596 2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 11:35:32.060566 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050599 2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 11:35:32.060566 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050601 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 11:35:32.060566 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050603 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 11:35:32.060566 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050605 2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 11:35:32.060566 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050609 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 11:35:32.060566 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050612 2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 11:35:32.060566 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050615 2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 11:35:32.060566 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050618 2561 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 11:35:32.060566 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050620 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 11:35:32.060566 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050622 2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 11:35:32.060566 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050624 2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 11:35:32.060566 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050627 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 11:35:32.061022 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050629 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 11:35:32.061022 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050631 2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 11:35:32.061022 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050635 2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 11:35:32.061022 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050638 2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 11:35:32.061022 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050641 2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 11:35:32.061022 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050644 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 11:35:32.061022 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050646 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 11:35:32.061022 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050649 2561 feature_gate.go:328] unrecognized feature gate: Example Jul 23 11:35:32.061022 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050653 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 11:35:32.061022 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050656 2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 11:35:32.061022 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050659 2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 11:35:32.061022 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050661 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 11:35:32.061022 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050664 2561 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 11:35:32.061022 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050666 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 11:35:32.061022 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050668 2561 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 11:35:32.061022 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050671 2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 11:35:32.061022 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050674 2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 11:35:32.061022 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050676 2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 11:35:32.061022 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050678 2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 11:35:32.061022 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050680 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 11:35:32.061500 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050683 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 11:35:32.061500 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050685 2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 11:35:32.061500 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.050687 2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 11:35:32.061500 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.050693 2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 23 11:35:32.061500 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.057270 2561 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 23 11:35:32.061500 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.057286 2561 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 23 11:35:32.061500 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.057304 2561 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 23 11:35:32.061500 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.057310 2561 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 23 11:35:32.061500 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057363 2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 11:35:32.061500 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057367 2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 11:35:32.061500 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057371 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 11:35:32.061500 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057374 2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 11:35:32.061500 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057376 2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 11:35:32.061500 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057379 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 11:35:32.061832 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057383 2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 11:35:32.061832 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057387 2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 11:35:32.061832 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057390 2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 11:35:32.061832 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057393 2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 11:35:32.061832 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057396 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 11:35:32.061832 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057399 2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 11:35:32.061832 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057401 2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 11:35:32.061832 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057403 2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 11:35:32.061832 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057406 2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 11:35:32.061832 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057408 2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 11:35:32.061832 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057411 2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 11:35:32.061832 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057413 2561 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 11:35:32.061832 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057415 2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 11:35:32.061832 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057418 2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 11:35:32.061832 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057421 2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 11:35:32.061832 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057423 2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 11:35:32.061832 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057426 2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 11:35:32.061832 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057428 2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 11:35:32.061832 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057431 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 11:35:32.062289 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057433 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 11:35:32.062289 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057436 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 11:35:32.062289 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057438 2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 11:35:32.062289 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057441 2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 11:35:32.062289 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057444 2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 11:35:32.062289 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057447 2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 11:35:32.062289 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057450 2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 11:35:32.062289 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057452 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 11:35:32.062289 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057455 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 11:35:32.062289 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057458 2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 11:35:32.062289 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057460 2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 11:35:32.062289 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057462 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 11:35:32.062289 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057465 2561 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 11:35:32.062289 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057467 2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 11:35:32.062289 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057469 2561 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 11:35:32.062289 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057472 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 11:35:32.062289 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057474 2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 11:35:32.062289 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057476 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 11:35:32.062289 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057479 2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 11:35:32.062289 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057481 2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 11:35:32.062746 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057483 2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 11:35:32.062746 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057486 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 11:35:32.062746 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057488 2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 11:35:32.062746 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057490 2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 11:35:32.062746 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057492 2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 11:35:32.062746 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057496 2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 11:35:32.062746 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057499 2561 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 11:35:32.062746 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057502 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 11:35:32.062746 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057504 2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 11:35:32.062746 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057507 2561 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 11:35:32.062746 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057509 2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 11:35:32.062746 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057512 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 11:35:32.062746 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057514 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 11:35:32.062746 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057516 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 11:35:32.062746 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057519 2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 11:35:32.062746 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057521 2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 11:35:32.062746 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057524 2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 11:35:32.062746 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057527 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 11:35:32.062746 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057529 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 11:35:32.062746 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057531 2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 11:35:32.063224 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057534 2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 11:35:32.063224 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057536 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 11:35:32.063224 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057539 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 11:35:32.063224 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057541 2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 11:35:32.063224 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057544 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 11:35:32.063224 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057546 2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 11:35:32.063224 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057548 2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 11:35:32.063224 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057552 2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 11:35:32.063224 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057555 2561 feature_gate.go:328] unrecognized feature gate: Example Jul 23 11:35:32.063224 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057558 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 11:35:32.063224 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057560 2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 11:35:32.063224 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057563 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 11:35:32.063224 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057565 2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 11:35:32.063224 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057568 2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 11:35:32.063224 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057570 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 11:35:32.063224 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057572 2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 11:35:32.063224 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057575 2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 11:35:32.063224 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057577 2561 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 11:35:32.063224 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057580 2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 11:35:32.063672 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057582 2561 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 11:35:32.063672 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057585 2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 11:35:32.063672 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057587 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 11:35:32.063672 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057590 2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 11:35:32.063672 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057592 2561 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 11:35:32.063672 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057595 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 11:35:32.063672 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057597 2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 11:35:32.063672 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057599 2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 11:35:32.063672 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057602 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 11:35:32.063672 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057605 2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 11:35:32.063672 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057607 2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 11:35:32.063672 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057610 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 11:35:32.063672 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057612 2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 11:35:32.063672 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057615 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 11:35:32.063672 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057617 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 11:35:32.064016 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.057622 2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 23 11:35:32.064016 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057720 2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 11:35:32.064016 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057724 2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 11:35:32.064016 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057727 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 11:35:32.064016 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057730 2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 11:35:32.064016 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057732 2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 11:35:32.064016 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057735 2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 11:35:32.064016 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057737 2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 11:35:32.064016 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057740 2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 11:35:32.064016 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057742 2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 11:35:32.064016 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057745 2561 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 11:35:32.064016 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057747 2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 11:35:32.064016 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057749 2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 11:35:32.064016 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057751 2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 11:35:32.064016 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057754 2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 11:35:32.064465 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057756 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 11:35:32.064465 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057758 2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 11:35:32.064465 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057760 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 11:35:32.064465 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057763 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 11:35:32.064465 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057766 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 11:35:32.064465 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057768 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 11:35:32.064465 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057771 2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 11:35:32.064465 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057773 2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 11:35:32.064465 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057775 2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 11:35:32.064465 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057778 2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 11:35:32.064465 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057780 2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 11:35:32.064465 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057783 2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 11:35:32.064465 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057785 2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 11:35:32.064465 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057787 2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 11:35:32.064465 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057790 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 11:35:32.064465 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057792 2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 11:35:32.064465 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057794 2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 11:35:32.064465 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057797 2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 11:35:32.064465 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057799 2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 11:35:32.064465 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057802 2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 11:35:32.064937 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057804 2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 11:35:32.064937 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057806 2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 11:35:32.064937 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057809 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 11:35:32.064937 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057811 2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 11:35:32.064937 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057813 2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 11:35:32.064937 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057816 2561 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 11:35:32.064937 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057818 2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 11:35:32.064937 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057820 2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 11:35:32.064937 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057823 2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 11:35:32.064937 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057825 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 11:35:32.064937 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057827 2561 feature_gate.go:328] unrecognized feature gate: Example Jul 23 11:35:32.064937 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057831 2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 11:35:32.064937 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057835 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 11:35:32.064937 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057837 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 11:35:32.064937 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057840 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 11:35:32.064937 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057842 2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 11:35:32.064937 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057845 2561 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 11:35:32.064937 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057847 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 11:35:32.064937 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057850 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 11:35:32.065390 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057852 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 11:35:32.065390 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057855 2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 11:35:32.065390 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057857 2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 11:35:32.065390 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057860 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 11:35:32.065390 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057862 2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 11:35:32.065390 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057864 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 11:35:32.065390 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057867 2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 11:35:32.065390 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057869 2561 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 11:35:32.065390 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057871 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 11:35:32.065390 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057874 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 11:35:32.065390 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057876 2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 11:35:32.065390 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057878 2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 11:35:32.065390 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057881 2561 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 11:35:32.065390 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057883 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 11:35:32.065390 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057885 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 11:35:32.065390 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057888 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 11:35:32.065390 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057890 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 11:35:32.065390 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057893 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 11:35:32.065390 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057895 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 11:35:32.065390 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057897 2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 11:35:32.065834 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057900 2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 11:35:32.065834 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057903 2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 11:35:32.065834 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057907 2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 11:35:32.065834 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057909 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 11:35:32.065834 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057912 2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 11:35:32.065834 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057915 2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 11:35:32.065834 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057917 2561 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 11:35:32.065834 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057920 2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 11:35:32.065834 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057923 2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 11:35:32.065834 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057925 2561 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 11:35:32.065834 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057927 2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 11:35:32.065834 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057930 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 11:35:32.065834 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057932 2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 11:35:32.065834 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057935 2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 11:35:32.065834 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057937 2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 11:35:32.065834 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057939 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 11:35:32.065834 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057942 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 11:35:32.065834 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057945 2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 11:35:32.065834 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057948 2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 11:35:32.066300 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057950 2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 11:35:32.066300 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057952 2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 11:35:32.066300 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057955 2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 11:35:32.066300 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057957 2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 11:35:32.066300 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057959 2561 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 11:35:32.066300 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057962 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 11:35:32.066300 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:32.057964 2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 11:35:32.066300 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.057968 2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 23 11:35:32.066300 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.058047 2561 server.go:962] "Client rotation is on, will bootstrap in background" Jul 23 11:35:32.066300 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.060805 2561 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 23 11:35:32.066300 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.062273 2561 server.go:1019] "Starting client certificate rotation" Jul 23 11:35:32.066300 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.062365 2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 23 11:35:32.066300 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.062392 2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 23 11:35:32.066613 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.064793 2561 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 23 11:35:32.066613 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.065890 2561 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 23 11:35:32.075298 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.075282 2561 log.go:25] "Validated CRI v1 runtime API" Jul 23 11:35:32.078265 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.078251 2561 log.go:25] "Validated CRI v1 image API" Jul 23 11:35:32.078960 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.078947 2561 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 23 11:35:32.081401 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.081385 2561 fs.go:135] Filesystem UUIDs: map[3acb8ab3-eda9-47d7-b1ea-e9a63ec6fe67:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 d8e3570b-f037-46d1-9efb-61bdf039443e:/dev/nvme0n1p3] Jul 23 11:35:32.081464 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.081401 2561 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 23 11:35:32.088478 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.088365 2561 manager.go:216] Machine: {Timestamp:2026-07-23 11:35:32.08681968 +0000 UTC m=+0.345366950 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3195218 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d7e6bfe8b192ff46e1777d0c45cd8 SystemUUID:ec2d7e6b-fe8b-192f-f46e-1777d0c45cd8 BootID:4588426a-1480-40b6-9f86-7f1ab4705c24 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:60:ff:1d:14:85 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:60:ff:1d:14:85 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:72:51:9a:e5:28:b3 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 23 11:35:32.088478 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.088469 2561 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 23 11:35:32.088585 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.088541 2561 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 23 11:35:32.092235 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.092214 2561 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 23 11:35:32.092362 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.092239 2561 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-236.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 23 11:35:32.093085 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.093075 2561 topology_manager.go:138] "Creating topology manager with none policy" Jul 23 11:35:32.093116 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.093087 2561 container_manager_linux.go:309] "Creating device plugin manager" Jul 23 11:35:32.093116 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.093095 2561 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 23 11:35:32.093683 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.093669 2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 23 11:35:32.093736 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.093697 2561 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 23 11:35:32.095267 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.095256 2561 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 23 11:35:32.098191 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.098180 2561 state_mem.go:36] "Initialized new in-memory state store" Jul 23 11:35:32.098308 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.098299 2561 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 23 11:35:32.098343 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.098328 2561 kubelet.go:486] "Attempting to sync node with API server" Jul 23 11:35:32.098369 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.098343 2561 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 23 11:35:32.098369 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.098357 2561 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 23 11:35:32.098369 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.098364 2561 kubelet.go:398] "Adding apiserver pod source" Jul 23 11:35:32.098470 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.098372 2561 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 23 11:35:32.098849 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.098833 2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 23 11:35:32.098887 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.098867 2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 23 11:35:32.102336 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.102316 2561 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 23 11:35:32.104399 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.104379 2561 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 23 11:35:32.104474 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.104427 2561 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 23 11:35:32.105690 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.105678 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 23 11:35:32.105722 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.105696 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 23 11:35:32.105722 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.105701 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 23 11:35:32.105722 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.105706 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 23 11:35:32.105722 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.105715 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 23 11:35:32.105819 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.105723 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 23 11:35:32.105819 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.105733 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 23 11:35:32.105819 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.105738 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 23 11:35:32.105819 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.105744 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 23 11:35:32.105819 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.105749 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 23 11:35:32.105819 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.105756 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 23 11:35:32.105819 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.105761 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 23 11:35:32.106612 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.106603 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 23 11:35:32.106612 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.106612 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 23 11:35:32.109854 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.109841 2561 server.go:1268] "Started kubelet" Jul 23 11:35:32.109977 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.109948 2561 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 23 11:35:32.110154 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.109932 2561 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 23 11:35:32.110302 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.110176 2561 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 23 11:35:32.110766 ip-10-0-134-236 systemd[1]: Started Kubernetes Kubelet. Jul 23 11:35:32.111167 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:32.111138 2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-236.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 23 11:35:32.111292 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.111224 2561 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-236.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 23 11:35:32.111469 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:32.111363 2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 23 11:35:32.111904 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.111887 2561 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 23 11:35:32.112426 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.112414 2561 server.go:310] "Adding debug handlers to kubelet server" Jul 23 11:35:32.115295 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:32.115259 2561 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 23 11:35:32.116406 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:32.115455 2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-236.ec2.internal.18c4e815ab33ca66 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-236.ec2.internal,UID:ip-10-0-134-236.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-236.ec2.internal,},FirstTimestamp:2026-07-23 11:35:32.109814374 +0000 UTC m=+0.368361643,LastTimestamp:2026-07-23 11:35:32.109814374 +0000 UTC m=+0.368361643,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-236.ec2.internal,}" Jul 23 11:35:32.117782 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.117763 2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 23 11:35:32.117853 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.117770 2561 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 23 11:35:32.118213 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:32.117991 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-236.ec2.internal\" not found" Jul 23 11:35:32.118429 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.118411 2561 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 23 11:35:32.118508 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.118464 2561 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 23 11:35:32.118508 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.118478 2561 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 23 11:35:32.118619 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.118562 2561 factory.go:55] Registering systemd factory Jul 23 11:35:32.118619 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.118587 2561 factory.go:223] Registration of the systemd container factory successfully Jul 23 11:35:32.118701 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.118656 2561 reconstruct.go:97] "Volume reconstruction finished" Jul 23 11:35:32.118701 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.118663 2561 reconciler.go:29] "Reconciler: start to sync state" Jul 23 11:35:32.118850 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.118836 2561 factory.go:153] Registering CRI-O factory Jul 23 11:35:32.118925 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.118852 2561 factory.go:223] Registration of the crio container factory successfully Jul 23 11:35:32.118925 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.118916 2561 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 23 11:35:32.119019 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.118938 2561 factory.go:103] Registering Raw factory Jul 23 11:35:32.119019 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.118953 2561 manager.go:1196] Started watching for new ooms in manager Jul 23 11:35:32.119482 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.119469 2561 manager.go:349] Starting recovery of all containers Jul 23 11:35:32.125510 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:32.125485 2561 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 23 11:35:32.125510 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:32.125486 2561 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-236.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 23 11:35:32.127842 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.127703 2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 23 11:35:32.129293 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.129274 2561 manager.go:354] Recovery completed Jul 23 11:35:32.134226 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.134212 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 11:35:32.135805 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.135785 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-236.ec2.internal" event="NodeHasSufficientMemory" Jul 23 11:35:32.135879 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.135815 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-236.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 11:35:32.135879 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.135831 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-236.ec2.internal" event="NodeHasSufficientPID" Jul 23 11:35:32.136314 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.136299 2561 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 23 11:35:32.136314 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.136311 2561 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 23 11:35:32.136456 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.136324 2561 state_mem.go:36] "Initialized new in-memory state store" Jul 23 11:35:32.136456 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.136303 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 11:35:32.137492 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.137480 2561 policy_none.go:49] "None policy: Start" Jul 23 11:35:32.137532 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.137495 2561 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 23 11:35:32.137532 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.137504 2561 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 23 11:35:32.137964 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:32.137909 2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-236.ec2.internal.18c4e815acc05bb5 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-236.ec2.internal,UID:ip-10-0-134-236.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-236.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-236.ec2.internal,},FirstTimestamp:2026-07-23 11:35:32.135803829 +0000 UTC m=+0.394351102,LastTimestamp:2026-07-23 11:35:32.135803829 +0000 UTC m=+0.394351102,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-236.ec2.internal,}" Jul 23 11:35:32.138522 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.138512 2561 policy_none.go:47] "Start" Jul 23 11:35:32.140545 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.140529 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-236.ec2.internal" event="NodeHasSufficientMemory" Jul 23 11:35:32.140611 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.140555 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-236.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 11:35:32.140611 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.140564 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-236.ec2.internal" event="NodeHasSufficientPID" Jul 23 11:35:32.144576 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.144560 2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tznh7" Jul 23 11:35:32.150215 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:32.150139 2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-236.ec2.internal.18c4e815acc0a3be default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-236.ec2.internal,UID:ip-10-0-134-236.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-134-236.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-134-236.ec2.internal,},FirstTimestamp:2026-07-23 11:35:32.13582227 +0000 UTC m=+0.394369541,LastTimestamp:2026-07-23 11:35:32.13582227 +0000 UTC m=+0.394369541,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-236.ec2.internal,}" Jul 23 11:35:32.153599 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.153586 2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tznh7" Jul 23 11:35:32.158919 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:32.158861 2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-236.ec2.internal.18c4e815acc0dcb1 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-236.ec2.internal,UID:ip-10-0-134-236.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-134-236.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-134-236.ec2.internal,},FirstTimestamp:2026-07-23 11:35:32.135836849 +0000 UTC m=+0.394384119,LastTimestamp:2026-07-23 11:35:32.135836849 +0000 UTC m=+0.394384119,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-236.ec2.internal,}" Jul 23 11:35:32.174701 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.174686 2561 manager.go:337] "Starting Device Plugin manager" Jul 23 11:35:32.174759 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:32.174712 2561 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 23 11:35:32.174759 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.174721 2561 server.go:87] "Starting device plugin registration server" Jul 23 11:35:32.174920 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.174909 2561 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 23 11:35:32.174961 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.174920 2561 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 23 11:35:32.175012 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.174986 2561 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 23 11:35:32.175101 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.175089 2561 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 23 11:35:32.175158 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.175101 2561 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 23 11:35:32.175474 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:32.175459 2561 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 23 11:35:32.175520 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:32.175496 2561 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-236.ec2.internal\" not found" Jul 23 11:35:32.261784 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.261764 2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 23 11:35:32.262839 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.261795 2561 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 23 11:35:32.262839 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.261810 2561 kubelet.go:2448] "Starting kubelet main sync loop" Jul 23 11:35:32.262839 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:32.261838 2561 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 23 11:35:32.263876 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.263862 2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 11:35:32.275664 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.275646 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 11:35:32.276313 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.276300 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-236.ec2.internal" event="NodeHasSufficientMemory" Jul 23 11:35:32.276363 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.276323 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-236.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 11:35:32.276363 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.276333 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-236.ec2.internal" event="NodeHasSufficientPID" Jul 23 11:35:32.276421 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.276367 2561 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-236.ec2.internal" Jul 23 11:35:32.285397 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.285385 2561 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-236.ec2.internal" Jul 23 11:35:32.285436 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:32.285403 2561 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-236.ec2.internal\": node \"ip-10-0-134-236.ec2.internal\" not found" Jul 23 11:35:32.315935 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:32.315893 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-236.ec2.internal\" not found" Jul 23 11:35:32.361945 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.361925 2561 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-236.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-236.ec2.internal"] Jul 23 11:35:32.362017 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.361975 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 11:35:32.362732 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.362718 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-236.ec2.internal" event="NodeHasSufficientMemory" Jul 23 11:35:32.362796 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.362741 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-236.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 11:35:32.362796 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.362750 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-236.ec2.internal" event="NodeHasSufficientPID" Jul 23 11:35:32.364086 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.364074 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 11:35:32.364189 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.364176 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-236.ec2.internal" Jul 23 11:35:32.364227 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.364209 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 11:35:32.364899 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.364885 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-236.ec2.internal" event="NodeHasSufficientMemory" Jul 23 11:35:32.364963 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.364909 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-236.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 11:35:32.364963 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.364919 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-236.ec2.internal" event="NodeHasSufficientPID" Jul 23 11:35:32.365584 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.365568 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-236.ec2.internal" event="NodeHasSufficientMemory" Jul 23 11:35:32.365665 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.365591 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-236.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 11:35:32.365665 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.365604 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-236.ec2.internal" event="NodeHasSufficientPID" Jul 23 11:35:32.366885 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.366867 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-236.ec2.internal" Jul 23 11:35:32.366968 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.366894 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 11:35:32.367537 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.367518 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-236.ec2.internal" event="NodeHasSufficientMemory" Jul 23 11:35:32.367600 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.367544 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-236.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 11:35:32.367600 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.367553 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-236.ec2.internal" event="NodeHasSufficientPID" Jul 23 11:35:32.380529 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:32.380507 2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-236.ec2.internal\" not found" node="ip-10-0-134-236.ec2.internal" Jul 23 11:35:32.384137 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:32.384123 2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-236.ec2.internal\" not found" node="ip-10-0-134-236.ec2.internal" Jul 23 11:35:32.415997 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:32.415979 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-236.ec2.internal\" not found" Jul 23 11:35:32.516372 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:32.516354 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-236.ec2.internal\" not found" Jul 23 11:35:32.519679 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.519645 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f6f6dd6dc338193833341a6aab50de06-config\") pod \"kube-apiserver-proxy-ip-10-0-134-236.ec2.internal\" (UID: \"f6f6dd6dc338193833341a6aab50de06\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-236.ec2.internal" Jul 23 11:35:32.519741 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.519686 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b1b25b788fe3c6b4ce0ebd3a8ca6fd57-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-236.ec2.internal\" (UID: \"b1b25b788fe3c6b4ce0ebd3a8ca6fd57\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-236.ec2.internal" Jul 23 11:35:32.519741 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.519703 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b1b25b788fe3c6b4ce0ebd3a8ca6fd57-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-236.ec2.internal\" (UID: \"b1b25b788fe3c6b4ce0ebd3a8ca6fd57\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-236.ec2.internal" Jul 23 11:35:32.617097 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:32.617037 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-236.ec2.internal\" not found" Jul 23 11:35:32.620324 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.620308 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b1b25b788fe3c6b4ce0ebd3a8ca6fd57-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-236.ec2.internal\" (UID: \"b1b25b788fe3c6b4ce0ebd3a8ca6fd57\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-236.ec2.internal" Jul 23 11:35:32.620367 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.620332 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b1b25b788fe3c6b4ce0ebd3a8ca6fd57-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-236.ec2.internal\" (UID: \"b1b25b788fe3c6b4ce0ebd3a8ca6fd57\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-236.ec2.internal" Jul 23 11:35:32.620367 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.620350 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f6f6dd6dc338193833341a6aab50de06-config\") pod \"kube-apiserver-proxy-ip-10-0-134-236.ec2.internal\" (UID: \"f6f6dd6dc338193833341a6aab50de06\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-236.ec2.internal" Jul 23 11:35:32.620428 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.620378 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f6f6dd6dc338193833341a6aab50de06-config\") pod \"kube-apiserver-proxy-ip-10-0-134-236.ec2.internal\" (UID: \"f6f6dd6dc338193833341a6aab50de06\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-236.ec2.internal" Jul 23 11:35:32.620428 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.620409 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b1b25b788fe3c6b4ce0ebd3a8ca6fd57-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-236.ec2.internal\" (UID: \"b1b25b788fe3c6b4ce0ebd3a8ca6fd57\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-236.ec2.internal" Jul 23 11:35:32.620428 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.620417 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b1b25b788fe3c6b4ce0ebd3a8ca6fd57-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-236.ec2.internal\" (UID: \"b1b25b788fe3c6b4ce0ebd3a8ca6fd57\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-236.ec2.internal" Jul 23 11:35:32.683468 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.683444 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-236.ec2.internal" Jul 23 11:35:32.687277 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:32.687265 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-236.ec2.internal" Jul 23 11:35:32.717951 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:32.717919 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-236.ec2.internal\" not found" Jul 23 11:35:32.818343 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:32.818322 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-236.ec2.internal\" not found" Jul 23 11:35:32.918862 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:32.918814 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-236.ec2.internal\" not found" Jul 23 11:35:34.206159 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:34.206141 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-236.ec2.internal\" not found" Jul 23 11:35:34.220573 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:34.220553 2561 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 11:35:34.248560 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:34.248541 2561 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 23 11:35:34.249051 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:34.248681 2561 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 23 11:35:34.249051 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:34.248719 2561 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 23 11:35:34.306593 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:34.306572 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-236.ec2.internal\" not found" Jul 23 11:35:34.347087 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:34.347049 2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-22 11:30:33 +0000 UTC" deadline="2028-04-27 16:55:34.082022484 +0000 UTC" Jul 23 11:35:34.347087 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:34.347079 2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15461h19m59.734945776s" Jul 23 11:35:34.407513 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:34.407496 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-236.ec2.internal\" not found" Jul 23 11:35:34.418802 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:34.418782 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 11:35:34.420164 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:34.420144 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-236.ec2.internal" event="NodeHasSufficientMemory" Jul 23 11:35:34.420258 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:34.420186 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-236.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 11:35:34.420258 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:34.420201 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-236.ec2.internal" event="NodeHasSufficientPID" Jul 23 11:35:34.431287 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:34.431266 2561 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 11:35:34.503634 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:34.503599 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6f6dd6dc338193833341a6aab50de06.slice/crio-a90e0389ca453c04fb0a1c658bc6ecfb1a9e98560a7746504373f2ef926ce52f WatchSource:0}: Error finding container a90e0389ca453c04fb0a1c658bc6ecfb1a9e98560a7746504373f2ef926ce52f: Status 404 returned error can't find the container with id a90e0389ca453c04fb0a1c658bc6ecfb1a9e98560a7746504373f2ef926ce52f Jul 23 11:35:34.504147 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:34.504132 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1b25b788fe3c6b4ce0ebd3a8ca6fd57.slice/crio-cd0150862617567211f5a2c8df8c102c194718a5a0ddf14b9bb36b064c77911a WatchSource:0}: Error finding container cd0150862617567211f5a2c8df8c102c194718a5a0ddf14b9bb36b064c77911a: Status 404 returned error can't find the container with id cd0150862617567211f5a2c8df8c102c194718a5a0ddf14b9bb36b064c77911a Jul 23 11:35:34.507626 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:34.507602 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-236.ec2.internal\" not found" Jul 23 11:35:34.508487 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:34.508476 2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 11:35:34.607966 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:34.607941 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-236.ec2.internal\" not found" Jul 23 11:35:34.708521 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:34.708480 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-236.ec2.internal\" not found" Jul 23 11:35:34.765069 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:34.765054 2561 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 11:35:34.805252 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:34.805231 2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-236.ec2.internal" Jul 23 11:35:34.818164 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:34.818145 2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 23 11:35:34.819418 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:34.819406 2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-236.ec2.internal" Jul 23 11:35:34.832590 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:34.832574 2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 23 11:35:35.287643 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.287607 2561 apiserver.go:52] "Watching apiserver" Jul 23 11:35:35.298008 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.297975 2561 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 23 11:35:35.298470 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.298447 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-lp5f4","kube-system/kube-apiserver-proxy-ip-10-0-134-236.ec2.internal","openshift-multus/multus-additional-cni-plugins-6jdp4","openshift-multus/network-metrics-daemon-x7q2v","openshift-network-diagnostics/network-check-target-fv772","openshift-ovn-kubernetes/ovnkube-node-92n98","kube-system/global-pull-secret-syncer-8df69","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sf88g","openshift-cluster-node-tuning-operator/tuned-chtf2","openshift-dns/node-resolver-pkwm5","openshift-image-registry/node-ca-6cbrx","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-236.ec2.internal","openshift-multus/multus-8d65t","openshift-network-operator/iptables-alerter-zfngm"] Jul 23 11:35:35.300029 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.300008 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8df69" Jul 23 11:35:35.300126 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:35.300087 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8df69" podUID="1e281f98-6114-4c25-9b31-79c65f9100e2" Jul 23 11:35:35.302012 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.301987 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8d65t" Jul 23 11:35:35.307152 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.303222 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6jdp4" Jul 23 11:35:35.307152 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.305631 2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 23 11:35:35.307152 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.306064 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-xj877\"" Jul 23 11:35:35.307152 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.306243 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 23 11:35:35.307152 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.306392 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 23 11:35:35.307152 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.306519 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 23 11:35:35.307152 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.306632 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 23 11:35:35.307152 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.306665 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 23 11:35:35.307152 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.306827 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-nzlgs\"" Jul 23 11:35:35.307152 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.306835 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7q2v" Jul 23 11:35:35.307152 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:35.306877 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x7q2v" podUID="8886ab91-1d26-47c4-835e-a2905af1aa16" Jul 23 11:35:35.307152 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.306987 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fv772" Jul 23 11:35:35.307152 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.307035 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 23 11:35:35.307152 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:35.307053 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fv772" podUID="c92ad538-9642-4c08-b92c-886738d3c413" Jul 23 11:35:35.308862 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.308004 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-92n98" Jul 23 11:35:35.309573 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.309555 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-chtf2" Jul 23 11:35:35.310246 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.310229 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 23 11:35:35.310458 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.310445 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 23 11:35:35.310631 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.310617 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 23 11:35:35.310871 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.310854 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-l49kp\"" Jul 23 11:35:35.311336 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.311062 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 23 11:35:35.311336 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.311125 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lp5f4" Jul 23 11:35:35.311679 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.311662 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 23 11:35:35.311842 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.311824 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 23 11:35:35.311956 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.311943 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 23 11:35:35.312250 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.312065 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-twz25\"" Jul 23 11:35:35.312250 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.311944 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 23 11:35:35.312426 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.312293 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zfngm" Jul 23 11:35:35.313319 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.313300 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 23 11:35:35.313417 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.313389 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-qcvpg\"" Jul 23 11:35:35.313562 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.313544 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 23 11:35:35.314953 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.314415 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-pkwm5" Jul 23 11:35:35.314953 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.314676 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 23 11:35:35.314953 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.314794 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-9jgrc\"" Jul 23 11:35:35.314953 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.314883 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f898b85f-57f0-4973-9e99-117e9fd60fa8-cnibin\") pod \"multus-additional-cni-plugins-6jdp4\" (UID: \"f898b85f-57f0-4973-9e99-117e9fd60fa8\") " pod="openshift-multus/multus-additional-cni-plugins-6jdp4" Jul 23 11:35:35.314953 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.314903 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f898b85f-57f0-4973-9e99-117e9fd60fa8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6jdp4\" (UID: \"f898b85f-57f0-4973-9e99-117e9fd60fa8\") " pod="openshift-multus/multus-additional-cni-plugins-6jdp4" Jul 23 11:35:35.314953 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.314885 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 23 11:35:35.314953 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.314948 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2c75320d-d67b-4e9e-b648-cc748f5a2870-host-run-ovn-kubernetes\") pod \"ovnkube-node-92n98\" (UID: \"2c75320d-d67b-4e9e-b648-cc748f5a2870\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n98" Jul 23 11:35:35.315310 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.314966 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2c75320d-d67b-4e9e-b648-cc748f5a2870-ovnkube-config\") pod \"ovnkube-node-92n98\" (UID: \"2c75320d-d67b-4e9e-b648-cc748f5a2870\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n98" Jul 23 11:35:35.315310 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.314995 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1e281f98-6114-4c25-9b31-79c65f9100e2-original-pull-secret\") pod \"global-pull-secret-syncer-8df69\" (UID: \"1e281f98-6114-4c25-9b31-79c65f9100e2\") " pod="kube-system/global-pull-secret-syncer-8df69" Jul 23 11:35:35.315310 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.315009 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/792c59f0-f23f-4694-9cde-37b9431760f4-os-release\") pod \"multus-8d65t\" (UID: \"792c59f0-f23f-4694-9cde-37b9431760f4\") " pod="openshift-multus/multus-8d65t" Jul 23 11:35:35.315310 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.315033 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf7hx\" (UniqueName: \"kubernetes.io/projected/792c59f0-f23f-4694-9cde-37b9431760f4-kube-api-access-pf7hx\") pod \"multus-8d65t\" (UID: \"792c59f0-f23f-4694-9cde-37b9431760f4\") " pod="openshift-multus/multus-8d65t" Jul 23 11:35:35.315310 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.315094 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8886ab91-1d26-47c4-835e-a2905af1aa16-metrics-certs\") pod \"network-metrics-daemon-x7q2v\" (UID: \"8886ab91-1d26-47c4-835e-a2905af1aa16\") " pod="openshift-multus/network-metrics-daemon-x7q2v" Jul 23 11:35:35.315310 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.315107 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 23 11:35:35.315310 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.315125 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2c75320d-d67b-4e9e-b648-cc748f5a2870-var-lib-openvswitch\") pod \"ovnkube-node-92n98\" (UID: \"2c75320d-d67b-4e9e-b648-cc748f5a2870\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n98" Jul 23 11:35:35.315310 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.315148 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2c75320d-d67b-4e9e-b648-cc748f5a2870-etc-openvswitch\") pod \"ovnkube-node-92n98\" (UID: \"2c75320d-d67b-4e9e-b648-cc748f5a2870\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n98" Jul 23 11:35:35.315310 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.315169 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2c75320d-d67b-4e9e-b648-cc748f5a2870-run-ovn\") pod \"ovnkube-node-92n98\" (UID: \"2c75320d-d67b-4e9e-b648-cc748f5a2870\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n98" Jul 23 11:35:35.315310 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.315200 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/792c59f0-f23f-4694-9cde-37b9431760f4-multus-socket-dir-parent\") pod \"multus-8d65t\" (UID: \"792c59f0-f23f-4694-9cde-37b9431760f4\") " pod="openshift-multus/multus-8d65t" Jul 23 11:35:35.315310 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.315221 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/792c59f0-f23f-4694-9cde-37b9431760f4-host-run-netns\") pod \"multus-8d65t\" (UID: \"792c59f0-f23f-4694-9cde-37b9431760f4\") " pod="openshift-multus/multus-8d65t" Jul 23 11:35:35.315310 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.315244 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/792c59f0-f23f-4694-9cde-37b9431760f4-multus-daemon-config\") pod \"multus-8d65t\" (UID: \"792c59f0-f23f-4694-9cde-37b9431760f4\") " pod="openshift-multus/multus-8d65t" Jul 23 11:35:35.315310 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.315282 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f898b85f-57f0-4973-9e99-117e9fd60fa8-os-release\") pod \"multus-additional-cni-plugins-6jdp4\" (UID: \"f898b85f-57f0-4973-9e99-117e9fd60fa8\") " pod="openshift-multus/multus-additional-cni-plugins-6jdp4" Jul 23 11:35:35.315310 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.315309 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2c75320d-d67b-4e9e-b648-cc748f5a2870-host-kubelet\") pod \"ovnkube-node-92n98\" (UID: \"2c75320d-d67b-4e9e-b648-cc748f5a2870\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n98" Jul 23 11:35:35.315840 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.315334 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2c75320d-d67b-4e9e-b648-cc748f5a2870-host-run-netns\") pod \"ovnkube-node-92n98\" (UID: \"2c75320d-d67b-4e9e-b648-cc748f5a2870\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n98" Jul 23 11:35:35.315840 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.315375 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2c75320d-d67b-4e9e-b648-cc748f5a2870-host-cni-bin\") pod \"ovnkube-node-92n98\" (UID: \"2c75320d-d67b-4e9e-b648-cc748f5a2870\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n98" Jul 23 11:35:35.315840 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.315417 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2c75320d-d67b-4e9e-b648-cc748f5a2870-ovnkube-script-lib\") pod \"ovnkube-node-92n98\" (UID: \"2c75320d-d67b-4e9e-b648-cc748f5a2870\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n98" Jul 23 11:35:35.315840 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.315440 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/792c59f0-f23f-4694-9cde-37b9431760f4-cnibin\") pod \"multus-8d65t\" (UID: \"792c59f0-f23f-4694-9cde-37b9431760f4\") " pod="openshift-multus/multus-8d65t" Jul 23 11:35:35.315840 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.315482 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/792c59f0-f23f-4694-9cde-37b9431760f4-host-run-multus-certs\") pod \"multus-8d65t\" (UID: \"792c59f0-f23f-4694-9cde-37b9431760f4\") " pod="openshift-multus/multus-8d65t" Jul 23 11:35:35.315840 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.315521 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggtpg\" (UniqueName: \"kubernetes.io/projected/f898b85f-57f0-4973-9e99-117e9fd60fa8-kube-api-access-ggtpg\") pod \"multus-additional-cni-plugins-6jdp4\" (UID: \"f898b85f-57f0-4973-9e99-117e9fd60fa8\") " pod="openshift-multus/multus-additional-cni-plugins-6jdp4" Jul 23 11:35:35.315840 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.315545 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mghl5\" (UniqueName: \"kubernetes.io/projected/8886ab91-1d26-47c4-835e-a2905af1aa16-kube-api-access-mghl5\") pod \"network-metrics-daemon-x7q2v\" (UID: \"8886ab91-1d26-47c4-835e-a2905af1aa16\") " pod="openshift-multus/network-metrics-daemon-x7q2v" Jul 23 11:35:35.315840 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.315582 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnlxz\" (UniqueName: \"kubernetes.io/projected/c92ad538-9642-4c08-b92c-886738d3c413-kube-api-access-gnlxz\") pod \"network-check-target-fv772\" (UID: \"c92ad538-9642-4c08-b92c-886738d3c413\") " pod="openshift-network-diagnostics/network-check-target-fv772" Jul 23 11:35:35.315840 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.315638 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2c75320d-d67b-4e9e-b648-cc748f5a2870-log-socket\") pod \"ovnkube-node-92n98\" (UID: \"2c75320d-d67b-4e9e-b648-cc748f5a2870\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n98" Jul 23 11:35:35.315840 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.315672 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2c75320d-d67b-4e9e-b648-cc748f5a2870-env-overrides\") pod \"ovnkube-node-92n98\" (UID: \"2c75320d-d67b-4e9e-b648-cc748f5a2870\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n98" Jul 23 11:35:35.315840 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.315721 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1e281f98-6114-4c25-9b31-79c65f9100e2-kubelet-config\") pod \"global-pull-secret-syncer-8df69\" (UID: \"1e281f98-6114-4c25-9b31-79c65f9100e2\") " pod="kube-system/global-pull-secret-syncer-8df69" Jul 23 11:35:35.315840 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.315748 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1e281f98-6114-4c25-9b31-79c65f9100e2-dbus\") pod \"global-pull-secret-syncer-8df69\" (UID: \"1e281f98-6114-4c25-9b31-79c65f9100e2\") " pod="kube-system/global-pull-secret-syncer-8df69" Jul 23 11:35:35.315840 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.315780 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/792c59f0-f23f-4694-9cde-37b9431760f4-system-cni-dir\") pod \"multus-8d65t\" (UID: \"792c59f0-f23f-4694-9cde-37b9431760f4\") " pod="openshift-multus/multus-8d65t" Jul 23 11:35:35.315840 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.315810 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f898b85f-57f0-4973-9e99-117e9fd60fa8-system-cni-dir\") pod \"multus-additional-cni-plugins-6jdp4\" (UID: \"f898b85f-57f0-4973-9e99-117e9fd60fa8\") " pod="openshift-multus/multus-additional-cni-plugins-6jdp4" Jul 23 11:35:35.315840 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.315833 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f898b85f-57f0-4973-9e99-117e9fd60fa8-cni-binary-copy\") pod \"multus-additional-cni-plugins-6jdp4\" (UID: \"f898b85f-57f0-4973-9e99-117e9fd60fa8\") " pod="openshift-multus/multus-additional-cni-plugins-6jdp4" Jul 23 11:35:35.316504 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.315858 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f898b85f-57f0-4973-9e99-117e9fd60fa8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6jdp4\" (UID: \"f898b85f-57f0-4973-9e99-117e9fd60fa8\") " pod="openshift-multus/multus-additional-cni-plugins-6jdp4" Jul 23 11:35:35.316504 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.315880 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2c75320d-d67b-4e9e-b648-cc748f5a2870-run-systemd\") pod \"ovnkube-node-92n98\" (UID: \"2c75320d-d67b-4e9e-b648-cc748f5a2870\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n98" Jul 23 11:35:35.316504 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.315890 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6cbrx" Jul 23 11:35:35.316504 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.315923 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2c75320d-d67b-4e9e-b648-cc748f5a2870-ovn-node-metrics-cert\") pod \"ovnkube-node-92n98\" (UID: \"2c75320d-d67b-4e9e-b648-cc748f5a2870\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n98" Jul 23 11:35:35.316504 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.315947 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/792c59f0-f23f-4694-9cde-37b9431760f4-multus-cni-dir\") pod \"multus-8d65t\" (UID: \"792c59f0-f23f-4694-9cde-37b9431760f4\") " pod="openshift-multus/multus-8d65t" Jul 23 11:35:35.316504 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.315969 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/792c59f0-f23f-4694-9cde-37b9431760f4-host-run-k8s-cni-cncf-io\") pod \"multus-8d65t\" (UID: \"792c59f0-f23f-4694-9cde-37b9431760f4\") " pod="openshift-multus/multus-8d65t" Jul 23 11:35:35.316504 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.315990 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/792c59f0-f23f-4694-9cde-37b9431760f4-host-var-lib-kubelet\") pod \"multus-8d65t\" (UID: \"792c59f0-f23f-4694-9cde-37b9431760f4\") " pod="openshift-multus/multus-8d65t" Jul 23 11:35:35.316504 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.316011 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2c75320d-d67b-4e9e-b648-cc748f5a2870-systemd-units\") pod \"ovnkube-node-92n98\" (UID: \"2c75320d-d67b-4e9e-b648-cc748f5a2870\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n98" Jul 23 11:35:35.316504 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.316051 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2c75320d-d67b-4e9e-b648-cc748f5a2870-run-openvswitch\") pod \"ovnkube-node-92n98\" (UID: \"2c75320d-d67b-4e9e-b648-cc748f5a2870\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n98" Jul 23 11:35:35.316504 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.316077 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2c75320d-d67b-4e9e-b648-cc748f5a2870-node-log\") pod \"ovnkube-node-92n98\" (UID: \"2c75320d-d67b-4e9e-b648-cc748f5a2870\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n98" Jul 23 11:35:35.316504 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.316102 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2c75320d-d67b-4e9e-b648-cc748f5a2870-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-92n98\" (UID: \"2c75320d-d67b-4e9e-b648-cc748f5a2870\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n98" Jul 23 11:35:35.316504 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.316127 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/792c59f0-f23f-4694-9cde-37b9431760f4-cni-binary-copy\") pod \"multus-8d65t\" (UID: \"792c59f0-f23f-4694-9cde-37b9431760f4\") " pod="openshift-multus/multus-8d65t" Jul 23 11:35:35.316504 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.316169 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/792c59f0-f23f-4694-9cde-37b9431760f4-host-var-lib-cni-bin\") pod \"multus-8d65t\" (UID: \"792c59f0-f23f-4694-9cde-37b9431760f4\") " pod="openshift-multus/multus-8d65t" Jul 23 11:35:35.316504 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.316191 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/792c59f0-f23f-4694-9cde-37b9431760f4-hostroot\") pod \"multus-8d65t\" (UID: \"792c59f0-f23f-4694-9cde-37b9431760f4\") " pod="openshift-multus/multus-8d65t" Jul 23 11:35:35.316504 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.316212 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/792c59f0-f23f-4694-9cde-37b9431760f4-multus-conf-dir\") pod \"multus-8d65t\" (UID: \"792c59f0-f23f-4694-9cde-37b9431760f4\") " pod="openshift-multus/multus-8d65t" Jul 23 11:35:35.316504 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.316240 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2c75320d-d67b-4e9e-b648-cc748f5a2870-host-cni-netd\") pod \"ovnkube-node-92n98\" (UID: \"2c75320d-d67b-4e9e-b648-cc748f5a2870\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n98" Jul 23 11:35:35.316504 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.316280 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f898b85f-57f0-4973-9e99-117e9fd60fa8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6jdp4\" (UID: \"f898b85f-57f0-4973-9e99-117e9fd60fa8\") " pod="openshift-multus/multus-additional-cni-plugins-6jdp4" Jul 23 11:35:35.317292 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.316312 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2c75320d-d67b-4e9e-b648-cc748f5a2870-host-slash\") pod \"ovnkube-node-92n98\" (UID: \"2c75320d-d67b-4e9e-b648-cc748f5a2870\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n98" Jul 23 11:35:35.317292 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.316336 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nphvk\" (UniqueName: \"kubernetes.io/projected/2c75320d-d67b-4e9e-b648-cc748f5a2870-kube-api-access-nphvk\") pod \"ovnkube-node-92n98\" (UID: \"2c75320d-d67b-4e9e-b648-cc748f5a2870\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n98" Jul 23 11:35:35.317292 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.316358 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/792c59f0-f23f-4694-9cde-37b9431760f4-host-var-lib-cni-multus\") pod \"multus-8d65t\" (UID: \"792c59f0-f23f-4694-9cde-37b9431760f4\") " pod="openshift-multus/multus-8d65t" Jul 23 11:35:35.317292 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.316381 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/792c59f0-f23f-4694-9cde-37b9431760f4-etc-kubernetes\") pod \"multus-8d65t\" (UID: \"792c59f0-f23f-4694-9cde-37b9431760f4\") " pod="openshift-multus/multus-8d65t" Jul 23 11:35:35.317292 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.316616 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 23 11:35:35.317292 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.316984 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-qbvb9\"" Jul 23 11:35:35.317555 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.317334 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 23 11:35:35.317669 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.317631 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sf88g" Jul 23 11:35:35.318304 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.318134 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 23 11:35:35.318304 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.318226 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 23 11:35:35.318562 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.318540 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 23 11:35:35.319005 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.318985 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-dsntn\"" Jul 23 11:35:35.319900 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.319878 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 23 11:35:35.320011 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.319930 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 23 11:35:35.320011 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.319942 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-gzxvp\"" Jul 23 11:35:35.320153 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.320116 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 23 11:35:35.324778 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.324569 2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 23 11:35:35.349430 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.349407 2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-5qhzq" Jul 23 11:35:35.364650 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.363802 2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-5qhzq" Jul 23 11:35:35.406641 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.406611 2561 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 23 11:35:35.417068 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.417040 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2c75320d-d67b-4e9e-b648-cc748f5a2870-host-cni-netd\") pod \"ovnkube-node-92n98\" (UID: \"2c75320d-d67b-4e9e-b648-cc748f5a2870\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n98" Jul 23 11:35:35.417068 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.417071 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2c75320d-d67b-4e9e-b648-cc748f5a2870-host-slash\") pod \"ovnkube-node-92n98\" (UID: \"2c75320d-d67b-4e9e-b648-cc748f5a2870\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n98" Jul 23 11:35:35.417271 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.417090 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nphvk\" (UniqueName: \"kubernetes.io/projected/2c75320d-d67b-4e9e-b648-cc748f5a2870-kube-api-access-nphvk\") pod \"ovnkube-node-92n98\" (UID: \"2c75320d-d67b-4e9e-b648-cc748f5a2870\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n98" Jul 23 11:35:35.417271 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.417112 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/792c59f0-f23f-4694-9cde-37b9431760f4-host-var-lib-cni-multus\") pod \"multus-8d65t\" (UID: \"792c59f0-f23f-4694-9cde-37b9431760f4\") " pod="openshift-multus/multus-8d65t" Jul 23 11:35:35.417271 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.417139 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn7s2\" (UniqueName: \"kubernetes.io/projected/917d3c0c-f95b-43ac-a39a-b21dfdc0c07c-kube-api-access-xn7s2\") pod \"iptables-alerter-zfngm\" (UID: \"917d3c0c-f95b-43ac-a39a-b21dfdc0c07c\") " pod="openshift-network-operator/iptables-alerter-zfngm" Jul 23 11:35:35.417271 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.417158 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2c75320d-d67b-4e9e-b648-cc748f5a2870-host-slash\") pod \"ovnkube-node-92n98\" (UID: \"2c75320d-d67b-4e9e-b648-cc748f5a2870\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n98" Jul 23 11:35:35.417271 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.417164 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f898b85f-57f0-4973-9e99-117e9fd60fa8-cnibin\") pod \"multus-additional-cni-plugins-6jdp4\" (UID: \"f898b85f-57f0-4973-9e99-117e9fd60fa8\") " pod="openshift-multus/multus-additional-cni-plugins-6jdp4" Jul 23 11:35:35.417271 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.417156 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2c75320d-d67b-4e9e-b648-cc748f5a2870-host-cni-netd\") pod \"ovnkube-node-92n98\" (UID: \"2c75320d-d67b-4e9e-b648-cc748f5a2870\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n98" Jul 23 11:35:35.417271 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.417184 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/792c59f0-f23f-4694-9cde-37b9431760f4-host-var-lib-cni-multus\") pod \"multus-8d65t\" (UID: \"792c59f0-f23f-4694-9cde-37b9431760f4\") " pod="openshift-multus/multus-8d65t" Jul 23 11:35:35.417271 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.417218 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f898b85f-57f0-4973-9e99-117e9fd60fa8-cnibin\") pod \"multus-additional-cni-plugins-6jdp4\" (UID: \"f898b85f-57f0-4973-9e99-117e9fd60fa8\") " pod="openshift-multus/multus-additional-cni-plugins-6jdp4" Jul 23 11:35:35.417271 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.417190 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f898b85f-57f0-4973-9e99-117e9fd60fa8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6jdp4\" (UID: \"f898b85f-57f0-4973-9e99-117e9fd60fa8\") " pod="openshift-multus/multus-additional-cni-plugins-6jdp4" Jul 23 11:35:35.417580 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.417302 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2c75320d-d67b-4e9e-b648-cc748f5a2870-host-run-ovn-kubernetes\") pod \"ovnkube-node-92n98\" (UID: \"2c75320d-d67b-4e9e-b648-cc748f5a2870\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n98" Jul 23 11:35:35.417580 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.417325 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ebf70217-31df-4e81-be74-df699db721f0-registration-dir\") pod \"aws-ebs-csi-driver-node-sf88g\" (UID: \"ebf70217-31df-4e81-be74-df699db721f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sf88g" Jul 23 11:35:35.417580 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.417342 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ba1ca519-e6da-43b0-9659-ec51fac539a4-tmp\") pod \"tuned-chtf2\" (UID: \"ba1ca519-e6da-43b0-9659-ec51fac539a4\") " pod="openshift-cluster-node-tuning-operator/tuned-chtf2" Jul 23 11:35:35.417580 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.417364 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/792c59f0-f23f-4694-9cde-37b9431760f4-os-release\") pod \"multus-8d65t\" (UID: \"792c59f0-f23f-4694-9cde-37b9431760f4\") " pod="openshift-multus/multus-8d65t" Jul 23 11:35:35.417580 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.417388 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/620c5e1f-97de-42e6-b752-50d951728a5d-konnectivity-ca\") pod \"konnectivity-agent-lp5f4\" (UID: \"620c5e1f-97de-42e6-b752-50d951728a5d\") " pod="kube-system/konnectivity-agent-lp5f4" Jul 23 11:35:35.417580 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.417400 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2c75320d-d67b-4e9e-b648-cc748f5a2870-host-run-ovn-kubernetes\") pod \"ovnkube-node-92n98\" (UID: \"2c75320d-d67b-4e9e-b648-cc748f5a2870\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n98" Jul 23 11:35:35.417580 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.417410 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ebf70217-31df-4e81-be74-df699db721f0-device-dir\") pod \"aws-ebs-csi-driver-node-sf88g\" (UID: \"ebf70217-31df-4e81-be74-df699db721f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sf88g" Jul 23 11:35:35.417580 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.417433 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2c75320d-d67b-4e9e-b648-cc748f5a2870-etc-openvswitch\") pod \"ovnkube-node-92n98\" (UID: \"2c75320d-d67b-4e9e-b648-cc748f5a2870\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n98" Jul 23 11:35:35.417580 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.417456 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2c75320d-d67b-4e9e-b648-cc748f5a2870-run-ovn\") pod \"ovnkube-node-92n98\" (UID: \"2c75320d-d67b-4e9e-b648-cc748f5a2870\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n98" Jul 23 11:35:35.417580 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.417468 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/792c59f0-f23f-4694-9cde-37b9431760f4-os-release\") pod \"multus-8d65t\" (UID: \"792c59f0-f23f-4694-9cde-37b9431760f4\") " pod="openshift-multus/multus-8d65t" Jul 23 11:35:35.417580 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.417479 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/792c59f0-f23f-4694-9cde-37b9431760f4-multus-socket-dir-parent\") pod \"multus-8d65t\" (UID: \"792c59f0-f23f-4694-9cde-37b9431760f4\") " pod="openshift-multus/multus-8d65t" Jul 23 11:35:35.417580 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.417502 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/792c59f0-f23f-4694-9cde-37b9431760f4-multus-daemon-config\") pod \"multus-8d65t\" (UID: \"792c59f0-f23f-4694-9cde-37b9431760f4\") " pod="openshift-multus/multus-8d65t" Jul 23 11:35:35.417580 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.417508 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2c75320d-d67b-4e9e-b648-cc748f5a2870-run-ovn\") pod \"ovnkube-node-92n98\" (UID: \"2c75320d-d67b-4e9e-b648-cc748f5a2870\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n98" Jul 23 11:35:35.417580 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.417509 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2c75320d-d67b-4e9e-b648-cc748f5a2870-etc-openvswitch\") pod \"ovnkube-node-92n98\" (UID: \"2c75320d-d67b-4e9e-b648-cc748f5a2870\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n98" Jul 23 11:35:35.417580 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.417526 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ebf70217-31df-4e81-be74-df699db721f0-socket-dir\") pod \"aws-ebs-csi-driver-node-sf88g\" (UID: \"ebf70217-31df-4e81-be74-df699db721f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sf88g" Jul 23 11:35:35.417580 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.417549 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a17ee9c0-c387-4e9d-a6a4-481501f5b97b-hosts-file\") pod \"node-resolver-pkwm5\" (UID: \"a17ee9c0-c387-4e9d-a6a4-481501f5b97b\") " pod="openshift-dns/node-resolver-pkwm5" Jul 23 11:35:35.417580 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.417569 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/792c59f0-f23f-4694-9cde-37b9431760f4-multus-socket-dir-parent\") pod \"multus-8d65t\" (UID: \"792c59f0-f23f-4694-9cde-37b9431760f4\") " pod="openshift-multus/multus-8d65t" Jul 23 11:35:35.418147 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.417571 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a17ee9c0-c387-4e9d-a6a4-481501f5b97b-tmp-dir\") pod \"node-resolver-pkwm5\" (UID: \"a17ee9c0-c387-4e9d-a6a4-481501f5b97b\") " pod="openshift-dns/node-resolver-pkwm5" Jul 23 11:35:35.418147 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.417615 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f898b85f-57f0-4973-9e99-117e9fd60fa8-os-release\") pod \"multus-additional-cni-plugins-6jdp4\" (UID: \"f898b85f-57f0-4973-9e99-117e9fd60fa8\") " pod="openshift-multus/multus-additional-cni-plugins-6jdp4" Jul 23 11:35:35.418147 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.417641 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2c75320d-d67b-4e9e-b648-cc748f5a2870-host-cni-bin\") pod \"ovnkube-node-92n98\" (UID: \"2c75320d-d67b-4e9e-b648-cc748f5a2870\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n98" Jul 23 11:35:35.418147 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.417664 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/792c59f0-f23f-4694-9cde-37b9431760f4-cnibin\") pod \"multus-8d65t\" (UID: \"792c59f0-f23f-4694-9cde-37b9431760f4\") " pod="openshift-multus/multus-8d65t" Jul 23 11:35:35.418147 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.417690 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ebf70217-31df-4e81-be74-df699db721f0-sys-fs\") pod \"aws-ebs-csi-driver-node-sf88g\" (UID: \"ebf70217-31df-4e81-be74-df699db721f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sf88g" Jul 23 11:35:35.418147 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.417712 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba1ca519-e6da-43b0-9659-ec51fac539a4-var-lib-kubelet\") pod \"tuned-chtf2\" (UID: \"ba1ca519-e6da-43b0-9659-ec51fac539a4\") " pod="openshift-cluster-node-tuning-operator/tuned-chtf2" Jul 23 11:35:35.418147 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.417716 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f898b85f-57f0-4973-9e99-117e9fd60fa8-os-release\") pod \"multus-additional-cni-plugins-6jdp4\" (UID: \"f898b85f-57f0-4973-9e99-117e9fd60fa8\") " pod="openshift-multus/multus-additional-cni-plugins-6jdp4" Jul 23 11:35:35.418147 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.417744 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mghl5\" (UniqueName: \"kubernetes.io/projected/8886ab91-1d26-47c4-835e-a2905af1aa16-kube-api-access-mghl5\") pod \"network-metrics-daemon-x7q2v\" (UID: \"8886ab91-1d26-47c4-835e-a2905af1aa16\") " pod="openshift-multus/network-metrics-daemon-x7q2v" Jul 23 11:35:35.418147 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.417760 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f898b85f-57f0-4973-9e99-117e9fd60fa8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6jdp4\" (UID: \"f898b85f-57f0-4973-9e99-117e9fd60fa8\") " pod="openshift-multus/multus-additional-cni-plugins-6jdp4" Jul 23 11:35:35.418147 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.417769 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnlxz\" (UniqueName: \"kubernetes.io/projected/c92ad538-9642-4c08-b92c-886738d3c413-kube-api-access-gnlxz\") pod \"network-check-target-fv772\" (UID: \"c92ad538-9642-4c08-b92c-886738d3c413\") " pod="openshift-network-diagnostics/network-check-target-fv772" Jul 23 11:35:35.418147 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.417793 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1e281f98-6114-4c25-9b31-79c65f9100e2-kubelet-config\") pod \"global-pull-secret-syncer-8df69\" (UID: \"1e281f98-6114-4c25-9b31-79c65f9100e2\") " pod="kube-system/global-pull-secret-syncer-8df69" Jul 23 11:35:35.418147 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.417798 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2c75320d-d67b-4e9e-b648-cc748f5a2870-host-cni-bin\") pod \"ovnkube-node-92n98\" (UID: \"2c75320d-d67b-4e9e-b648-cc748f5a2870\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n98" Jul 23 11:35:35.418147 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.417816 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/792c59f0-f23f-4694-9cde-37b9431760f4-system-cni-dir\") pod \"multus-8d65t\" (UID: \"792c59f0-f23f-4694-9cde-37b9431760f4\") " pod="openshift-multus/multus-8d65t" Jul 23 11:35:35.418147 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.417771 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/792c59f0-f23f-4694-9cde-37b9431760f4-cnibin\") pod \"multus-8d65t\" (UID: \"792c59f0-f23f-4694-9cde-37b9431760f4\") " pod="openshift-multus/multus-8d65t" Jul 23 11:35:35.418147 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.417840 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wp8xl\" (UniqueName: \"kubernetes.io/projected/aa374cc8-936e-41ca-a905-c0449df6825d-kube-api-access-wp8xl\") pod \"node-ca-6cbrx\" (UID: \"aa374cc8-936e-41ca-a905-c0449df6825d\") " pod="openshift-image-registry/node-ca-6cbrx" Jul 23 11:35:35.418147 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.417864 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ba1ca519-e6da-43b0-9659-ec51fac539a4-etc-modprobe-d\") pod \"tuned-chtf2\" (UID: \"ba1ca519-e6da-43b0-9659-ec51fac539a4\") " pod="openshift-cluster-node-tuning-operator/tuned-chtf2" Jul 23 11:35:35.418147 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.417945 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1e281f98-6114-4c25-9b31-79c65f9100e2-kubelet-config\") pod \"global-pull-secret-syncer-8df69\" (UID: \"1e281f98-6114-4c25-9b31-79c65f9100e2\") " pod="kube-system/global-pull-secret-syncer-8df69" Jul 23 11:35:35.418962 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.417992 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/792c59f0-f23f-4694-9cde-37b9431760f4-system-cni-dir\") pod \"multus-8d65t\" (UID: \"792c59f0-f23f-4694-9cde-37b9431760f4\") " pod="openshift-multus/multus-8d65t" Jul 23 11:35:35.418962 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.418026 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ba1ca519-e6da-43b0-9659-ec51fac539a4-etc-sysctl-d\") pod \"tuned-chtf2\" (UID: \"ba1ca519-e6da-43b0-9659-ec51fac539a4\") " pod="openshift-cluster-node-tuning-operator/tuned-chtf2" Jul 23 11:35:35.418962 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.418042 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/792c59f0-f23f-4694-9cde-37b9431760f4-multus-daemon-config\") pod \"multus-8d65t\" (UID: \"792c59f0-f23f-4694-9cde-37b9431760f4\") " pod="openshift-multus/multus-8d65t" Jul 23 11:35:35.418962 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.418065 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f898b85f-57f0-4973-9e99-117e9fd60fa8-system-cni-dir\") pod \"multus-additional-cni-plugins-6jdp4\" (UID: \"f898b85f-57f0-4973-9e99-117e9fd60fa8\") " pod="openshift-multus/multus-additional-cni-plugins-6jdp4" Jul 23 11:35:35.418962 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.418091 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2c75320d-d67b-4e9e-b648-cc748f5a2870-ovn-node-metrics-cert\") pod \"ovnkube-node-92n98\" (UID: \"2c75320d-d67b-4e9e-b648-cc748f5a2870\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n98" Jul 23 11:35:35.418962 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.418106 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f898b85f-57f0-4973-9e99-117e9fd60fa8-system-cni-dir\") pod \"multus-additional-cni-plugins-6jdp4\" (UID: \"f898b85f-57f0-4973-9e99-117e9fd60fa8\") " pod="openshift-multus/multus-additional-cni-plugins-6jdp4" Jul 23 11:35:35.418962 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.418118 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/792c59f0-f23f-4694-9cde-37b9431760f4-host-var-lib-kubelet\") pod \"multus-8d65t\" (UID: \"792c59f0-f23f-4694-9cde-37b9431760f4\") " pod="openshift-multus/multus-8d65t" Jul 23 11:35:35.418962 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.418163 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/792c59f0-f23f-4694-9cde-37b9431760f4-host-var-lib-kubelet\") pod \"multus-8d65t\" (UID: \"792c59f0-f23f-4694-9cde-37b9431760f4\") " pod="openshift-multus/multus-8d65t" Jul 23 11:35:35.418962 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.418204 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/792c59f0-f23f-4694-9cde-37b9431760f4-hostroot\") pod \"multus-8d65t\" (UID: \"792c59f0-f23f-4694-9cde-37b9431760f4\") " pod="openshift-multus/multus-8d65t" Jul 23 11:35:35.418962 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.418242 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ba1ca519-e6da-43b0-9659-ec51fac539a4-run\") pod \"tuned-chtf2\" (UID: \"ba1ca519-e6da-43b0-9659-ec51fac539a4\") " pod="openshift-cluster-node-tuning-operator/tuned-chtf2" Jul 23 11:35:35.418962 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.418273 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2c75320d-d67b-4e9e-b648-cc748f5a2870-run-openvswitch\") pod \"ovnkube-node-92n98\" (UID: \"2c75320d-d67b-4e9e-b648-cc748f5a2870\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n98" Jul 23 11:35:35.418962 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.418300 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2c75320d-d67b-4e9e-b648-cc748f5a2870-node-log\") pod \"ovnkube-node-92n98\" (UID: \"2c75320d-d67b-4e9e-b648-cc748f5a2870\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n98" Jul 23 11:35:35.418962 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.418318 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/792c59f0-f23f-4694-9cde-37b9431760f4-hostroot\") pod \"multus-8d65t\" (UID: \"792c59f0-f23f-4694-9cde-37b9431760f4\") " pod="openshift-multus/multus-8d65t" Jul 23 11:35:35.418962 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.418324 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/792c59f0-f23f-4694-9cde-37b9431760f4-cni-binary-copy\") pod \"multus-8d65t\" (UID: \"792c59f0-f23f-4694-9cde-37b9431760f4\") " pod="openshift-multus/multus-8d65t" Jul 23 11:35:35.418962 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.418341 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2c75320d-d67b-4e9e-b648-cc748f5a2870-run-openvswitch\") pod \"ovnkube-node-92n98\" (UID: \"2c75320d-d67b-4e9e-b648-cc748f5a2870\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n98" Jul 23 11:35:35.418962 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.418356 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/792c59f0-f23f-4694-9cde-37b9431760f4-host-var-lib-cni-bin\") pod \"multus-8d65t\" (UID: \"792c59f0-f23f-4694-9cde-37b9431760f4\") " pod="openshift-multus/multus-8d65t" Jul 23 11:35:35.418962 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.418385 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7776\" (UniqueName: \"kubernetes.io/projected/ba1ca519-e6da-43b0-9659-ec51fac539a4-kube-api-access-c7776\") pod \"tuned-chtf2\" (UID: \"ba1ca519-e6da-43b0-9659-ec51fac539a4\") " pod="openshift-cluster-node-tuning-operator/tuned-chtf2" Jul 23 11:35:35.418962 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.418390 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2c75320d-d67b-4e9e-b648-cc748f5a2870-node-log\") pod \"ovnkube-node-92n98\" (UID: \"2c75320d-d67b-4e9e-b648-cc748f5a2870\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n98" Jul 23 11:35:35.419790 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.418416 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f898b85f-57f0-4973-9e99-117e9fd60fa8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6jdp4\" (UID: \"f898b85f-57f0-4973-9e99-117e9fd60fa8\") " pod="openshift-multus/multus-additional-cni-plugins-6jdp4" Jul 23 11:35:35.419790 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.418427 2561 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 23 11:35:35.419790 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.418439 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2c75320d-d67b-4e9e-b648-cc748f5a2870-ovnkube-config\") pod \"ovnkube-node-92n98\" (UID: \"2c75320d-d67b-4e9e-b648-cc748f5a2870\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n98" Jul 23 11:35:35.419790 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.418446 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/792c59f0-f23f-4694-9cde-37b9431760f4-host-var-lib-cni-bin\") pod \"multus-8d65t\" (UID: \"792c59f0-f23f-4694-9cde-37b9431760f4\") " pod="openshift-multus/multus-8d65t" Jul 23 11:35:35.419790 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.418460 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ba1ca519-e6da-43b0-9659-ec51fac539a4-sys\") pod \"tuned-chtf2\" (UID: \"ba1ca519-e6da-43b0-9659-ec51fac539a4\") " pod="openshift-cluster-node-tuning-operator/tuned-chtf2" Jul 23 11:35:35.419790 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.418484 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1e281f98-6114-4c25-9b31-79c65f9100e2-original-pull-secret\") pod \"global-pull-secret-syncer-8df69\" (UID: \"1e281f98-6114-4c25-9b31-79c65f9100e2\") " pod="kube-system/global-pull-secret-syncer-8df69" Jul 23 11:35:35.419790 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.418514 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ba1ca519-e6da-43b0-9659-ec51fac539a4-etc-systemd\") pod \"tuned-chtf2\" (UID: \"ba1ca519-e6da-43b0-9659-ec51fac539a4\") " pod="openshift-cluster-node-tuning-operator/tuned-chtf2" Jul 23 11:35:35.419790 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.418538 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8886ab91-1d26-47c4-835e-a2905af1aa16-metrics-certs\") pod \"network-metrics-daemon-x7q2v\" (UID: \"8886ab91-1d26-47c4-835e-a2905af1aa16\") " pod="openshift-multus/network-metrics-daemon-x7q2v" Jul 23 11:35:35.419790 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.418563 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2c75320d-d67b-4e9e-b648-cc748f5a2870-var-lib-openvswitch\") pod \"ovnkube-node-92n98\" (UID: \"2c75320d-d67b-4e9e-b648-cc748f5a2870\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n98" Jul 23 11:35:35.419790 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.418575 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f898b85f-57f0-4973-9e99-117e9fd60fa8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6jdp4\" (UID: \"f898b85f-57f0-4973-9e99-117e9fd60fa8\") " pod="openshift-multus/multus-additional-cni-plugins-6jdp4" Jul 23 11:35:35.419790 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.418589 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/792c59f0-f23f-4694-9cde-37b9431760f4-host-run-netns\") pod \"multus-8d65t\" (UID: \"792c59f0-f23f-4694-9cde-37b9431760f4\") " pod="openshift-multus/multus-8d65t" Jul 23 11:35:35.419790 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.418628 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ebf70217-31df-4e81-be74-df699db721f0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sf88g\" (UID: \"ebf70217-31df-4e81-be74-df699db721f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sf88g" Jul 23 11:35:35.419790 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.418651 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ba1ca519-e6da-43b0-9659-ec51fac539a4-host\") pod \"tuned-chtf2\" (UID: \"ba1ca519-e6da-43b0-9659-ec51fac539a4\") " pod="openshift-cluster-node-tuning-operator/tuned-chtf2" Jul 23 11:35:35.419790 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.418677 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf7hx\" (UniqueName: \"kubernetes.io/projected/792c59f0-f23f-4694-9cde-37b9431760f4-kube-api-access-pf7hx\") pod \"multus-8d65t\" (UID: \"792c59f0-f23f-4694-9cde-37b9431760f4\") " pod="openshift-multus/multus-8d65t" Jul 23 11:35:35.419790 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:35.418682 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 11:35:35.419790 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.418705 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2c75320d-d67b-4e9e-b648-cc748f5a2870-host-kubelet\") pod \"ovnkube-node-92n98\" (UID: \"2c75320d-d67b-4e9e-b648-cc748f5a2870\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n98" Jul 23 11:35:35.419790 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.418730 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2c75320d-d67b-4e9e-b648-cc748f5a2870-host-run-netns\") pod \"ovnkube-node-92n98\" (UID: \"2c75320d-d67b-4e9e-b648-cc748f5a2870\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n98" Jul 23 11:35:35.420640 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:35.418752 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8886ab91-1d26-47c4-835e-a2905af1aa16-metrics-certs podName:8886ab91-1d26-47c4-835e-a2905af1aa16 nodeName:}" failed. No retries permitted until 2026-07-23 11:35:35.918733216 +0000 UTC m=+2.990423187 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8886ab91-1d26-47c4-835e-a2905af1aa16-metrics-certs") pod "network-metrics-daemon-x7q2v" (UID: "8886ab91-1d26-47c4-835e-a2905af1aa16") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 11:35:35.420640 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.418773 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2c75320d-d67b-4e9e-b648-cc748f5a2870-host-run-netns\") pod \"ovnkube-node-92n98\" (UID: \"2c75320d-d67b-4e9e-b648-cc748f5a2870\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n98" Jul 23 11:35:35.420640 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.418780 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2c75320d-d67b-4e9e-b648-cc748f5a2870-ovnkube-script-lib\") pod \"ovnkube-node-92n98\" (UID: \"2c75320d-d67b-4e9e-b648-cc748f5a2870\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n98" Jul 23 11:35:35.420640 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.418784 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/792c59f0-f23f-4694-9cde-37b9431760f4-cni-binary-copy\") pod \"multus-8d65t\" (UID: \"792c59f0-f23f-4694-9cde-37b9431760f4\") " pod="openshift-multus/multus-8d65t" Jul 23 11:35:35.420640 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.418818 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2c75320d-d67b-4e9e-b648-cc748f5a2870-var-lib-openvswitch\") pod \"ovnkube-node-92n98\" (UID: \"2c75320d-d67b-4e9e-b648-cc748f5a2870\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n98" Jul 23 11:35:35.420640 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.418823 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/792c59f0-f23f-4694-9cde-37b9431760f4-host-run-multus-certs\") pod \"multus-8d65t\" (UID: \"792c59f0-f23f-4694-9cde-37b9431760f4\") " pod="openshift-multus/multus-8d65t" Jul 23 11:35:35.420640 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.418845 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/917d3c0c-f95b-43ac-a39a-b21dfdc0c07c-host-slash\") pod \"iptables-alerter-zfngm\" (UID: \"917d3c0c-f95b-43ac-a39a-b21dfdc0c07c\") " pod="openshift-network-operator/iptables-alerter-zfngm" Jul 23 11:35:35.420640 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.418860 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ebf70217-31df-4e81-be74-df699db721f0-etc-selinux\") pod \"aws-ebs-csi-driver-node-sf88g\" (UID: \"ebf70217-31df-4e81-be74-df699db721f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sf88g" Jul 23 11:35:35.420640 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.418878 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggtpg\" (UniqueName: \"kubernetes.io/projected/f898b85f-57f0-4973-9e99-117e9fd60fa8-kube-api-access-ggtpg\") pod \"multus-additional-cni-plugins-6jdp4\" (UID: \"f898b85f-57f0-4973-9e99-117e9fd60fa8\") " pod="openshift-multus/multus-additional-cni-plugins-6jdp4" Jul 23 11:35:35.420640 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.418927 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2c75320d-d67b-4e9e-b648-cc748f5a2870-log-socket\") pod \"ovnkube-node-92n98\" (UID: \"2c75320d-d67b-4e9e-b648-cc748f5a2870\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n98" Jul 23 11:35:35.420640 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.418943 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2c75320d-d67b-4e9e-b648-cc748f5a2870-env-overrides\") pod \"ovnkube-node-92n98\" (UID: \"2c75320d-d67b-4e9e-b648-cc748f5a2870\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n98" Jul 23 11:35:35.420640 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.418958 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1e281f98-6114-4c25-9b31-79c65f9100e2-dbus\") pod \"global-pull-secret-syncer-8df69\" (UID: \"1e281f98-6114-4c25-9b31-79c65f9100e2\") " pod="kube-system/global-pull-secret-syncer-8df69" Jul 23 11:35:35.420640 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.418985 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/792c59f0-f23f-4694-9cde-37b9431760f4-host-run-netns\") pod \"multus-8d65t\" (UID: \"792c59f0-f23f-4694-9cde-37b9431760f4\") " pod="openshift-multus/multus-8d65t" Jul 23 11:35:35.420640 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.418978 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/792c59f0-f23f-4694-9cde-37b9431760f4-multus-cni-dir\") pod \"multus-8d65t\" (UID: \"792c59f0-f23f-4694-9cde-37b9431760f4\") " pod="openshift-multus/multus-8d65t" Jul 23 11:35:35.420640 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.419035 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2c75320d-d67b-4e9e-b648-cc748f5a2870-log-socket\") pod \"ovnkube-node-92n98\" (UID: \"2c75320d-d67b-4e9e-b648-cc748f5a2870\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n98" Jul 23 11:35:35.420640 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.419042 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/792c59f0-f23f-4694-9cde-37b9431760f4-etc-kubernetes\") pod \"multus-8d65t\" (UID: \"792c59f0-f23f-4694-9cde-37b9431760f4\") " pod="openshift-multus/multus-8d65t" Jul 23 11:35:35.420640 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.419069 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/620c5e1f-97de-42e6-b752-50d951728a5d-agent-certs\") pod \"konnectivity-agent-lp5f4\" (UID: \"620c5e1f-97de-42e6-b752-50d951728a5d\") " pod="kube-system/konnectivity-agent-lp5f4" Jul 23 11:35:35.421521 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.419070 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2c75320d-d67b-4e9e-b648-cc748f5a2870-ovnkube-config\") pod \"ovnkube-node-92n98\" (UID: \"2c75320d-d67b-4e9e-b648-cc748f5a2870\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n98" Jul 23 11:35:35.421521 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.419097 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/aa374cc8-936e-41ca-a905-c0449df6825d-host\") pod \"node-ca-6cbrx\" (UID: \"aa374cc8-936e-41ca-a905-c0449df6825d\") " pod="openshift-image-registry/node-ca-6cbrx" Jul 23 11:35:35.421521 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.419115 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f898b85f-57f0-4973-9e99-117e9fd60fa8-cni-binary-copy\") pod \"multus-additional-cni-plugins-6jdp4\" (UID: \"f898b85f-57f0-4973-9e99-117e9fd60fa8\") " pod="openshift-multus/multus-additional-cni-plugins-6jdp4" Jul 23 11:35:35.421521 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.419142 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/792c59f0-f23f-4694-9cde-37b9431760f4-multus-cni-dir\") pod \"multus-8d65t\" (UID: \"792c59f0-f23f-4694-9cde-37b9431760f4\") " pod="openshift-multus/multus-8d65t" Jul 23 11:35:35.421521 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.419147 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f898b85f-57f0-4973-9e99-117e9fd60fa8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6jdp4\" (UID: \"f898b85f-57f0-4973-9e99-117e9fd60fa8\") " pod="openshift-multus/multus-additional-cni-plugins-6jdp4" Jul 23 11:35:35.421521 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.419177 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1e281f98-6114-4c25-9b31-79c65f9100e2-dbus\") pod \"global-pull-secret-syncer-8df69\" (UID: \"1e281f98-6114-4c25-9b31-79c65f9100e2\") " pod="kube-system/global-pull-secret-syncer-8df69" Jul 23 11:35:35.421521 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.419183 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2c75320d-d67b-4e9e-b648-cc748f5a2870-run-systemd\") pod \"ovnkube-node-92n98\" (UID: \"2c75320d-d67b-4e9e-b648-cc748f5a2870\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n98" Jul 23 11:35:35.421521 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.419260 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/792c59f0-f23f-4694-9cde-37b9431760f4-host-run-k8s-cni-cncf-io\") pod \"multus-8d65t\" (UID: \"792c59f0-f23f-4694-9cde-37b9431760f4\") " pod="openshift-multus/multus-8d65t" Jul 23 11:35:35.421521 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:35.419275 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 11:35:35.421521 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.419282 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/792c59f0-f23f-4694-9cde-37b9431760f4-multus-conf-dir\") pod \"multus-8d65t\" (UID: \"792c59f0-f23f-4694-9cde-37b9431760f4\") " pod="openshift-multus/multus-8d65t" Jul 23 11:35:35.421521 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.419352 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/aa374cc8-936e-41ca-a905-c0449df6825d-serviceca\") pod \"node-ca-6cbrx\" (UID: \"aa374cc8-936e-41ca-a905-c0449df6825d\") " pod="openshift-image-registry/node-ca-6cbrx" Jul 23 11:35:35.421521 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.419417 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba1ca519-e6da-43b0-9659-ec51fac539a4-etc-kubernetes\") pod \"tuned-chtf2\" (UID: \"ba1ca519-e6da-43b0-9659-ec51fac539a4\") " pod="openshift-cluster-node-tuning-operator/tuned-chtf2" Jul 23 11:35:35.421521 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.419441 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ba1ca519-e6da-43b0-9659-ec51fac539a4-etc-tuned\") pod \"tuned-chtf2\" (UID: \"ba1ca519-e6da-43b0-9659-ec51fac539a4\") " pod="openshift-cluster-node-tuning-operator/tuned-chtf2" Jul 23 11:35:35.421521 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.419468 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2c75320d-d67b-4e9e-b648-cc748f5a2870-systemd-units\") pod \"ovnkube-node-92n98\" (UID: \"2c75320d-d67b-4e9e-b648-cc748f5a2870\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n98" Jul 23 11:35:35.421521 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.419477 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f898b85f-57f0-4973-9e99-117e9fd60fa8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6jdp4\" (UID: \"f898b85f-57f0-4973-9e99-117e9fd60fa8\") " pod="openshift-multus/multus-additional-cni-plugins-6jdp4" Jul 23 11:35:35.421521 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:35.419501 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e281f98-6114-4c25-9b31-79c65f9100e2-original-pull-secret podName:1e281f98-6114-4c25-9b31-79c65f9100e2 nodeName:}" failed. No retries permitted until 2026-07-23 11:35:35.91948477 +0000 UTC m=+2.991174739 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1e281f98-6114-4c25-9b31-79c65f9100e2-original-pull-secret") pod "global-pull-secret-syncer-8df69" (UID: "1e281f98-6114-4c25-9b31-79c65f9100e2") : object "kube-system"/"original-pull-secret" not registered Jul 23 11:35:35.421521 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.419512 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2c75320d-d67b-4e9e-b648-cc748f5a2870-systemd-units\") pod \"ovnkube-node-92n98\" (UID: \"2c75320d-d67b-4e9e-b648-cc748f5a2870\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n98" Jul 23 11:35:35.422258 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.419521 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2c75320d-d67b-4e9e-b648-cc748f5a2870-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-92n98\" (UID: \"2c75320d-d67b-4e9e-b648-cc748f5a2870\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n98" Jul 23 11:35:35.422258 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.419561 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/792c59f0-f23f-4694-9cde-37b9431760f4-host-run-multus-certs\") pod \"multus-8d65t\" (UID: \"792c59f0-f23f-4694-9cde-37b9431760f4\") " pod="openshift-multus/multus-8d65t" Jul 23 11:35:35.422258 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.419589 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2c75320d-d67b-4e9e-b648-cc748f5a2870-host-kubelet\") pod \"ovnkube-node-92n98\" (UID: \"2c75320d-d67b-4e9e-b648-cc748f5a2870\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n98" Jul 23 11:35:35.422258 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.419641 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/792c59f0-f23f-4694-9cde-37b9431760f4-etc-kubernetes\") pod \"multus-8d65t\" (UID: \"792c59f0-f23f-4694-9cde-37b9431760f4\") " pod="openshift-multus/multus-8d65t" Jul 23 11:35:35.422258 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.419679 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/917d3c0c-f95b-43ac-a39a-b21dfdc0c07c-iptables-alerter-script\") pod \"iptables-alerter-zfngm\" (UID: \"917d3c0c-f95b-43ac-a39a-b21dfdc0c07c\") " pod="openshift-network-operator/iptables-alerter-zfngm" Jul 23 11:35:35.422258 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.419685 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2c75320d-d67b-4e9e-b648-cc748f5a2870-ovnkube-script-lib\") pod \"ovnkube-node-92n98\" (UID: \"2c75320d-d67b-4e9e-b648-cc748f5a2870\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n98" Jul 23 11:35:35.422258 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.419702 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcr9p\" (UniqueName: \"kubernetes.io/projected/ebf70217-31df-4e81-be74-df699db721f0-kube-api-access-vcr9p\") pod \"aws-ebs-csi-driver-node-sf88g\" (UID: \"ebf70217-31df-4e81-be74-df699db721f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sf88g" Jul 23 11:35:35.422258 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.419732 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2c75320d-d67b-4e9e-b648-cc748f5a2870-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-92n98\" (UID: \"2c75320d-d67b-4e9e-b648-cc748f5a2870\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n98" Jul 23 11:35:35.422258 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.419739 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ba1ca519-e6da-43b0-9659-ec51fac539a4-etc-sysconfig\") pod \"tuned-chtf2\" (UID: \"ba1ca519-e6da-43b0-9659-ec51fac539a4\") " pod="openshift-cluster-node-tuning-operator/tuned-chtf2" Jul 23 11:35:35.422258 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.419742 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2c75320d-d67b-4e9e-b648-cc748f5a2870-run-systemd\") pod \"ovnkube-node-92n98\" (UID: \"2c75320d-d67b-4e9e-b648-cc748f5a2870\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n98" Jul 23 11:35:35.422258 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.419748 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2c75320d-d67b-4e9e-b648-cc748f5a2870-env-overrides\") pod \"ovnkube-node-92n98\" (UID: \"2c75320d-d67b-4e9e-b648-cc748f5a2870\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n98" Jul 23 11:35:35.422258 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.419760 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ba1ca519-e6da-43b0-9659-ec51fac539a4-etc-sysctl-conf\") pod \"tuned-chtf2\" (UID: \"ba1ca519-e6da-43b0-9659-ec51fac539a4\") " pod="openshift-cluster-node-tuning-operator/tuned-chtf2" Jul 23 11:35:35.422258 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.419769 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/792c59f0-f23f-4694-9cde-37b9431760f4-host-run-k8s-cni-cncf-io\") pod \"multus-8d65t\" (UID: \"792c59f0-f23f-4694-9cde-37b9431760f4\") " pod="openshift-multus/multus-8d65t" Jul 23 11:35:35.422258 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.419798 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/792c59f0-f23f-4694-9cde-37b9431760f4-multus-conf-dir\") pod \"multus-8d65t\" (UID: \"792c59f0-f23f-4694-9cde-37b9431760f4\") " pod="openshift-multus/multus-8d65t" Jul 23 11:35:35.422258 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.419800 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ba1ca519-e6da-43b0-9659-ec51fac539a4-lib-modules\") pod \"tuned-chtf2\" (UID: \"ba1ca519-e6da-43b0-9659-ec51fac539a4\") " pod="openshift-cluster-node-tuning-operator/tuned-chtf2" Jul 23 11:35:35.422258 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.419826 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlt4h\" (UniqueName: \"kubernetes.io/projected/a17ee9c0-c387-4e9d-a6a4-481501f5b97b-kube-api-access-mlt4h\") pod \"node-resolver-pkwm5\" (UID: \"a17ee9c0-c387-4e9d-a6a4-481501f5b97b\") " pod="openshift-dns/node-resolver-pkwm5" Jul 23 11:35:35.422258 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.420219 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f898b85f-57f0-4973-9e99-117e9fd60fa8-cni-binary-copy\") pod \"multus-additional-cni-plugins-6jdp4\" (UID: \"f898b85f-57f0-4973-9e99-117e9fd60fa8\") " pod="openshift-multus/multus-additional-cni-plugins-6jdp4" Jul 23 11:35:35.423126 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.422112 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2c75320d-d67b-4e9e-b648-cc748f5a2870-ovn-node-metrics-cert\") pod \"ovnkube-node-92n98\" (UID: \"2c75320d-d67b-4e9e-b648-cc748f5a2870\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n98" Jul 23 11:35:35.427377 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:35.427351 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 11:35:35.427377 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:35.427379 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 11:35:35.427541 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:35.427389 2561 projected.go:196] Error preparing data for projected volume kube-api-access-gnlxz for pod openshift-network-diagnostics/network-check-target-fv772: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 11:35:35.427541 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:35.427437 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c92ad538-9642-4c08-b92c-886738d3c413-kube-api-access-gnlxz podName:c92ad538-9642-4c08-b92c-886738d3c413 nodeName:}" failed. No retries permitted until 2026-07-23 11:35:35.927423728 +0000 UTC m=+2.999113696 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-gnlxz" (UniqueName: "kubernetes.io/projected/c92ad538-9642-4c08-b92c-886738d3c413-kube-api-access-gnlxz") pod "network-check-target-fv772" (UID: "c92ad538-9642-4c08-b92c-886738d3c413") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 11:35:35.430601 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.430480 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mghl5\" (UniqueName: \"kubernetes.io/projected/8886ab91-1d26-47c4-835e-a2905af1aa16-kube-api-access-mghl5\") pod \"network-metrics-daemon-x7q2v\" (UID: \"8886ab91-1d26-47c4-835e-a2905af1aa16\") " pod="openshift-multus/network-metrics-daemon-x7q2v" Jul 23 11:35:35.431343 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.431317 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nphvk\" (UniqueName: \"kubernetes.io/projected/2c75320d-d67b-4e9e-b648-cc748f5a2870-kube-api-access-nphvk\") pod \"ovnkube-node-92n98\" (UID: \"2c75320d-d67b-4e9e-b648-cc748f5a2870\") " pod="openshift-ovn-kubernetes/ovnkube-node-92n98" Jul 23 11:35:35.431442 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.431409 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf7hx\" (UniqueName: \"kubernetes.io/projected/792c59f0-f23f-4694-9cde-37b9431760f4-kube-api-access-pf7hx\") pod \"multus-8d65t\" (UID: \"792c59f0-f23f-4694-9cde-37b9431760f4\") " pod="openshift-multus/multus-8d65t" Jul 23 11:35:35.431824 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.431796 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggtpg\" (UniqueName: \"kubernetes.io/projected/f898b85f-57f0-4973-9e99-117e9fd60fa8-kube-api-access-ggtpg\") pod \"multus-additional-cni-plugins-6jdp4\" (UID: \"f898b85f-57f0-4973-9e99-117e9fd60fa8\") " pod="openshift-multus/multus-additional-cni-plugins-6jdp4" Jul 23 11:35:35.455056 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.455002 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-236.ec2.internal" event={"ID":"b1b25b788fe3c6b4ce0ebd3a8ca6fd57","Type":"ContainerStarted","Data":"cd0150862617567211f5a2c8df8c102c194718a5a0ddf14b9bb36b064c77911a"} Jul 23 11:35:35.456290 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.456258 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-236.ec2.internal" event={"ID":"f6f6dd6dc338193833341a6aab50de06","Type":"ContainerStarted","Data":"a90e0389ca453c04fb0a1c658bc6ecfb1a9e98560a7746504373f2ef926ce52f"} Jul 23 11:35:35.520403 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.520369 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ba1ca519-e6da-43b0-9659-ec51fac539a4-etc-systemd\") pod \"tuned-chtf2\" (UID: \"ba1ca519-e6da-43b0-9659-ec51fac539a4\") " pod="openshift-cluster-node-tuning-operator/tuned-chtf2" Jul 23 11:35:35.520579 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.520418 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ebf70217-31df-4e81-be74-df699db721f0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sf88g\" (UID: \"ebf70217-31df-4e81-be74-df699db721f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sf88g" Jul 23 11:35:35.520579 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.520439 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ba1ca519-e6da-43b0-9659-ec51fac539a4-host\") pod \"tuned-chtf2\" (UID: \"ba1ca519-e6da-43b0-9659-ec51fac539a4\") " pod="openshift-cluster-node-tuning-operator/tuned-chtf2" Jul 23 11:35:35.520579 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.520474 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/917d3c0c-f95b-43ac-a39a-b21dfdc0c07c-host-slash\") pod \"iptables-alerter-zfngm\" (UID: \"917d3c0c-f95b-43ac-a39a-b21dfdc0c07c\") " pod="openshift-network-operator/iptables-alerter-zfngm" Jul 23 11:35:35.520579 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.520494 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ebf70217-31df-4e81-be74-df699db721f0-etc-selinux\") pod \"aws-ebs-csi-driver-node-sf88g\" (UID: \"ebf70217-31df-4e81-be74-df699db721f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sf88g" Jul 23 11:35:35.520579 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.520522 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ba1ca519-e6da-43b0-9659-ec51fac539a4-etc-systemd\") pod \"tuned-chtf2\" (UID: \"ba1ca519-e6da-43b0-9659-ec51fac539a4\") " pod="openshift-cluster-node-tuning-operator/tuned-chtf2" Jul 23 11:35:35.520579 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.520573 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/620c5e1f-97de-42e6-b752-50d951728a5d-agent-certs\") pod \"konnectivity-agent-lp5f4\" (UID: \"620c5e1f-97de-42e6-b752-50d951728a5d\") " pod="kube-system/konnectivity-agent-lp5f4" Jul 23 11:35:35.520869 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.520596 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/aa374cc8-936e-41ca-a905-c0449df6825d-host\") pod \"node-ca-6cbrx\" (UID: \"aa374cc8-936e-41ca-a905-c0449df6825d\") " pod="openshift-image-registry/node-ca-6cbrx" Jul 23 11:35:35.520869 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.520618 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ebf70217-31df-4e81-be74-df699db721f0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sf88g\" (UID: \"ebf70217-31df-4e81-be74-df699db721f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sf88g" Jul 23 11:35:35.520869 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.520621 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/aa374cc8-936e-41ca-a905-c0449df6825d-serviceca\") pod \"node-ca-6cbrx\" (UID: \"aa374cc8-936e-41ca-a905-c0449df6825d\") " pod="openshift-image-registry/node-ca-6cbrx" Jul 23 11:35:35.520869 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.520678 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba1ca519-e6da-43b0-9659-ec51fac539a4-etc-kubernetes\") pod \"tuned-chtf2\" (UID: \"ba1ca519-e6da-43b0-9659-ec51fac539a4\") " pod="openshift-cluster-node-tuning-operator/tuned-chtf2" Jul 23 11:35:35.520869 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.520699 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ba1ca519-e6da-43b0-9659-ec51fac539a4-etc-tuned\") pod \"tuned-chtf2\" (UID: \"ba1ca519-e6da-43b0-9659-ec51fac539a4\") " pod="openshift-cluster-node-tuning-operator/tuned-chtf2" Jul 23 11:35:35.520869 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.520725 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/917d3c0c-f95b-43ac-a39a-b21dfdc0c07c-iptables-alerter-script\") pod \"iptables-alerter-zfngm\" (UID: \"917d3c0c-f95b-43ac-a39a-b21dfdc0c07c\") " pod="openshift-network-operator/iptables-alerter-zfngm" Jul 23 11:35:35.520869 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.520762 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcr9p\" (UniqueName: \"kubernetes.io/projected/ebf70217-31df-4e81-be74-df699db721f0-kube-api-access-vcr9p\") pod \"aws-ebs-csi-driver-node-sf88g\" (UID: \"ebf70217-31df-4e81-be74-df699db721f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sf88g" Jul 23 11:35:35.520869 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.520784 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ba1ca519-e6da-43b0-9659-ec51fac539a4-etc-sysconfig\") pod \"tuned-chtf2\" (UID: \"ba1ca519-e6da-43b0-9659-ec51fac539a4\") " pod="openshift-cluster-node-tuning-operator/tuned-chtf2" Jul 23 11:35:35.520869 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.520806 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ba1ca519-e6da-43b0-9659-ec51fac539a4-etc-sysctl-conf\") pod \"tuned-chtf2\" (UID: \"ba1ca519-e6da-43b0-9659-ec51fac539a4\") " pod="openshift-cluster-node-tuning-operator/tuned-chtf2" Jul 23 11:35:35.520869 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.520828 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ba1ca519-e6da-43b0-9659-ec51fac539a4-lib-modules\") pod \"tuned-chtf2\" (UID: \"ba1ca519-e6da-43b0-9659-ec51fac539a4\") " pod="openshift-cluster-node-tuning-operator/tuned-chtf2" Jul 23 11:35:35.520869 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.520868 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlt4h\" (UniqueName: \"kubernetes.io/projected/a17ee9c0-c387-4e9d-a6a4-481501f5b97b-kube-api-access-mlt4h\") pod \"node-resolver-pkwm5\" (UID: \"a17ee9c0-c387-4e9d-a6a4-481501f5b97b\") " pod="openshift-dns/node-resolver-pkwm5" Jul 23 11:35:35.521416 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.520896 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn7s2\" (UniqueName: \"kubernetes.io/projected/917d3c0c-f95b-43ac-a39a-b21dfdc0c07c-kube-api-access-xn7s2\") pod \"iptables-alerter-zfngm\" (UID: \"917d3c0c-f95b-43ac-a39a-b21dfdc0c07c\") " pod="openshift-network-operator/iptables-alerter-zfngm" Jul 23 11:35:35.521416 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.520951 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ebf70217-31df-4e81-be74-df699db721f0-registration-dir\") pod \"aws-ebs-csi-driver-node-sf88g\" (UID: \"ebf70217-31df-4e81-be74-df699db721f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sf88g" Jul 23 11:35:35.521416 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.520985 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ba1ca519-e6da-43b0-9659-ec51fac539a4-tmp\") pod \"tuned-chtf2\" (UID: \"ba1ca519-e6da-43b0-9659-ec51fac539a4\") " pod="openshift-cluster-node-tuning-operator/tuned-chtf2" Jul 23 11:35:35.521416 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.521018 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/620c5e1f-97de-42e6-b752-50d951728a5d-konnectivity-ca\") pod \"konnectivity-agent-lp5f4\" (UID: \"620c5e1f-97de-42e6-b752-50d951728a5d\") " pod="kube-system/konnectivity-agent-lp5f4" Jul 23 11:35:35.521416 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.521040 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ebf70217-31df-4e81-be74-df699db721f0-device-dir\") pod \"aws-ebs-csi-driver-node-sf88g\" (UID: \"ebf70217-31df-4e81-be74-df699db721f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sf88g" Jul 23 11:35:35.521416 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.521144 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ba1ca519-e6da-43b0-9659-ec51fac539a4-etc-sysconfig\") pod \"tuned-chtf2\" (UID: \"ba1ca519-e6da-43b0-9659-ec51fac539a4\") " pod="openshift-cluster-node-tuning-operator/tuned-chtf2" Jul 23 11:35:35.521416 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.521171 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ebf70217-31df-4e81-be74-df699db721f0-socket-dir\") pod \"aws-ebs-csi-driver-node-sf88g\" (UID: \"ebf70217-31df-4e81-be74-df699db721f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sf88g" Jul 23 11:35:35.521416 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.521207 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a17ee9c0-c387-4e9d-a6a4-481501f5b97b-hosts-file\") pod \"node-resolver-pkwm5\" (UID: \"a17ee9c0-c387-4e9d-a6a4-481501f5b97b\") " pod="openshift-dns/node-resolver-pkwm5" Jul 23 11:35:35.521416 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.521217 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ba1ca519-e6da-43b0-9659-ec51fac539a4-lib-modules\") pod \"tuned-chtf2\" (UID: \"ba1ca519-e6da-43b0-9659-ec51fac539a4\") " pod="openshift-cluster-node-tuning-operator/tuned-chtf2" Jul 23 11:35:35.521416 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.521269 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ba1ca519-e6da-43b0-9659-ec51fac539a4-etc-sysctl-conf\") pod \"tuned-chtf2\" (UID: \"ba1ca519-e6da-43b0-9659-ec51fac539a4\") " pod="openshift-cluster-node-tuning-operator/tuned-chtf2" Jul 23 11:35:35.521416 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.521253 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a17ee9c0-c387-4e9d-a6a4-481501f5b97b-tmp-dir\") pod \"node-resolver-pkwm5\" (UID: \"a17ee9c0-c387-4e9d-a6a4-481501f5b97b\") " pod="openshift-dns/node-resolver-pkwm5" Jul 23 11:35:35.521416 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.521314 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ebf70217-31df-4e81-be74-df699db721f0-sys-fs\") pod \"aws-ebs-csi-driver-node-sf88g\" (UID: \"ebf70217-31df-4e81-be74-df699db721f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sf88g" Jul 23 11:35:35.521416 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.521336 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba1ca519-e6da-43b0-9659-ec51fac539a4-var-lib-kubelet\") pod \"tuned-chtf2\" (UID: \"ba1ca519-e6da-43b0-9659-ec51fac539a4\") " pod="openshift-cluster-node-tuning-operator/tuned-chtf2" Jul 23 11:35:35.521416 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.521388 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ebf70217-31df-4e81-be74-df699db721f0-registration-dir\") pod \"aws-ebs-csi-driver-node-sf88g\" (UID: \"ebf70217-31df-4e81-be74-df699db721f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sf88g" Jul 23 11:35:35.521416 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.521401 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wp8xl\" (UniqueName: \"kubernetes.io/projected/aa374cc8-936e-41ca-a905-c0449df6825d-kube-api-access-wp8xl\") pod \"node-ca-6cbrx\" (UID: \"aa374cc8-936e-41ca-a905-c0449df6825d\") " pod="openshift-image-registry/node-ca-6cbrx" Jul 23 11:35:35.522106 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.521438 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ba1ca519-e6da-43b0-9659-ec51fac539a4-etc-modprobe-d\") pod \"tuned-chtf2\" (UID: \"ba1ca519-e6da-43b0-9659-ec51fac539a4\") " pod="openshift-cluster-node-tuning-operator/tuned-chtf2" Jul 23 11:35:35.522106 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.521471 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ba1ca519-e6da-43b0-9659-ec51fac539a4-etc-sysctl-d\") pod \"tuned-chtf2\" (UID: \"ba1ca519-e6da-43b0-9659-ec51fac539a4\") " pod="openshift-cluster-node-tuning-operator/tuned-chtf2" Jul 23 11:35:35.522106 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.521523 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ba1ca519-e6da-43b0-9659-ec51fac539a4-run\") pod \"tuned-chtf2\" (UID: \"ba1ca519-e6da-43b0-9659-ec51fac539a4\") " pod="openshift-cluster-node-tuning-operator/tuned-chtf2" Jul 23 11:35:35.522106 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.521555 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7776\" (UniqueName: \"kubernetes.io/projected/ba1ca519-e6da-43b0-9659-ec51fac539a4-kube-api-access-c7776\") pod \"tuned-chtf2\" (UID: \"ba1ca519-e6da-43b0-9659-ec51fac539a4\") " pod="openshift-cluster-node-tuning-operator/tuned-chtf2" Jul 23 11:35:35.522106 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.521556 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/917d3c0c-f95b-43ac-a39a-b21dfdc0c07c-iptables-alerter-script\") pod \"iptables-alerter-zfngm\" (UID: \"917d3c0c-f95b-43ac-a39a-b21dfdc0c07c\") " pod="openshift-network-operator/iptables-alerter-zfngm" Jul 23 11:35:35.522106 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.521579 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ba1ca519-e6da-43b0-9659-ec51fac539a4-sys\") pod \"tuned-chtf2\" (UID: \"ba1ca519-e6da-43b0-9659-ec51fac539a4\") " pod="openshift-cluster-node-tuning-operator/tuned-chtf2" Jul 23 11:35:35.522106 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.521621 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ba1ca519-e6da-43b0-9659-ec51fac539a4-host\") pod \"tuned-chtf2\" (UID: \"ba1ca519-e6da-43b0-9659-ec51fac539a4\") " pod="openshift-cluster-node-tuning-operator/tuned-chtf2" Jul 23 11:35:35.522393 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.522319 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a17ee9c0-c387-4e9d-a6a4-481501f5b97b-tmp-dir\") pod \"node-resolver-pkwm5\" (UID: \"a17ee9c0-c387-4e9d-a6a4-481501f5b97b\") " pod="openshift-dns/node-resolver-pkwm5" Jul 23 11:35:35.522393 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.522384 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/620c5e1f-97de-42e6-b752-50d951728a5d-konnectivity-ca\") pod \"konnectivity-agent-lp5f4\" (UID: \"620c5e1f-97de-42e6-b752-50d951728a5d\") " pod="kube-system/konnectivity-agent-lp5f4" Jul 23 11:35:35.523163 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.523139 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ba1ca519-e6da-43b0-9659-ec51fac539a4-sys\") pod \"tuned-chtf2\" (UID: \"ba1ca519-e6da-43b0-9659-ec51fac539a4\") " pod="openshift-cluster-node-tuning-operator/tuned-chtf2" Jul 23 11:35:35.523261 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.523194 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/917d3c0c-f95b-43ac-a39a-b21dfdc0c07c-host-slash\") pod \"iptables-alerter-zfngm\" (UID: \"917d3c0c-f95b-43ac-a39a-b21dfdc0c07c\") " pod="openshift-network-operator/iptables-alerter-zfngm" Jul 23 11:35:35.524099 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.524076 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ebf70217-31df-4e81-be74-df699db721f0-sys-fs\") pod \"aws-ebs-csi-driver-node-sf88g\" (UID: \"ebf70217-31df-4e81-be74-df699db721f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sf88g" Jul 23 11:35:35.524199 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.524183 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ebf70217-31df-4e81-be74-df699db721f0-etc-selinux\") pod \"aws-ebs-csi-driver-node-sf88g\" (UID: \"ebf70217-31df-4e81-be74-df699db721f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sf88g" Jul 23 11:35:35.524258 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.524230 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba1ca519-e6da-43b0-9659-ec51fac539a4-etc-kubernetes\") pod \"tuned-chtf2\" (UID: \"ba1ca519-e6da-43b0-9659-ec51fac539a4\") " pod="openshift-cluster-node-tuning-operator/tuned-chtf2" Jul 23 11:35:35.524309 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.524284 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/aa374cc8-936e-41ca-a905-c0449df6825d-host\") pod \"node-ca-6cbrx\" (UID: \"aa374cc8-936e-41ca-a905-c0449df6825d\") " pod="openshift-image-registry/node-ca-6cbrx" Jul 23 11:35:35.524355 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.524319 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/620c5e1f-97de-42e6-b752-50d951728a5d-agent-certs\") pod \"konnectivity-agent-lp5f4\" (UID: \"620c5e1f-97de-42e6-b752-50d951728a5d\") " pod="kube-system/konnectivity-agent-lp5f4" Jul 23 11:35:35.524410 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.524392 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a17ee9c0-c387-4e9d-a6a4-481501f5b97b-hosts-file\") pod \"node-resolver-pkwm5\" (UID: \"a17ee9c0-c387-4e9d-a6a4-481501f5b97b\") " pod="openshift-dns/node-resolver-pkwm5" Jul 23 11:35:35.524462 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.524443 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba1ca519-e6da-43b0-9659-ec51fac539a4-var-lib-kubelet\") pod \"tuned-chtf2\" (UID: \"ba1ca519-e6da-43b0-9659-ec51fac539a4\") " pod="openshift-cluster-node-tuning-operator/tuned-chtf2" Jul 23 11:35:35.524570 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.524557 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ba1ca519-e6da-43b0-9659-ec51fac539a4-etc-sysctl-d\") pod \"tuned-chtf2\" (UID: \"ba1ca519-e6da-43b0-9659-ec51fac539a4\") " pod="openshift-cluster-node-tuning-operator/tuned-chtf2" Jul 23 11:35:35.524655 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.524626 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ba1ca519-e6da-43b0-9659-ec51fac539a4-etc-modprobe-d\") pod \"tuned-chtf2\" (UID: \"ba1ca519-e6da-43b0-9659-ec51fac539a4\") " pod="openshift-cluster-node-tuning-operator/tuned-chtf2" Jul 23 11:35:35.524704 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.524658 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/aa374cc8-936e-41ca-a905-c0449df6825d-serviceca\") pod \"node-ca-6cbrx\" (UID: \"aa374cc8-936e-41ca-a905-c0449df6825d\") " pod="openshift-image-registry/node-ca-6cbrx" Jul 23 11:35:35.524751 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.524718 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ebf70217-31df-4e81-be74-df699db721f0-device-dir\") pod \"aws-ebs-csi-driver-node-sf88g\" (UID: \"ebf70217-31df-4e81-be74-df699db721f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sf88g" Jul 23 11:35:35.524840 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.524822 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ebf70217-31df-4e81-be74-df699db721f0-socket-dir\") pod \"aws-ebs-csi-driver-node-sf88g\" (UID: \"ebf70217-31df-4e81-be74-df699db721f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sf88g" Jul 23 11:35:35.524895 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.524881 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ba1ca519-e6da-43b0-9659-ec51fac539a4-run\") pod \"tuned-chtf2\" (UID: \"ba1ca519-e6da-43b0-9659-ec51fac539a4\") " pod="openshift-cluster-node-tuning-operator/tuned-chtf2" Jul 23 11:35:35.526639 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.526355 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ba1ca519-e6da-43b0-9659-ec51fac539a4-tmp\") pod \"tuned-chtf2\" (UID: \"ba1ca519-e6da-43b0-9659-ec51fac539a4\") " pod="openshift-cluster-node-tuning-operator/tuned-chtf2" Jul 23 11:35:35.528346 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.528327 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ba1ca519-e6da-43b0-9659-ec51fac539a4-etc-tuned\") pod \"tuned-chtf2\" (UID: \"ba1ca519-e6da-43b0-9659-ec51fac539a4\") " pod="openshift-cluster-node-tuning-operator/tuned-chtf2" Jul 23 11:35:35.532261 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.532218 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcr9p\" (UniqueName: \"kubernetes.io/projected/ebf70217-31df-4e81-be74-df699db721f0-kube-api-access-vcr9p\") pod \"aws-ebs-csi-driver-node-sf88g\" (UID: \"ebf70217-31df-4e81-be74-df699db721f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sf88g" Jul 23 11:35:35.532536 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.532515 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlt4h\" (UniqueName: \"kubernetes.io/projected/a17ee9c0-c387-4e9d-a6a4-481501f5b97b-kube-api-access-mlt4h\") pod \"node-resolver-pkwm5\" (UID: \"a17ee9c0-c387-4e9d-a6a4-481501f5b97b\") " pod="openshift-dns/node-resolver-pkwm5" Jul 23 11:35:35.533016 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.533001 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn7s2\" (UniqueName: \"kubernetes.io/projected/917d3c0c-f95b-43ac-a39a-b21dfdc0c07c-kube-api-access-xn7s2\") pod \"iptables-alerter-zfngm\" (UID: \"917d3c0c-f95b-43ac-a39a-b21dfdc0c07c\") " pod="openshift-network-operator/iptables-alerter-zfngm" Jul 23 11:35:35.534538 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.534514 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wp8xl\" (UniqueName: \"kubernetes.io/projected/aa374cc8-936e-41ca-a905-c0449df6825d-kube-api-access-wp8xl\") pod \"node-ca-6cbrx\" (UID: \"aa374cc8-936e-41ca-a905-c0449df6825d\") " pod="openshift-image-registry/node-ca-6cbrx" Jul 23 11:35:35.535975 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.535958 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7776\" (UniqueName: \"kubernetes.io/projected/ba1ca519-e6da-43b0-9659-ec51fac539a4-kube-api-access-c7776\") pod \"tuned-chtf2\" (UID: \"ba1ca519-e6da-43b0-9659-ec51fac539a4\") " pod="openshift-cluster-node-tuning-operator/tuned-chtf2" Jul 23 11:35:35.617611 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.617539 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8d65t" Jul 23 11:35:35.625148 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.625126 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6jdp4" Jul 23 11:35:35.636204 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.636184 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-92n98" Jul 23 11:35:35.641751 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.641732 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-chtf2" Jul 23 11:35:35.649471 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.649453 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lp5f4" Jul 23 11:35:35.658121 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.658102 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zfngm" Jul 23 11:35:35.665782 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.665764 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-pkwm5" Jul 23 11:35:35.673497 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.673478 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6cbrx" Jul 23 11:35:35.679217 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.679202 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sf88g" Jul 23 11:35:35.709610 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.709589 2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 11:35:35.926367 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.926283 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1e281f98-6114-4c25-9b31-79c65f9100e2-original-pull-secret\") pod \"global-pull-secret-syncer-8df69\" (UID: \"1e281f98-6114-4c25-9b31-79c65f9100e2\") " pod="kube-system/global-pull-secret-syncer-8df69" Jul 23 11:35:35.926367 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:35.926325 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8886ab91-1d26-47c4-835e-a2905af1aa16-metrics-certs\") pod \"network-metrics-daemon-x7q2v\" (UID: \"8886ab91-1d26-47c4-835e-a2905af1aa16\") " pod="openshift-multus/network-metrics-daemon-x7q2v" Jul 23 11:35:35.926573 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:35.926443 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 11:35:35.926573 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:35.926509 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e281f98-6114-4c25-9b31-79c65f9100e2-original-pull-secret podName:1e281f98-6114-4c25-9b31-79c65f9100e2 nodeName:}" failed. No retries permitted until 2026-07-23 11:35:36.926486343 +0000 UTC m=+3.998176311 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1e281f98-6114-4c25-9b31-79c65f9100e2-original-pull-secret") pod "global-pull-secret-syncer-8df69" (UID: "1e281f98-6114-4c25-9b31-79c65f9100e2") : object "kube-system"/"original-pull-secret" not registered Jul 23 11:35:35.926573 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:35.926443 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 11:35:35.926573 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:35.926562 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8886ab91-1d26-47c4-835e-a2905af1aa16-metrics-certs podName:8886ab91-1d26-47c4-835e-a2905af1aa16 nodeName:}" failed. No retries permitted until 2026-07-23 11:35:36.926550265 +0000 UTC m=+3.998240239 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8886ab91-1d26-47c4-835e-a2905af1aa16-metrics-certs") pod "network-metrics-daemon-x7q2v" (UID: "8886ab91-1d26-47c4-835e-a2905af1aa16") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 11:35:36.026649 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:36.026612 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnlxz\" (UniqueName: \"kubernetes.io/projected/c92ad538-9642-4c08-b92c-886738d3c413-kube-api-access-gnlxz\") pod \"network-check-target-fv772\" (UID: \"c92ad538-9642-4c08-b92c-886738d3c413\") " pod="openshift-network-diagnostics/network-check-target-fv772" Jul 23 11:35:36.026797 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:36.026733 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 11:35:36.026797 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:36.026747 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 11:35:36.026797 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:36.026754 2561 projected.go:196] Error preparing data for projected volume kube-api-access-gnlxz for pod openshift-network-diagnostics/network-check-target-fv772: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 11:35:36.026885 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:36.026801 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c92ad538-9642-4c08-b92c-886738d3c413-kube-api-access-gnlxz podName:c92ad538-9642-4c08-b92c-886738d3c413 nodeName:}" failed. No retries permitted until 2026-07-23 11:35:37.02678917 +0000 UTC m=+4.098479138 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-gnlxz" (UniqueName: "kubernetes.io/projected/c92ad538-9642-4c08-b92c-886738d3c413-kube-api-access-gnlxz") pod "network-check-target-fv772" (UID: "c92ad538-9642-4c08-b92c-886738d3c413") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 11:35:36.365505 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:36.365480 2561 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 11:35:36.366267 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:36.366244 2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-22 11:30:35 +0000 UTC" deadline="2028-03-24 10:05:00.801510017 +0000 UTC" Jul 23 11:35:36.366267 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:36.366267 2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14638h29m24.435245431s" Jul 23 11:35:36.404767 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:36.404720 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod620c5e1f_97de_42e6_b752_50d951728a5d.slice/crio-7b3930a80afcd1eedb79fceb71c3d0fa567cd69062914e2909fdf4211bc33475 WatchSource:0}: Error finding container 7b3930a80afcd1eedb79fceb71c3d0fa567cd69062914e2909fdf4211bc33475: Status 404 returned error can't find the container with id 7b3930a80afcd1eedb79fceb71c3d0fa567cd69062914e2909fdf4211bc33475 Jul 23 11:35:36.414178 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:36.413871 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf898b85f_57f0_4973_9e99_117e9fd60fa8.slice/crio-9f78d9a560911ada3129e255435204b5682bb4a5f0db0ba42308d010ad282fec WatchSource:0}: Error finding container 9f78d9a560911ada3129e255435204b5682bb4a5f0db0ba42308d010ad282fec: Status 404 returned error can't find the container with id 9f78d9a560911ada3129e255435204b5682bb4a5f0db0ba42308d010ad282fec Jul 23 11:35:36.415812 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:36.415760 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c75320d_d67b_4e9e_b648_cc748f5a2870.slice/crio-3a86fa0295277a725f3b3e83d50b1a64fed04cc56d40de45da9dc2a2b2bcb3c0 WatchSource:0}: Error finding container 3a86fa0295277a725f3b3e83d50b1a64fed04cc56d40de45da9dc2a2b2bcb3c0: Status 404 returned error can't find the container with id 3a86fa0295277a725f3b3e83d50b1a64fed04cc56d40de45da9dc2a2b2bcb3c0 Jul 23 11:35:36.421234 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:35:36.421181 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod792c59f0_f23f_4694_9cde_37b9431760f4.slice/crio-7347784b55a1f6e8699e50c9082029def9ffda61541291d981e5f353657270f5 WatchSource:0}: Error finding container 7347784b55a1f6e8699e50c9082029def9ffda61541291d981e5f353657270f5: Status 404 returned error can't find the container with id 7347784b55a1f6e8699e50c9082029def9ffda61541291d981e5f353657270f5 Jul 23 11:35:36.459186 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:36.458984 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-chtf2" event={"ID":"ba1ca519-e6da-43b0-9659-ec51fac539a4","Type":"ContainerStarted","Data":"bb74d351e9fadf22e5e819783c28c6af3117c79dfdd51d4a933a210ed5ee6365"} Jul 23 11:35:36.460068 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:36.460046 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-pkwm5" event={"ID":"a17ee9c0-c387-4e9d-a6a4-481501f5b97b","Type":"ContainerStarted","Data":"d61b0adf1a6d6f033682022d390631279c005bca814df5b4b2f1097e87080fbb"} Jul 23 11:35:36.461467 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:36.461441 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-92n98" event={"ID":"2c75320d-d67b-4e9e-b648-cc748f5a2870","Type":"ContainerStarted","Data":"3a86fa0295277a725f3b3e83d50b1a64fed04cc56d40de45da9dc2a2b2bcb3c0"} Jul 23 11:35:36.465855 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:36.465835 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6jdp4" event={"ID":"f898b85f-57f0-4973-9e99-117e9fd60fa8","Type":"ContainerStarted","Data":"9f78d9a560911ada3129e255435204b5682bb4a5f0db0ba42308d010ad282fec"} Jul 23 11:35:36.468666 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:36.468618 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sf88g" event={"ID":"ebf70217-31df-4e81-be74-df699db721f0","Type":"ContainerStarted","Data":"de8fded55756349213ae47e077d3206b73cd1eb0f28371b349605a6cd7a91dd3"} Jul 23 11:35:36.470801 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:36.470784 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zfngm" event={"ID":"917d3c0c-f95b-43ac-a39a-b21dfdc0c07c","Type":"ContainerStarted","Data":"e9e0f5ab35a674fa88d6de7fc052ccccce513ce5e2d3e560dfa90389a0e2ca8f"} Jul 23 11:35:36.471796 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:36.471756 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lp5f4" event={"ID":"620c5e1f-97de-42e6-b752-50d951728a5d","Type":"ContainerStarted","Data":"7b3930a80afcd1eedb79fceb71c3d0fa567cd69062914e2909fdf4211bc33475"} Jul 23 11:35:36.472731 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:36.472706 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8d65t" event={"ID":"792c59f0-f23f-4694-9cde-37b9431760f4","Type":"ContainerStarted","Data":"7347784b55a1f6e8699e50c9082029def9ffda61541291d981e5f353657270f5"} Jul 23 11:35:36.473708 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:36.473666 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6cbrx" event={"ID":"aa374cc8-936e-41ca-a905-c0449df6825d","Type":"ContainerStarted","Data":"520e7ac3ac4bfbe46f09d52dfd158853667809fb4395252f4bdfd65f32e72b42"} Jul 23 11:35:36.931086 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:36.931002 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1e281f98-6114-4c25-9b31-79c65f9100e2-original-pull-secret\") pod \"global-pull-secret-syncer-8df69\" (UID: \"1e281f98-6114-4c25-9b31-79c65f9100e2\") " pod="kube-system/global-pull-secret-syncer-8df69" Jul 23 11:35:36.931239 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:36.931089 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8886ab91-1d26-47c4-835e-a2905af1aa16-metrics-certs\") pod \"network-metrics-daemon-x7q2v\" (UID: \"8886ab91-1d26-47c4-835e-a2905af1aa16\") " pod="openshift-multus/network-metrics-daemon-x7q2v" Jul 23 11:35:36.931345 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:36.931326 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 11:35:36.931413 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:36.931400 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8886ab91-1d26-47c4-835e-a2905af1aa16-metrics-certs podName:8886ab91-1d26-47c4-835e-a2905af1aa16 nodeName:}" failed. No retries permitted until 2026-07-23 11:35:38.931381369 +0000 UTC m=+6.003071339 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8886ab91-1d26-47c4-835e-a2905af1aa16-metrics-certs") pod "network-metrics-daemon-x7q2v" (UID: "8886ab91-1d26-47c4-835e-a2905af1aa16") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 11:35:36.931850 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:36.931831 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 11:35:36.931951 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:36.931884 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e281f98-6114-4c25-9b31-79c65f9100e2-original-pull-secret podName:1e281f98-6114-4c25-9b31-79c65f9100e2 nodeName:}" failed. No retries permitted until 2026-07-23 11:35:38.931870052 +0000 UTC m=+6.003560025 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1e281f98-6114-4c25-9b31-79c65f9100e2-original-pull-secret") pod "global-pull-secret-syncer-8df69" (UID: "1e281f98-6114-4c25-9b31-79c65f9100e2") : object "kube-system"/"original-pull-secret" not registered Jul 23 11:35:37.032320 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:37.031652 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnlxz\" (UniqueName: \"kubernetes.io/projected/c92ad538-9642-4c08-b92c-886738d3c413-kube-api-access-gnlxz\") pod \"network-check-target-fv772\" (UID: \"c92ad538-9642-4c08-b92c-886738d3c413\") " pod="openshift-network-diagnostics/network-check-target-fv772" Jul 23 11:35:37.032320 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:37.031853 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 11:35:37.032320 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:37.031871 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 11:35:37.032320 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:37.031897 2561 projected.go:196] Error preparing data for projected volume kube-api-access-gnlxz for pod openshift-network-diagnostics/network-check-target-fv772: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 11:35:37.032320 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:37.031974 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c92ad538-9642-4c08-b92c-886738d3c413-kube-api-access-gnlxz podName:c92ad538-9642-4c08-b92c-886738d3c413 nodeName:}" failed. No retries permitted until 2026-07-23 11:35:39.031956168 +0000 UTC m=+6.103646141 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-gnlxz" (UniqueName: "kubernetes.io/projected/c92ad538-9642-4c08-b92c-886738d3c413-kube-api-access-gnlxz") pod "network-check-target-fv772" (UID: "c92ad538-9642-4c08-b92c-886738d3c413") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 11:35:37.367405 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:37.367269 2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-22 11:30:35 +0000 UTC" deadline="2028-01-19 17:04:00.641182291 +0000 UTC" Jul 23 11:35:37.367405 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:37.367326 2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13085h28m23.273859675s" Jul 23 11:35:37.450792 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:37.450032 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8df69" Jul 23 11:35:37.450792 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:37.450161 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8df69" podUID="1e281f98-6114-4c25-9b31-79c65f9100e2" Jul 23 11:35:37.450792 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:37.450606 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7q2v" Jul 23 11:35:37.450792 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:37.450718 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x7q2v" podUID="8886ab91-1d26-47c4-835e-a2905af1aa16" Jul 23 11:35:37.451392 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:37.451196 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fv772" Jul 23 11:35:37.451392 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:37.451288 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fv772" podUID="c92ad538-9642-4c08-b92c-886738d3c413" Jul 23 11:35:37.490453 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:37.490417 2561 generic.go:356] "Generic (PLEG): container finished" podID="b1b25b788fe3c6b4ce0ebd3a8ca6fd57" containerID="9881f7f337579021f2f27e8d6dbf0f78e43e880f1724aa8501fea23046c833c2" exitCode=0 Jul 23 11:35:37.490612 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:37.490509 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-236.ec2.internal" event={"ID":"b1b25b788fe3c6b4ce0ebd3a8ca6fd57","Type":"ContainerDied","Data":"9881f7f337579021f2f27e8d6dbf0f78e43e880f1724aa8501fea23046c833c2"} Jul 23 11:35:37.498310 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:37.498275 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-236.ec2.internal" event={"ID":"f6f6dd6dc338193833341a6aab50de06","Type":"ContainerStarted","Data":"cea56b9fab93bc46bff570f2e6e3c233debe997f5df9f88d826b39fccb44431c"} Jul 23 11:35:37.525628 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:37.525571 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-236.ec2.internal" podStartSLOduration=3.525549947 podStartE2EDuration="3.525549947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 11:35:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:35:37.5248515 +0000 UTC m=+4.596541490" watchObservedRunningTime="2026-07-23 11:35:37.525549947 +0000 UTC m=+4.597239938" Jul 23 11:35:38.532401 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:38.532368 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-236.ec2.internal" event={"ID":"b1b25b788fe3c6b4ce0ebd3a8ca6fd57","Type":"ContainerStarted","Data":"8bc69516da2e56ae3cf7955e2ed39041a18aec5d54a2e9299932833d5c588602"} Jul 23 11:35:38.947011 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:38.946933 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1e281f98-6114-4c25-9b31-79c65f9100e2-original-pull-secret\") pod \"global-pull-secret-syncer-8df69\" (UID: \"1e281f98-6114-4c25-9b31-79c65f9100e2\") " pod="kube-system/global-pull-secret-syncer-8df69" Jul 23 11:35:38.947011 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:38.946988 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8886ab91-1d26-47c4-835e-a2905af1aa16-metrics-certs\") pod \"network-metrics-daemon-x7q2v\" (UID: \"8886ab91-1d26-47c4-835e-a2905af1aa16\") " pod="openshift-multus/network-metrics-daemon-x7q2v" Jul 23 11:35:38.947201 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:38.947102 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 11:35:38.947201 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:38.947121 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 11:35:38.947201 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:38.947167 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8886ab91-1d26-47c4-835e-a2905af1aa16-metrics-certs podName:8886ab91-1d26-47c4-835e-a2905af1aa16 nodeName:}" failed. No retries permitted until 2026-07-23 11:35:42.947147437 +0000 UTC m=+10.018837405 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8886ab91-1d26-47c4-835e-a2905af1aa16-metrics-certs") pod "network-metrics-daemon-x7q2v" (UID: "8886ab91-1d26-47c4-835e-a2905af1aa16") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 11:35:38.947201 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:38.947183 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e281f98-6114-4c25-9b31-79c65f9100e2-original-pull-secret podName:1e281f98-6114-4c25-9b31-79c65f9100e2 nodeName:}" failed. No retries permitted until 2026-07-23 11:35:42.947174408 +0000 UTC m=+10.018864384 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1e281f98-6114-4c25-9b31-79c65f9100e2-original-pull-secret") pod "global-pull-secret-syncer-8df69" (UID: "1e281f98-6114-4c25-9b31-79c65f9100e2") : object "kube-system"/"original-pull-secret" not registered Jul 23 11:35:39.048794 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:39.048152 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnlxz\" (UniqueName: \"kubernetes.io/projected/c92ad538-9642-4c08-b92c-886738d3c413-kube-api-access-gnlxz\") pod \"network-check-target-fv772\" (UID: \"c92ad538-9642-4c08-b92c-886738d3c413\") " pod="openshift-network-diagnostics/network-check-target-fv772" Jul 23 11:35:39.048794 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:39.048367 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 11:35:39.048794 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:39.048388 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 11:35:39.048794 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:39.048403 2561 projected.go:196] Error preparing data for projected volume kube-api-access-gnlxz for pod openshift-network-diagnostics/network-check-target-fv772: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 11:35:39.048794 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:39.048464 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c92ad538-9642-4c08-b92c-886738d3c413-kube-api-access-gnlxz podName:c92ad538-9642-4c08-b92c-886738d3c413 nodeName:}" failed. No retries permitted until 2026-07-23 11:35:43.048444194 +0000 UTC m=+10.120134166 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-gnlxz" (UniqueName: "kubernetes.io/projected/c92ad538-9642-4c08-b92c-886738d3c413-kube-api-access-gnlxz") pod "network-check-target-fv772" (UID: "c92ad538-9642-4c08-b92c-886738d3c413") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 11:35:39.451958 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:39.451013 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7q2v" Jul 23 11:35:39.451958 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:39.451157 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x7q2v" podUID="8886ab91-1d26-47c4-835e-a2905af1aa16" Jul 23 11:35:39.451958 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:39.451523 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fv772" Jul 23 11:35:39.451958 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:39.451617 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fv772" podUID="c92ad538-9642-4c08-b92c-886738d3c413" Jul 23 11:35:39.452500 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:39.452345 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8df69" Jul 23 11:35:39.452500 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:39.452445 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8df69" podUID="1e281f98-6114-4c25-9b31-79c65f9100e2" Jul 23 11:35:41.453956 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:41.453455 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fv772" Jul 23 11:35:41.453956 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:41.453565 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7q2v" Jul 23 11:35:41.453956 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:41.453578 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fv772" podUID="c92ad538-9642-4c08-b92c-886738d3c413" Jul 23 11:35:41.453956 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:41.453614 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8df69" Jul 23 11:35:41.453956 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:41.453694 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x7q2v" podUID="8886ab91-1d26-47c4-835e-a2905af1aa16" Jul 23 11:35:41.453956 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:41.453747 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8df69" podUID="1e281f98-6114-4c25-9b31-79c65f9100e2" Jul 23 11:35:42.982416 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:42.982378 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1e281f98-6114-4c25-9b31-79c65f9100e2-original-pull-secret\") pod \"global-pull-secret-syncer-8df69\" (UID: \"1e281f98-6114-4c25-9b31-79c65f9100e2\") " pod="kube-system/global-pull-secret-syncer-8df69" Jul 23 11:35:42.982781 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:42.982425 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8886ab91-1d26-47c4-835e-a2905af1aa16-metrics-certs\") pod \"network-metrics-daemon-x7q2v\" (UID: \"8886ab91-1d26-47c4-835e-a2905af1aa16\") " pod="openshift-multus/network-metrics-daemon-x7q2v" Jul 23 11:35:42.982781 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:42.982532 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 11:35:42.982781 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:42.982583 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8886ab91-1d26-47c4-835e-a2905af1aa16-metrics-certs podName:8886ab91-1d26-47c4-835e-a2905af1aa16 nodeName:}" failed. No retries permitted until 2026-07-23 11:35:50.982568617 +0000 UTC m=+18.054258589 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8886ab91-1d26-47c4-835e-a2905af1aa16-metrics-certs") pod "network-metrics-daemon-x7q2v" (UID: "8886ab91-1d26-47c4-835e-a2905af1aa16") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 11:35:42.982894 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:42.982877 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 11:35:42.982951 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:42.982938 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e281f98-6114-4c25-9b31-79c65f9100e2-original-pull-secret podName:1e281f98-6114-4c25-9b31-79c65f9100e2 nodeName:}" failed. No retries permitted until 2026-07-23 11:35:50.982924914 +0000 UTC m=+18.054614882 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1e281f98-6114-4c25-9b31-79c65f9100e2-original-pull-secret") pod "global-pull-secret-syncer-8df69" (UID: "1e281f98-6114-4c25-9b31-79c65f9100e2") : object "kube-system"/"original-pull-secret" not registered Jul 23 11:35:43.083775 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:43.083660 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnlxz\" (UniqueName: \"kubernetes.io/projected/c92ad538-9642-4c08-b92c-886738d3c413-kube-api-access-gnlxz\") pod \"network-check-target-fv772\" (UID: \"c92ad538-9642-4c08-b92c-886738d3c413\") " pod="openshift-network-diagnostics/network-check-target-fv772" Jul 23 11:35:43.083974 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:43.083828 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 11:35:43.083974 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:43.083854 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 11:35:43.083974 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:43.083867 2561 projected.go:196] Error preparing data for projected volume kube-api-access-gnlxz for pod openshift-network-diagnostics/network-check-target-fv772: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 11:35:43.083974 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:43.083952 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c92ad538-9642-4c08-b92c-886738d3c413-kube-api-access-gnlxz podName:c92ad538-9642-4c08-b92c-886738d3c413 nodeName:}" failed. No retries permitted until 2026-07-23 11:35:51.083923925 +0000 UTC m=+18.155613910 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-gnlxz" (UniqueName: "kubernetes.io/projected/c92ad538-9642-4c08-b92c-886738d3c413-kube-api-access-gnlxz") pod "network-check-target-fv772" (UID: "c92ad538-9642-4c08-b92c-886738d3c413") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 11:35:43.450050 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:43.449958 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7q2v" Jul 23 11:35:43.450050 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:43.449986 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fv772" Jul 23 11:35:43.450223 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:43.450106 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x7q2v" podUID="8886ab91-1d26-47c4-835e-a2905af1aa16" Jul 23 11:35:43.450332 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:43.450314 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8df69" Jul 23 11:35:43.450608 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:43.450414 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8df69" podUID="1e281f98-6114-4c25-9b31-79c65f9100e2" Jul 23 11:35:43.450608 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:43.450532 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fv772" podUID="c92ad538-9642-4c08-b92c-886738d3c413" Jul 23 11:35:45.449069 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:45.449032 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8df69" Jul 23 11:35:45.449069 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:45.449052 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7q2v" Jul 23 11:35:45.449712 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:45.449150 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8df69" podUID="1e281f98-6114-4c25-9b31-79c65f9100e2" Jul 23 11:35:45.449712 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:45.449209 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fv772" Jul 23 11:35:45.449712 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:45.449328 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x7q2v" podUID="8886ab91-1d26-47c4-835e-a2905af1aa16" Jul 23 11:35:45.449712 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:45.449401 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fv772" podUID="c92ad538-9642-4c08-b92c-886738d3c413" Jul 23 11:35:47.449297 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:47.449269 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7q2v" Jul 23 11:35:47.449652 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:47.449308 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8df69" Jul 23 11:35:47.449652 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:47.449384 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x7q2v" podUID="8886ab91-1d26-47c4-835e-a2905af1aa16" Jul 23 11:35:47.449652 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:47.449464 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8df69" podUID="1e281f98-6114-4c25-9b31-79c65f9100e2" Jul 23 11:35:47.449652 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:47.449520 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fv772" Jul 23 11:35:47.449652 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:47.449583 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fv772" podUID="c92ad538-9642-4c08-b92c-886738d3c413" Jul 23 11:35:49.450266 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:49.449805 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8df69" Jul 23 11:35:49.450266 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:49.449835 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fv772" Jul 23 11:35:49.450266 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:49.449805 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7q2v" Jul 23 11:35:49.450266 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:49.449938 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8df69" podUID="1e281f98-6114-4c25-9b31-79c65f9100e2" Jul 23 11:35:49.450266 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:49.450043 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x7q2v" podUID="8886ab91-1d26-47c4-835e-a2905af1aa16" Jul 23 11:35:49.450266 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:49.450110 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fv772" podUID="c92ad538-9642-4c08-b92c-886738d3c413" Jul 23 11:35:51.046107 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:51.046064 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1e281f98-6114-4c25-9b31-79c65f9100e2-original-pull-secret\") pod \"global-pull-secret-syncer-8df69\" (UID: \"1e281f98-6114-4c25-9b31-79c65f9100e2\") " pod="kube-system/global-pull-secret-syncer-8df69" Jul 23 11:35:51.046569 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:51.046115 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8886ab91-1d26-47c4-835e-a2905af1aa16-metrics-certs\") pod \"network-metrics-daemon-x7q2v\" (UID: \"8886ab91-1d26-47c4-835e-a2905af1aa16\") " pod="openshift-multus/network-metrics-daemon-x7q2v" Jul 23 11:35:51.046569 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:51.046245 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 11:35:51.046569 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:51.046250 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 11:35:51.046569 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:51.046305 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8886ab91-1d26-47c4-835e-a2905af1aa16-metrics-certs podName:8886ab91-1d26-47c4-835e-a2905af1aa16 nodeName:}" failed. No retries permitted until 2026-07-23 11:36:07.046289772 +0000 UTC m=+34.117979742 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8886ab91-1d26-47c4-835e-a2905af1aa16-metrics-certs") pod "network-metrics-daemon-x7q2v" (UID: "8886ab91-1d26-47c4-835e-a2905af1aa16") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 11:35:51.046569 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:51.046320 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e281f98-6114-4c25-9b31-79c65f9100e2-original-pull-secret podName:1e281f98-6114-4c25-9b31-79c65f9100e2 nodeName:}" failed. No retries permitted until 2026-07-23 11:36:07.046313627 +0000 UTC m=+34.118003595 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1e281f98-6114-4c25-9b31-79c65f9100e2-original-pull-secret") pod "global-pull-secret-syncer-8df69" (UID: "1e281f98-6114-4c25-9b31-79c65f9100e2") : object "kube-system"/"original-pull-secret" not registered Jul 23 11:35:51.146868 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:51.146822 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnlxz\" (UniqueName: \"kubernetes.io/projected/c92ad538-9642-4c08-b92c-886738d3c413-kube-api-access-gnlxz\") pod \"network-check-target-fv772\" (UID: \"c92ad538-9642-4c08-b92c-886738d3c413\") " pod="openshift-network-diagnostics/network-check-target-fv772" Jul 23 11:35:51.147058 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:51.147020 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 11:35:51.147058 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:51.147043 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 11:35:51.147058 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:51.147052 2561 projected.go:196] Error preparing data for projected volume kube-api-access-gnlxz for pod openshift-network-diagnostics/network-check-target-fv772: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 11:35:51.147185 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:51.147104 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c92ad538-9642-4c08-b92c-886738d3c413-kube-api-access-gnlxz podName:c92ad538-9642-4c08-b92c-886738d3c413 nodeName:}" failed. No retries permitted until 2026-07-23 11:36:07.147091707 +0000 UTC m=+34.218781675 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-gnlxz" (UniqueName: "kubernetes.io/projected/c92ad538-9642-4c08-b92c-886738d3c413-kube-api-access-gnlxz") pod "network-check-target-fv772" (UID: "c92ad538-9642-4c08-b92c-886738d3c413") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 11:35:51.449521 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:51.449479 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8df69" Jul 23 11:35:51.449521 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:51.449502 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fv772" Jul 23 11:35:51.449699 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:51.449491 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7q2v" Jul 23 11:35:51.449699 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:51.449581 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8df69" podUID="1e281f98-6114-4c25-9b31-79c65f9100e2" Jul 23 11:35:51.449699 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:51.449665 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fv772" podUID="c92ad538-9642-4c08-b92c-886738d3c413" Jul 23 11:35:51.449825 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:51.449734 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x7q2v" podUID="8886ab91-1d26-47c4-835e-a2905af1aa16" Jul 23 11:35:53.449791 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:53.449767 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8df69" Jul 23 11:35:53.450250 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:53.449870 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8df69" podUID="1e281f98-6114-4c25-9b31-79c65f9100e2" Jul 23 11:35:53.450250 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:53.449967 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7q2v" Jul 23 11:35:53.450250 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:53.450078 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fv772" Jul 23 11:35:53.450250 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:53.450103 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x7q2v" podUID="8886ab91-1d26-47c4-835e-a2905af1aa16" Jul 23 11:35:53.450250 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:53.450155 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fv772" podUID="c92ad538-9642-4c08-b92c-886738d3c413" Jul 23 11:35:55.450236 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:55.449884 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8df69" Jul 23 11:35:55.450958 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:55.449931 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7q2v" Jul 23 11:35:55.450958 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:55.450322 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8df69" podUID="1e281f98-6114-4c25-9b31-79c65f9100e2" Jul 23 11:35:55.450958 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:55.449953 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fv772" Jul 23 11:35:55.450958 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:55.450376 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x7q2v" podUID="8886ab91-1d26-47c4-835e-a2905af1aa16" Jul 23 11:35:55.450958 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:55.450469 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fv772" podUID="c92ad538-9642-4c08-b92c-886738d3c413" Jul 23 11:35:55.567024 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:55.566996 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sf88g" event={"ID":"ebf70217-31df-4e81-be74-df699db721f0","Type":"ContainerStarted","Data":"e4c16b239c6f180f8764cd98ddf3b9e1a18efae6551304a7a61e8fdc375ae55b"} Jul 23 11:35:55.568309 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:55.568281 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lp5f4" event={"ID":"620c5e1f-97de-42e6-b752-50d951728a5d","Type":"ContainerStarted","Data":"68f7eb05e84123fc9e899984e06e772729b8711cdde1cc8906dac24e1ee0d73c"} Jul 23 11:35:55.569578 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:55.569557 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8d65t" event={"ID":"792c59f0-f23f-4694-9cde-37b9431760f4","Type":"ContainerStarted","Data":"7c168d479425ad285c650afca1a8513e9db7d4eb85d126a39f2c77ede46aaaa5"} Jul 23 11:35:55.576086 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:55.576064 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6cbrx" event={"ID":"aa374cc8-936e-41ca-a905-c0449df6825d","Type":"ContainerStarted","Data":"a3949e0cc7ef53fd00c35bd839092c32cdcc8fcbef91a9681ccf375f74187b77"} Jul 23 11:35:55.577422 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:55.577394 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-chtf2" event={"ID":"ba1ca519-e6da-43b0-9659-ec51fac539a4","Type":"ContainerStarted","Data":"d384325d964cca956ec6b2b6b334582646c7b21833d12930c2ff61eacc596dc7"} Jul 23 11:35:55.578561 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:55.578537 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-pkwm5" event={"ID":"a17ee9c0-c387-4e9d-a6a4-481501f5b97b","Type":"ContainerStarted","Data":"aec37c03573fe29fd11459638583ed080398a20f1302526f9d6bbf4cbe7fc6b0"} Jul 23 11:35:55.580520 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:55.580495 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-92n98" event={"ID":"2c75320d-d67b-4e9e-b648-cc748f5a2870","Type":"ContainerStarted","Data":"3f4b93e5d8c200e91f60b0f8a94e120f81c755fd6afacc18070291bb7f439744"} Jul 23 11:35:55.580616 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:55.580565 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-92n98" event={"ID":"2c75320d-d67b-4e9e-b648-cc748f5a2870","Type":"ContainerStarted","Data":"91de9bf4c425822f8ff9621b1d8d35ac87ed3040ab05ec25e15ef7e936193ed6"} Jul 23 11:35:55.580616 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:55.580600 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-92n98" event={"ID":"2c75320d-d67b-4e9e-b648-cc748f5a2870","Type":"ContainerStarted","Data":"c5ea7298704c66d780323fbda5ad6e86aa04214b70835a024609ea13d5727553"} Jul 23 11:35:55.581859 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:55.581839 2561 generic.go:356] "Generic (PLEG): container finished" podID="f898b85f-57f0-4973-9e99-117e9fd60fa8" containerID="ce256c0d999666608f4cd559279ae5a6e6d4b527e954bc0b26fd15f0af6ca033" exitCode=0 Jul 23 11:35:55.581940 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:55.581867 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6jdp4" event={"ID":"f898b85f-57f0-4973-9e99-117e9fd60fa8","Type":"ContainerDied","Data":"ce256c0d999666608f4cd559279ae5a6e6d4b527e954bc0b26fd15f0af6ca033"} Jul 23 11:35:55.586314 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:55.586280 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-236.ec2.internal" podStartSLOduration=21.586269124 podStartE2EDuration="21.586269124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 11:35:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:35:38.552877666 +0000 UTC m=+5.624567655" watchObservedRunningTime="2026-07-23 11:35:55.586269124 +0000 UTC m=+22.657959116" Jul 23 11:35:55.586437 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:55.586419 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-lp5f4" podStartSLOduration=3.8791865850000002 podStartE2EDuration="22.586413965s" totalImagesPullingTime="18.70722738s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:35:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:35:55.586157353 +0000 UTC m=+22.657847336" watchObservedRunningTime="2026-07-23 11:35:55.586413965 +0000 UTC m=+22.658103954" Jul 23 11:35:55.601614 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:55.601537 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-pkwm5" podStartSLOduration=3.905688151 podStartE2EDuration="22.601529627s" totalImagesPullingTime="18.695841476s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 11:35:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:35:55.601292715 +0000 UTC m=+22.672982704" watchObservedRunningTime="2026-07-23 11:35:55.601529627 +0000 UTC m=+22.673219616" Jul 23 11:35:55.659626 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:55.659588 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-6cbrx" podStartSLOduration=3.963241907 podStartE2EDuration="22.659576283s" totalImagesPullingTime="18.696334376s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 11:35:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:35:55.639212903 +0000 UTC m=+22.710902893" watchObservedRunningTime="2026-07-23 11:35:55.659576283 +0000 UTC m=+22.731266333" Jul 23 11:35:55.659727 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:55.659691 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-8d65t" podStartSLOduration=3.912261628 podStartE2EDuration="22.659674215s" totalImagesPullingTime="18.747412587s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 11:35:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:35:55.659552664 +0000 UTC m=+22.731242657" watchObservedRunningTime="2026-07-23 11:35:55.659674215 +0000 UTC m=+22.731364221" Jul 23 11:35:55.683215 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:55.683187 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-chtf2" podStartSLOduration=3.988786533 podStartE2EDuration="22.683176551s" totalImagesPullingTime="18.694390018s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 11:35:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:35:55.682554043 +0000 UTC m=+22.754244032" watchObservedRunningTime="2026-07-23 11:35:55.683176551 +0000 UTC m=+22.754866541" Jul 23 11:35:56.585476 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:56.585442 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zfngm" event={"ID":"917d3c0c-f95b-43ac-a39a-b21dfdc0c07c","Type":"ContainerStarted","Data":"875ffe5a7ac7315f47f81b806295e1f3773f84508b77e4346719fc06aced563d"} Jul 23 11:35:56.588822 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:56.588781 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-92n98" event={"ID":"2c75320d-d67b-4e9e-b648-cc748f5a2870","Type":"ContainerStarted","Data":"b6b68662ffe2c68f314d78f11a99721b48169eec6d2184e93cc537d34c5df851"} Jul 23 11:35:56.588955 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:56.588843 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-92n98" event={"ID":"2c75320d-d67b-4e9e-b648-cc748f5a2870","Type":"ContainerStarted","Data":"c728656965c60ebb8ad2cc158e1dd02c6efcf8087d805e964adf813054b2d9a4"} Jul 23 11:35:56.588955 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:56.588872 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-92n98" event={"ID":"2c75320d-d67b-4e9e-b648-cc748f5a2870","Type":"ContainerStarted","Data":"e85564c0e7884359a236576d0ca115681d352926944fd8fc4bdc477a9a34fb57"} Jul 23 11:35:56.604269 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:56.604213 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-zfngm" podStartSLOduration=4.898873337 podStartE2EDuration="23.604198345s" totalImagesPullingTime="18.705325008s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 11:35:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:35:56.603112849 +0000 UTC m=+23.674802835" watchObservedRunningTime="2026-07-23 11:35:56.604198345 +0000 UTC m=+23.675888353" Jul 23 11:35:56.625551 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:56.625529 2561 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 23 11:35:57.343115 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:57.343083 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-lp5f4" Jul 23 11:35:57.343716 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:57.343696 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-lp5f4" Jul 23 11:35:57.372902 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:57.372818 2561 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-23T11:35:56.625548195Z","UUID":"52e27b51-6cdb-401d-8c9e-2a72f6e6190b","Handler":null,"Name":"","Endpoint":""} Jul 23 11:35:57.374413 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:57.374394 2561 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 23 11:35:57.374507 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:57.374434 2561 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 23 11:35:57.449084 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:57.449058 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8df69" Jul 23 11:35:57.449200 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:57.449098 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fv772" Jul 23 11:35:57.449200 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:57.449157 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8df69" podUID="1e281f98-6114-4c25-9b31-79c65f9100e2" Jul 23 11:35:57.449305 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:57.449058 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7q2v" Jul 23 11:35:57.449355 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:57.449292 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fv772" podUID="c92ad538-9642-4c08-b92c-886738d3c413" Jul 23 11:35:57.449409 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:57.449393 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x7q2v" podUID="8886ab91-1d26-47c4-835e-a2905af1aa16" Jul 23 11:35:57.591977 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:57.591951 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sf88g" event={"ID":"ebf70217-31df-4e81-be74-df699db721f0","Type":"ContainerStarted","Data":"5967defcb8deb766e41285775e24e91187a780dbdf0a3351e23d79b527376e92"} Jul 23 11:35:57.592354 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:57.592250 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-lp5f4" Jul 23 11:35:57.592650 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:57.592628 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-lp5f4" Jul 23 11:35:58.598483 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:58.598180 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-92n98" event={"ID":"2c75320d-d67b-4e9e-b648-cc748f5a2870","Type":"ContainerStarted","Data":"cc6cff23d1a9cfa9c81a9c97aeeb2fbf13d8954985af5b26d1726c4130093a3d"} Jul 23 11:35:58.600472 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:58.600448 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sf88g" event={"ID":"ebf70217-31df-4e81-be74-df699db721f0","Type":"ContainerStarted","Data":"801649864d4b4e1f3ec56992fcdcda000e2a750064fdb1f9187f6d51baec6034"} Jul 23 11:35:58.619000 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:58.618952 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sf88g" podStartSLOduration=4.08546316 podStartE2EDuration="25.618936351s" totalImagesPullingTime="21.533473191s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 11:35:33 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:35:58.618182758 +0000 UTC m=+25.689872759" watchObservedRunningTime="2026-07-23 11:35:58.618936351 +0000 UTC m=+25.690626340" Jul 23 11:35:59.449053 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:59.449024 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7q2v" Jul 23 11:35:59.449284 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:59.449028 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8df69" Jul 23 11:35:59.449284 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:35:59.449169 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fv772" Jul 23 11:35:59.449284 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:59.449167 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x7q2v" podUID="8886ab91-1d26-47c4-835e-a2905af1aa16" Jul 23 11:35:59.449445 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:59.449288 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8df69" podUID="1e281f98-6114-4c25-9b31-79c65f9100e2" Jul 23 11:35:59.449445 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:35:59.449372 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fv772" podUID="c92ad538-9642-4c08-b92c-886738d3c413" Jul 23 11:36:00.609135 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:00.608941 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-92n98" event={"ID":"2c75320d-d67b-4e9e-b648-cc748f5a2870","Type":"ContainerStarted","Data":"e4850a64df868f512cbf8448ba3223edfb18059f4e199a1ca57237f421c36655"} Jul 23 11:36:00.609873 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:00.609177 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-92n98" Jul 23 11:36:00.610805 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:00.610781 2561 generic.go:356] "Generic (PLEG): container finished" podID="f898b85f-57f0-4973-9e99-117e9fd60fa8" containerID="6d6bb066b9bc58c45ef14bcd1d42edcead89c3cb71875e99758e177ea1735b34" exitCode=0 Jul 23 11:36:00.610926 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:00.610828 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6jdp4" event={"ID":"f898b85f-57f0-4973-9e99-117e9fd60fa8","Type":"ContainerDied","Data":"6d6bb066b9bc58c45ef14bcd1d42edcead89c3cb71875e99758e177ea1735b34"} Jul 23 11:36:00.625735 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:00.625714 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-92n98" Jul 23 11:36:00.637559 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:00.637521 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-92n98" podStartSLOduration=8.870109339 podStartE2EDuration="27.637510939s" totalImagesPullingTime="18.7674016s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 11:35:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:36:00.637359489 +0000 UTC m=+27.709049478" watchObservedRunningTime="2026-07-23 11:36:00.637510939 +0000 UTC m=+27.709200927" Jul 23 11:36:01.449884 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:01.449856 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8df69" Jul 23 11:36:01.449884 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:01.449873 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7q2v" Jul 23 11:36:01.450100 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:01.449963 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8df69" podUID="1e281f98-6114-4c25-9b31-79c65f9100e2" Jul 23 11:36:01.450100 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:01.450007 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x7q2v" podUID="8886ab91-1d26-47c4-835e-a2905af1aa16" Jul 23 11:36:01.450100 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:01.450065 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fv772" Jul 23 11:36:01.450230 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:01.450154 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fv772" podUID="c92ad538-9642-4c08-b92c-886738d3c413" Jul 23 11:36:01.621902 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:01.620659 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-92n98" Jul 23 11:36:01.621902 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:01.620825 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-92n98" Jul 23 11:36:01.636797 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:01.636768 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-92n98" Jul 23 11:36:02.270192 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:02.270169 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fv772"] Jul 23 11:36:02.270335 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:02.270270 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fv772" Jul 23 11:36:02.270396 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:02.270374 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fv772" podUID="c92ad538-9642-4c08-b92c-886738d3c413" Jul 23 11:36:02.274599 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:02.274575 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8df69"] Jul 23 11:36:02.274712 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:02.274670 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8df69" Jul 23 11:36:02.274791 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:02.274770 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8df69" podUID="1e281f98-6114-4c25-9b31-79c65f9100e2" Jul 23 11:36:02.275155 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:02.275136 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-x7q2v"] Jul 23 11:36:02.275233 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:02.275220 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7q2v" Jul 23 11:36:02.275386 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:02.275304 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x7q2v" podUID="8886ab91-1d26-47c4-835e-a2905af1aa16" Jul 23 11:36:02.623819 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:02.623788 2561 generic.go:356] "Generic (PLEG): container finished" podID="f898b85f-57f0-4973-9e99-117e9fd60fa8" containerID="dffa8d4b6aa37e14eaae6d38bea944603dc84fd12363a729548196a863989ba0" exitCode=0 Jul 23 11:36:02.624216 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:02.623877 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6jdp4" event={"ID":"f898b85f-57f0-4973-9e99-117e9fd60fa8","Type":"ContainerDied","Data":"dffa8d4b6aa37e14eaae6d38bea944603dc84fd12363a729548196a863989ba0"} Jul 23 11:36:03.449922 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:03.449875 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fv772" Jul 23 11:36:03.450083 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:03.449976 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fv772" podUID="c92ad538-9642-4c08-b92c-886738d3c413" Jul 23 11:36:03.450083 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:03.450067 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7q2v" Jul 23 11:36:03.450179 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:03.450158 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x7q2v" podUID="8886ab91-1d26-47c4-835e-a2905af1aa16" Jul 23 11:36:04.449359 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:04.449335 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8df69" Jul 23 11:36:04.449655 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:04.449429 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8df69" podUID="1e281f98-6114-4c25-9b31-79c65f9100e2" Jul 23 11:36:04.629551 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:04.629517 2561 generic.go:356] "Generic (PLEG): container finished" podID="f898b85f-57f0-4973-9e99-117e9fd60fa8" containerID="97a20b19dae01f5ef65f1485f256c46a0d7702a7ced3d13ce2026d7a48c178a0" exitCode=0 Jul 23 11:36:04.629675 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:04.629560 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6jdp4" event={"ID":"f898b85f-57f0-4973-9e99-117e9fd60fa8","Type":"ContainerDied","Data":"97a20b19dae01f5ef65f1485f256c46a0d7702a7ced3d13ce2026d7a48c178a0"} Jul 23 11:36:05.449309 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:05.449118 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7q2v" Jul 23 11:36:05.449512 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:05.449171 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fv772" Jul 23 11:36:05.449512 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:05.449416 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x7q2v" podUID="8886ab91-1d26-47c4-835e-a2905af1aa16" Jul 23 11:36:05.449512 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:05.449498 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fv772" podUID="c92ad538-9642-4c08-b92c-886738d3c413" Jul 23 11:36:06.449120 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:06.449089 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8df69" Jul 23 11:36:06.449292 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:06.449196 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8df69" podUID="1e281f98-6114-4c25-9b31-79c65f9100e2" Jul 23 11:36:07.071305 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.071273 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1e281f98-6114-4c25-9b31-79c65f9100e2-original-pull-secret\") pod \"global-pull-secret-syncer-8df69\" (UID: \"1e281f98-6114-4c25-9b31-79c65f9100e2\") " pod="kube-system/global-pull-secret-syncer-8df69" Jul 23 11:36:07.071305 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.071308 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8886ab91-1d26-47c4-835e-a2905af1aa16-metrics-certs\") pod \"network-metrics-daemon-x7q2v\" (UID: \"8886ab91-1d26-47c4-835e-a2905af1aa16\") " pod="openshift-multus/network-metrics-daemon-x7q2v" Jul 23 11:36:07.071995 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:07.071417 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 11:36:07.071995 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:07.071417 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 11:36:07.071995 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:07.071478 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e281f98-6114-4c25-9b31-79c65f9100e2-original-pull-secret podName:1e281f98-6114-4c25-9b31-79c65f9100e2 nodeName:}" failed. No retries permitted until 2026-07-23 11:36:39.071462587 +0000 UTC m=+66.143152579 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1e281f98-6114-4c25-9b31-79c65f9100e2-original-pull-secret") pod "global-pull-secret-syncer-8df69" (UID: "1e281f98-6114-4c25-9b31-79c65f9100e2") : object "kube-system"/"original-pull-secret" not registered Jul 23 11:36:07.071995 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:07.071491 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8886ab91-1d26-47c4-835e-a2905af1aa16-metrics-certs podName:8886ab91-1d26-47c4-835e-a2905af1aa16 nodeName:}" failed. No retries permitted until 2026-07-23 11:36:39.071485254 +0000 UTC m=+66.143175222 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8886ab91-1d26-47c4-835e-a2905af1aa16-metrics-certs") pod "network-metrics-daemon-x7q2v" (UID: "8886ab91-1d26-47c4-835e-a2905af1aa16") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 11:36:07.172588 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.172557 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnlxz\" (UniqueName: \"kubernetes.io/projected/c92ad538-9642-4c08-b92c-886738d3c413-kube-api-access-gnlxz\") pod \"network-check-target-fv772\" (UID: \"c92ad538-9642-4c08-b92c-886738d3c413\") " pod="openshift-network-diagnostics/network-check-target-fv772" Jul 23 11:36:07.172767 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:07.172686 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 11:36:07.172767 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:07.172711 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 11:36:07.172767 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:07.172724 2561 projected.go:196] Error preparing data for projected volume kube-api-access-gnlxz for pod openshift-network-diagnostics/network-check-target-fv772: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 11:36:07.172767 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:07.172769 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c92ad538-9642-4c08-b92c-886738d3c413-kube-api-access-gnlxz podName:c92ad538-9642-4c08-b92c-886738d3c413 nodeName:}" failed. No retries permitted until 2026-07-23 11:36:39.172755691 +0000 UTC m=+66.244445660 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-gnlxz" (UniqueName: "kubernetes.io/projected/c92ad538-9642-4c08-b92c-886738d3c413-kube-api-access-gnlxz") pod "network-check-target-fv772" (UID: "c92ad538-9642-4c08-b92c-886738d3c413") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 11:36:07.449817 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.449754 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7q2v" Jul 23 11:36:07.449998 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:07.449868 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x7q2v" podUID="8886ab91-1d26-47c4-835e-a2905af1aa16" Jul 23 11:36:07.449998 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.449936 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fv772" Jul 23 11:36:07.450098 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:07.450040 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fv772" podUID="c92ad538-9642-4c08-b92c-886738d3c413" Jul 23 11:36:07.734029 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.733964 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-236.ec2.internal" event="NodeReady" Jul 23 11:36:07.734165 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.734072 2561 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 23 11:36:07.772904 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.772873 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-7slpj"] Jul 23 11:36:07.775320 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.775298 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-7slpj" Jul 23 11:36:07.776566 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.776189 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-j4sbd"] Jul 23 11:36:07.778170 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.778152 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-48j4r"] Jul 23 11:36:07.778277 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.778235 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-j4sbd" Jul 23 11:36:07.779170 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.779152 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 23 11:36:07.779266 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.779225 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 23 11:36:07.779381 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.779364 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 23 11:36:07.779450 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.779401 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-ss8sw\"" Jul 23 11:36:07.779505 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.779453 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 23 11:36:07.779727 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.779711 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-67fbb677d9-vf9k4"] Jul 23 11:36:07.779853 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.779837 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-48j4r" Jul 23 11:36:07.781326 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.781306 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 23 11:36:07.781326 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.781309 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 23 11:36:07.781488 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.781413 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-78fdc88b76-5w9ql"] Jul 23 11:36:07.781542 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.781525 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67fbb677d9-vf9k4" Jul 23 11:36:07.783046 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.783028 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-lfpvr\"" Jul 23 11:36:07.783139 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.783125 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-78fdc88b76-5w9ql" Jul 23 11:36:07.783307 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.783131 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 23 11:36:07.783470 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.783452 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 23 11:36:07.783895 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.783874 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 23 11:36:07.784360 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.784209 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-nz2dk\"" Jul 23 11:36:07.784360 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.784232 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 23 11:36:07.784360 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.784269 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 23 11:36:07.784530 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.784476 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 23 11:36:07.792086 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.792060 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 23 11:36:07.792616 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.792598 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 23 11:36:07.792770 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.792753 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 23 11:36:07.794364 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.793347 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 23 11:36:07.794364 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.793425 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 23 11:36:07.794364 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.793699 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 23 11:36:07.794364 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.793804 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 23 11:36:07.794364 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.793983 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 23 11:36:07.794364 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.794136 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-pmn7v\"" Jul 23 11:36:07.797027 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.797005 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-j9zgf\"" Jul 23 11:36:07.800559 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.800540 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 23 11:36:07.801313 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.801292 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-7slpj"] Jul 23 11:36:07.801655 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.801638 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 23 11:36:07.803375 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.803360 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-n2rn6"] Jul 23 11:36:07.805240 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.805218 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-5ss74"] Jul 23 11:36:07.805376 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.805353 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-n2rn6" Jul 23 11:36:07.805770 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.805747 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 23 11:36:07.807405 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.807386 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-fww69"] Jul 23 11:36:07.807710 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.807690 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-5ss74" Jul 23 11:36:07.809816 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.809800 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-555d9d7885-5txgl"] Jul 23 11:36:07.810185 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.810159 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fww69" Jul 23 11:36:07.811109 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.811094 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 23 11:36:07.811645 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.811626 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 23 11:36:07.812856 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.812834 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-8gtcv"] Jul 23 11:36:07.812967 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.812944 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-qfq6r\"" Jul 23 11:36:07.813128 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.812952 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 23 11:36:07.813412 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.813395 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 23 11:36:07.813945 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.813926 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-5txgl" Jul 23 11:36:07.814089 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.814074 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-cdth7\"" Jul 23 11:36:07.814625 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.814602 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 23 11:36:07.815079 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.815061 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-pb4w9\"" Jul 23 11:36:07.816530 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.815334 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 23 11:36:07.816530 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.816151 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-sr6b2"] Jul 23 11:36:07.816530 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.816267 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-8gtcv" Jul 23 11:36:07.816530 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.815893 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 23 11:36:07.816786 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.815980 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 23 11:36:07.817812 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.817798 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-48j4r"] Jul 23 11:36:07.818076 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.818061 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-j4sbd"] Jul 23 11:36:07.818220 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.818087 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-78fdc88b76-5w9ql"] Jul 23 11:36:07.818220 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.818108 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-fkvjh"] Jul 23 11:36:07.821137 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.818024 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-sr6b2" Jul 23 11:36:07.821137 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.820944 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 23 11:36:07.821137 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.821031 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 23 11:36:07.823545 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.823499 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 23 11:36:07.824007 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.823981 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-x294m\"" Jul 23 11:36:07.824197 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.824177 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 23 11:36:07.824346 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.824323 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 23 11:36:07.824727 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.824545 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 23 11:36:07.824727 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.824678 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 23 11:36:07.824796 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.824729 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 23 11:36:07.824938 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.824922 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-67fbb677d9-vf9k4"] Jul 23 11:36:07.825272 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.825257 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-n2rn6"] Jul 23 11:36:07.825392 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.824999 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-nk7hr\"" Jul 23 11:36:07.826084 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.825059 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fkvjh" Jul 23 11:36:07.826084 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.825146 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 23 11:36:07.826084 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.825305 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-442tk\"" Jul 23 11:36:07.826084 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.825965 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-5ss74"] Jul 23 11:36:07.826084 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.824938 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 23 11:36:07.827028 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.826734 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-sr6b2"] Jul 23 11:36:07.827779 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.827763 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fww69"] Jul 23 11:36:07.828316 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.828298 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-67ffh\"" Jul 23 11:36:07.828937 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.828613 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-5txgl"] Jul 23 11:36:07.828937 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.828614 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 23 11:36:07.828937 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.828662 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 23 11:36:07.829531 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.829506 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 23 11:36:07.829634 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.829582 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-8gtcv"] Jul 23 11:36:07.830994 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.830978 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fkvjh"] Jul 23 11:36:07.877663 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.877633 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5179eda-de1f-4d89-9e22-afb9d63d098b-metrics-certs\") pod \"router-default-78fdc88b76-5w9ql\" (UID: \"c5179eda-de1f-4d89-9e22-afb9d63d098b\") " pod="openshift-ingress/router-default-78fdc88b76-5w9ql" Jul 23 11:36:07.877763 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.877686 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/65247e1f-fc10-4b40-af67-5e646f352263-config\") pod \"console-operator-565b7b569d-7slpj\" (UID: \"65247e1f-fc10-4b40-af67-5e646f352263\") " pod="openshift-console-operator/console-operator-565b7b569d-7slpj" Jul 23 11:36:07.877763 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.877713 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj2wf\" (UniqueName: \"kubernetes.io/projected/65247e1f-fc10-4b40-af67-5e646f352263-kube-api-access-zj2wf\") pod \"console-operator-565b7b569d-7slpj\" (UID: \"65247e1f-fc10-4b40-af67-5e646f352263\") " pod="openshift-console-operator/console-operator-565b7b569d-7slpj" Jul 23 11:36:07.877763 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.877740 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b993c3b9-94b8-4c21-b391-6193659cf9ee-image-registry-private-configuration\") pod \"image-registry-67fbb677d9-vf9k4\" (UID: \"b993c3b9-94b8-4c21-b391-6193659cf9ee\") " pod="openshift-image-registry/image-registry-67fbb677d9-vf9k4" Jul 23 11:36:07.877862 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.877771 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f249368d-c49d-4bfe-906f-d1d12104dbd0-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-j4sbd\" (UID: \"f249368d-c49d-4bfe-906f-d1d12104dbd0\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-j4sbd" Jul 23 11:36:07.877862 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.877795 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b64d0632-5844-419e-9f53-ac57a0495a8d-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-48j4r\" (UID: \"b64d0632-5844-419e-9f53-ac57a0495a8d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-48j4r" Jul 23 11:36:07.877862 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.877818 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5179eda-de1f-4d89-9e22-afb9d63d098b-service-ca-bundle\") pod \"router-default-78fdc88b76-5w9ql\" (UID: \"c5179eda-de1f-4d89-9e22-afb9d63d098b\") " pod="openshift-ingress/router-default-78fdc88b76-5w9ql" Jul 23 11:36:07.877862 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.877840 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/c5179eda-de1f-4d89-9e22-afb9d63d098b-stats-auth\") pod \"router-default-78fdc88b76-5w9ql\" (UID: \"c5179eda-de1f-4d89-9e22-afb9d63d098b\") " pod="openshift-ingress/router-default-78fdc88b76-5w9ql" Jul 23 11:36:07.878045 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.877862 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zth9j\" (UniqueName: \"kubernetes.io/projected/c5179eda-de1f-4d89-9e22-afb9d63d098b-kube-api-access-zth9j\") pod \"router-default-78fdc88b76-5w9ql\" (UID: \"c5179eda-de1f-4d89-9e22-afb9d63d098b\") " pod="openshift-ingress/router-default-78fdc88b76-5w9ql" Jul 23 11:36:07.878045 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.877902 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b993c3b9-94b8-4c21-b391-6193659cf9ee-ca-trust-extracted\") pod \"image-registry-67fbb677d9-vf9k4\" (UID: \"b993c3b9-94b8-4c21-b391-6193659cf9ee\") " pod="openshift-image-registry/image-registry-67fbb677d9-vf9k4" Jul 23 11:36:07.878045 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.877956 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/c5179eda-de1f-4d89-9e22-afb9d63d098b-default-certificate\") pod \"router-default-78fdc88b76-5w9ql\" (UID: \"c5179eda-de1f-4d89-9e22-afb9d63d098b\") " pod="openshift-ingress/router-default-78fdc88b76-5w9ql" Jul 23 11:36:07.878178 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.878043 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f249368d-c49d-4bfe-906f-d1d12104dbd0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-j4sbd\" (UID: \"f249368d-c49d-4bfe-906f-d1d12104dbd0\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-j4sbd" Jul 23 11:36:07.878178 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.878088 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlkp9\" (UniqueName: \"kubernetes.io/projected/b993c3b9-94b8-4c21-b391-6193659cf9ee-kube-api-access-dlkp9\") pod \"image-registry-67fbb677d9-vf9k4\" (UID: \"b993c3b9-94b8-4c21-b391-6193659cf9ee\") " pod="openshift-image-registry/image-registry-67fbb677d9-vf9k4" Jul 23 11:36:07.878178 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.878144 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b64d0632-5844-419e-9f53-ac57a0495a8d-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-48j4r\" (UID: \"b64d0632-5844-419e-9f53-ac57a0495a8d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-48j4r" Jul 23 11:36:07.878320 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.878244 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/65247e1f-fc10-4b40-af67-5e646f352263-serving-cert\") pod \"console-operator-565b7b569d-7slpj\" (UID: \"65247e1f-fc10-4b40-af67-5e646f352263\") " pod="openshift-console-operator/console-operator-565b7b569d-7slpj" Jul 23 11:36:07.878320 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.878271 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/65247e1f-fc10-4b40-af67-5e646f352263-trusted-ca\") pod \"console-operator-565b7b569d-7slpj\" (UID: \"65247e1f-fc10-4b40-af67-5e646f352263\") " pod="openshift-console-operator/console-operator-565b7b569d-7slpj" Jul 23 11:36:07.878320 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.878293 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b993c3b9-94b8-4c21-b391-6193659cf9ee-bound-sa-token\") pod \"image-registry-67fbb677d9-vf9k4\" (UID: \"b993c3b9-94b8-4c21-b391-6193659cf9ee\") " pod="openshift-image-registry/image-registry-67fbb677d9-vf9k4" Jul 23 11:36:07.878460 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.878352 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vc6vp\" (UniqueName: \"kubernetes.io/projected/f249368d-c49d-4bfe-906f-d1d12104dbd0-kube-api-access-vc6vp\") pod \"cluster-monitoring-operator-76cf8966f8-j4sbd\" (UID: \"f249368d-c49d-4bfe-906f-d1d12104dbd0\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-j4sbd" Jul 23 11:36:07.878460 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.878384 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b993c3b9-94b8-4c21-b391-6193659cf9ee-trusted-ca\") pod \"image-registry-67fbb677d9-vf9k4\" (UID: \"b993c3b9-94b8-4c21-b391-6193659cf9ee\") " pod="openshift-image-registry/image-registry-67fbb677d9-vf9k4" Jul 23 11:36:07.878460 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.878428 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b993c3b9-94b8-4c21-b391-6193659cf9ee-registry-tls\") pod \"image-registry-67fbb677d9-vf9k4\" (UID: \"b993c3b9-94b8-4c21-b391-6193659cf9ee\") " pod="openshift-image-registry/image-registry-67fbb677d9-vf9k4" Jul 23 11:36:07.878576 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.878464 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b993c3b9-94b8-4c21-b391-6193659cf9ee-registry-certificates\") pod \"image-registry-67fbb677d9-vf9k4\" (UID: \"b993c3b9-94b8-4c21-b391-6193659cf9ee\") " pod="openshift-image-registry/image-registry-67fbb677d9-vf9k4" Jul 23 11:36:07.878576 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.878486 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b993c3b9-94b8-4c21-b391-6193659cf9ee-installation-pull-secrets\") pod \"image-registry-67fbb677d9-vf9k4\" (UID: \"b993c3b9-94b8-4c21-b391-6193659cf9ee\") " pod="openshift-image-registry/image-registry-67fbb677d9-vf9k4" Jul 23 11:36:07.878576 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.878521 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zdd4\" (UniqueName: \"kubernetes.io/projected/b64d0632-5844-419e-9f53-ac57a0495a8d-kube-api-access-4zdd4\") pod \"kube-storage-version-migrator-operator-7658d9b749-48j4r\" (UID: \"b64d0632-5844-419e-9f53-ac57a0495a8d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-48j4r" Jul 23 11:36:07.979844 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.979820 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b64d0632-5844-419e-9f53-ac57a0495a8d-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-48j4r\" (UID: \"b64d0632-5844-419e-9f53-ac57a0495a8d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-48j4r" Jul 23 11:36:07.980001 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.979858 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htc4j\" (UniqueName: \"kubernetes.io/projected/74038c52-71a3-4c88-b607-7f0f9ec11001-kube-api-access-htc4j\") pod \"dns-default-fkvjh\" (UID: \"74038c52-71a3-4c88-b607-7f0f9ec11001\") " pod="openshift-dns/dns-default-fkvjh" Jul 23 11:36:07.980001 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.979888 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b993c3b9-94b8-4c21-b391-6193659cf9ee-registry-certificates\") pod \"image-registry-67fbb677d9-vf9k4\" (UID: \"b993c3b9-94b8-4c21-b391-6193659cf9ee\") " pod="openshift-image-registry/image-registry-67fbb677d9-vf9k4" Jul 23 11:36:07.980001 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.979923 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/74038c52-71a3-4c88-b607-7f0f9ec11001-metrics-tls\") pod \"dns-default-fkvjh\" (UID: \"74038c52-71a3-4c88-b607-7f0f9ec11001\") " pod="openshift-dns/dns-default-fkvjh" Jul 23 11:36:07.980001 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.979944 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/98f50c4a-91e4-4708-be1a-cff80dfb6cc8-serving-cert\") pod \"service-ca-operator-5bcbc94c78-sr6b2\" (UID: \"98f50c4a-91e4-4708-be1a-cff80dfb6cc8\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-sr6b2" Jul 23 11:36:07.980001 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.979987 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21319854-f159-442b-bf88-4498cec86096-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-5txgl\" (UID: \"21319854-f159-442b-bf88-4498cec86096\") " pod="openshift-insights/insights-operator-555d9d7885-5txgl" Jul 23 11:36:07.980260 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.980011 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqqxx\" (UniqueName: \"kubernetes.io/projected/21319854-f159-442b-bf88-4498cec86096-kube-api-access-rqqxx\") pod \"insights-operator-555d9d7885-5txgl\" (UID: \"21319854-f159-442b-bf88-4498cec86096\") " pod="openshift-insights/insights-operator-555d9d7885-5txgl" Jul 23 11:36:07.980260 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.980105 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/65247e1f-fc10-4b40-af67-5e646f352263-trusted-ca\") pod \"console-operator-565b7b569d-7slpj\" (UID: \"65247e1f-fc10-4b40-af67-5e646f352263\") " pod="openshift-console-operator/console-operator-565b7b569d-7slpj" Jul 23 11:36:07.980260 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.980144 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vc6vp\" (UniqueName: \"kubernetes.io/projected/f249368d-c49d-4bfe-906f-d1d12104dbd0-kube-api-access-vc6vp\") pod \"cluster-monitoring-operator-76cf8966f8-j4sbd\" (UID: \"f249368d-c49d-4bfe-906f-d1d12104dbd0\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-j4sbd" Jul 23 11:36:07.980260 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.980170 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b993c3b9-94b8-4c21-b391-6193659cf9ee-trusted-ca\") pod \"image-registry-67fbb677d9-vf9k4\" (UID: \"b993c3b9-94b8-4c21-b391-6193659cf9ee\") " pod="openshift-image-registry/image-registry-67fbb677d9-vf9k4" Jul 23 11:36:07.980260 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.980195 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/21319854-f159-442b-bf88-4498cec86096-serving-cert\") pod \"insights-operator-555d9d7885-5txgl\" (UID: \"21319854-f159-442b-bf88-4498cec86096\") " pod="openshift-insights/insights-operator-555d9d7885-5txgl" Jul 23 11:36:07.980260 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.980228 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zdd4\" (UniqueName: \"kubernetes.io/projected/b64d0632-5844-419e-9f53-ac57a0495a8d-kube-api-access-4zdd4\") pod \"kube-storage-version-migrator-operator-7658d9b749-48j4r\" (UID: \"b64d0632-5844-419e-9f53-ac57a0495a8d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-48j4r" Jul 23 11:36:07.980615 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.980275 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5179eda-de1f-4d89-9e22-afb9d63d098b-metrics-certs\") pod \"router-default-78fdc88b76-5w9ql\" (UID: \"c5179eda-de1f-4d89-9e22-afb9d63d098b\") " pod="openshift-ingress/router-default-78fdc88b76-5w9ql" Jul 23 11:36:07.980615 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.980301 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksdjb\" (UniqueName: \"kubernetes.io/projected/52ccdb9c-03da-48cf-94f4-a916e42975ac-kube-api-access-ksdjb\") pod \"volume-data-source-validator-78cf4c6f6-8gtcv\" (UID: \"52ccdb9c-03da-48cf-94f4-a916e42975ac\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-8gtcv" Jul 23 11:36:07.980615 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.980331 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj2wf\" (UniqueName: \"kubernetes.io/projected/65247e1f-fc10-4b40-af67-5e646f352263-kube-api-access-zj2wf\") pod \"console-operator-565b7b569d-7slpj\" (UID: \"65247e1f-fc10-4b40-af67-5e646f352263\") " pod="openshift-console-operator/console-operator-565b7b569d-7slpj" Jul 23 11:36:07.980615 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.980356 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b993c3b9-94b8-4c21-b391-6193659cf9ee-image-registry-private-configuration\") pod \"image-registry-67fbb677d9-vf9k4\" (UID: \"b993c3b9-94b8-4c21-b391-6193659cf9ee\") " pod="openshift-image-registry/image-registry-67fbb677d9-vf9k4" Jul 23 11:36:07.980615 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.980380 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b64d0632-5844-419e-9f53-ac57a0495a8d-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-48j4r\" (UID: \"b64d0632-5844-419e-9f53-ac57a0495a8d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-48j4r" Jul 23 11:36:07.980615 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.980403 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74038c52-71a3-4c88-b607-7f0f9ec11001-tmp-dir\") pod \"dns-default-fkvjh\" (UID: \"74038c52-71a3-4c88-b607-7f0f9ec11001\") " pod="openshift-dns/dns-default-fkvjh" Jul 23 11:36:07.980615 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.980444 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/89ae8727-bb42-4860-a957-dd9a06933722-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-n2rn6\" (UID: \"89ae8727-bb42-4860-a957-dd9a06933722\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-n2rn6" Jul 23 11:36:07.980615 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.980470 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5179eda-de1f-4d89-9e22-afb9d63d098b-service-ca-bundle\") pod \"router-default-78fdc88b76-5w9ql\" (UID: \"c5179eda-de1f-4d89-9e22-afb9d63d098b\") " pod="openshift-ingress/router-default-78fdc88b76-5w9ql" Jul 23 11:36:07.980615 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.980496 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/c5179eda-de1f-4d89-9e22-afb9d63d098b-default-certificate\") pod \"router-default-78fdc88b76-5w9ql\" (UID: \"c5179eda-de1f-4d89-9e22-afb9d63d098b\") " pod="openshift-ingress/router-default-78fdc88b76-5w9ql" Jul 23 11:36:07.980615 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.980513 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b993c3b9-94b8-4c21-b391-6193659cf9ee-registry-certificates\") pod \"image-registry-67fbb677d9-vf9k4\" (UID: \"b993c3b9-94b8-4c21-b391-6193659cf9ee\") " pod="openshift-image-registry/image-registry-67fbb677d9-vf9k4" Jul 23 11:36:07.980615 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.980520 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/74038c52-71a3-4c88-b607-7f0f9ec11001-config-volume\") pod \"dns-default-fkvjh\" (UID: \"74038c52-71a3-4c88-b607-7f0f9ec11001\") " pod="openshift-dns/dns-default-fkvjh" Jul 23 11:36:07.980615 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.980583 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlkp9\" (UniqueName: \"kubernetes.io/projected/b993c3b9-94b8-4c21-b391-6193659cf9ee-kube-api-access-dlkp9\") pod \"image-registry-67fbb677d9-vf9k4\" (UID: \"b993c3b9-94b8-4c21-b391-6193659cf9ee\") " pod="openshift-image-registry/image-registry-67fbb677d9-vf9k4" Jul 23 11:36:07.980615 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.980610 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b993c3b9-94b8-4c21-b391-6193659cf9ee-installation-pull-secrets\") pod \"image-registry-67fbb677d9-vf9k4\" (UID: \"b993c3b9-94b8-4c21-b391-6193659cf9ee\") " pod="openshift-image-registry/image-registry-67fbb677d9-vf9k4" Jul 23 11:36:07.981253 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.980638 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/98f50c4a-91e4-4708-be1a-cff80dfb6cc8-config\") pod \"service-ca-operator-5bcbc94c78-sr6b2\" (UID: \"98f50c4a-91e4-4708-be1a-cff80dfb6cc8\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-sr6b2" Jul 23 11:36:07.981253 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.980661 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/21319854-f159-442b-bf88-4498cec86096-snapshots\") pod \"insights-operator-555d9d7885-5txgl\" (UID: \"21319854-f159-442b-bf88-4498cec86096\") " pod="openshift-insights/insights-operator-555d9d7885-5txgl" Jul 23 11:36:07.981253 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:07.980682 2561 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 23 11:36:07.981253 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.980696 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch96j\" (UniqueName: \"kubernetes.io/projected/89ae8727-bb42-4860-a957-dd9a06933722-kube-api-access-ch96j\") pod \"cluster-samples-operator-79b7c55f87-n2rn6\" (UID: \"89ae8727-bb42-4860-a957-dd9a06933722\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-n2rn6" Jul 23 11:36:07.981253 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.980725 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szfm8\" (UniqueName: \"kubernetes.io/projected/0ceda81a-92dd-4858-b012-f05506292db4-kube-api-access-szfm8\") pod \"network-check-source-7bc5b6bf85-5ss74\" (UID: \"0ceda81a-92dd-4858-b012-f05506292db4\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-5ss74" Jul 23 11:36:07.981253 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:07.980737 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5179eda-de1f-4d89-9e22-afb9d63d098b-metrics-certs podName:c5179eda-de1f-4d89-9e22-afb9d63d098b nodeName:}" failed. No retries permitted until 2026-07-23 11:36:08.480721459 +0000 UTC m=+35.552411427 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c5179eda-de1f-4d89-9e22-afb9d63d098b-metrics-certs") pod "router-default-78fdc88b76-5w9ql" (UID: "c5179eda-de1f-4d89-9e22-afb9d63d098b") : secret "router-metrics-certs-default" not found Jul 23 11:36:07.981253 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.980767 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/65247e1f-fc10-4b40-af67-5e646f352263-serving-cert\") pod \"console-operator-565b7b569d-7slpj\" (UID: \"65247e1f-fc10-4b40-af67-5e646f352263\") " pod="openshift-console-operator/console-operator-565b7b569d-7slpj" Jul 23 11:36:07.981253 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.980792 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b993c3b9-94b8-4c21-b391-6193659cf9ee-bound-sa-token\") pod \"image-registry-67fbb677d9-vf9k4\" (UID: \"b993c3b9-94b8-4c21-b391-6193659cf9ee\") " pod="openshift-image-registry/image-registry-67fbb677d9-vf9k4" Jul 23 11:36:07.981253 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.980816 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a46e320d-ae78-4a40-9314-075f03b0b6ce-cert\") pod \"ingress-canary-fww69\" (UID: \"a46e320d-ae78-4a40-9314-075f03b0b6ce\") " pod="openshift-ingress-canary/ingress-canary-fww69" Jul 23 11:36:07.981253 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.980841 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b993c3b9-94b8-4c21-b391-6193659cf9ee-registry-tls\") pod \"image-registry-67fbb677d9-vf9k4\" (UID: \"b993c3b9-94b8-4c21-b391-6193659cf9ee\") " pod="openshift-image-registry/image-registry-67fbb677d9-vf9k4" Jul 23 11:36:07.981253 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:07.980865 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c5179eda-de1f-4d89-9e22-afb9d63d098b-service-ca-bundle podName:c5179eda-de1f-4d89-9e22-afb9d63d098b nodeName:}" failed. No retries permitted until 2026-07-23 11:36:08.480850121 +0000 UTC m=+35.552540112 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c5179eda-de1f-4d89-9e22-afb9d63d098b-service-ca-bundle") pod "router-default-78fdc88b76-5w9ql" (UID: "c5179eda-de1f-4d89-9e22-afb9d63d098b") : configmap references non-existent config key: service-ca.crt Jul 23 11:36:07.981253 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:07.980898 2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 11:36:07.981253 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:07.980926 2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67fbb677d9-vf9k4: secret "image-registry-tls" not found Jul 23 11:36:07.981253 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.980900 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21319854-f159-442b-bf88-4498cec86096-service-ca-bundle\") pod \"insights-operator-555d9d7885-5txgl\" (UID: \"21319854-f159-442b-bf88-4498cec86096\") " pod="openshift-insights/insights-operator-555d9d7885-5txgl" Jul 23 11:36:07.981253 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.980956 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwdp7\" (UniqueName: \"kubernetes.io/projected/98f50c4a-91e4-4708-be1a-cff80dfb6cc8-kube-api-access-bwdp7\") pod \"service-ca-operator-5bcbc94c78-sr6b2\" (UID: \"98f50c4a-91e4-4708-be1a-cff80dfb6cc8\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-sr6b2" Jul 23 11:36:07.981253 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.980986 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/65247e1f-fc10-4b40-af67-5e646f352263-config\") pod \"console-operator-565b7b569d-7slpj\" (UID: \"65247e1f-fc10-4b40-af67-5e646f352263\") " pod="openshift-console-operator/console-operator-565b7b569d-7slpj" Jul 23 11:36:07.984097 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.981016 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f249368d-c49d-4bfe-906f-d1d12104dbd0-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-j4sbd\" (UID: \"f249368d-c49d-4bfe-906f-d1d12104dbd0\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-j4sbd" Jul 23 11:36:07.984097 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.981030 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/65247e1f-fc10-4b40-af67-5e646f352263-trusted-ca\") pod \"console-operator-565b7b569d-7slpj\" (UID: \"65247e1f-fc10-4b40-af67-5e646f352263\") " pod="openshift-console-operator/console-operator-565b7b569d-7slpj" Jul 23 11:36:07.984097 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.981043 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/21319854-f159-442b-bf88-4498cec86096-tmp\") pod \"insights-operator-555d9d7885-5txgl\" (UID: \"21319854-f159-442b-bf88-4498cec86096\") " pod="openshift-insights/insights-operator-555d9d7885-5txgl" Jul 23 11:36:07.984097 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.981083 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/c5179eda-de1f-4d89-9e22-afb9d63d098b-stats-auth\") pod \"router-default-78fdc88b76-5w9ql\" (UID: \"c5179eda-de1f-4d89-9e22-afb9d63d098b\") " pod="openshift-ingress/router-default-78fdc88b76-5w9ql" Jul 23 11:36:07.984097 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:07.981154 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b993c3b9-94b8-4c21-b391-6193659cf9ee-registry-tls podName:b993c3b9-94b8-4c21-b391-6193659cf9ee nodeName:}" failed. No retries permitted until 2026-07-23 11:36:08.481140585 +0000 UTC m=+35.552830560 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b993c3b9-94b8-4c21-b391-6193659cf9ee-registry-tls") pod "image-registry-67fbb677d9-vf9k4" (UID: "b993c3b9-94b8-4c21-b391-6193659cf9ee") : secret "image-registry-tls" not found Jul 23 11:36:07.984097 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.981183 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zth9j\" (UniqueName: \"kubernetes.io/projected/c5179eda-de1f-4d89-9e22-afb9d63d098b-kube-api-access-zth9j\") pod \"router-default-78fdc88b76-5w9ql\" (UID: \"c5179eda-de1f-4d89-9e22-afb9d63d098b\") " pod="openshift-ingress/router-default-78fdc88b76-5w9ql" Jul 23 11:36:07.984097 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.981220 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b993c3b9-94b8-4c21-b391-6193659cf9ee-ca-trust-extracted\") pod \"image-registry-67fbb677d9-vf9k4\" (UID: \"b993c3b9-94b8-4c21-b391-6193659cf9ee\") " pod="openshift-image-registry/image-registry-67fbb677d9-vf9k4" Jul 23 11:36:07.984097 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.981248 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbvpg\" (UniqueName: \"kubernetes.io/projected/a46e320d-ae78-4a40-9314-075f03b0b6ce-kube-api-access-pbvpg\") pod \"ingress-canary-fww69\" (UID: \"a46e320d-ae78-4a40-9314-075f03b0b6ce\") " pod="openshift-ingress-canary/ingress-canary-fww69" Jul 23 11:36:07.984097 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.981259 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b993c3b9-94b8-4c21-b391-6193659cf9ee-trusted-ca\") pod \"image-registry-67fbb677d9-vf9k4\" (UID: \"b993c3b9-94b8-4c21-b391-6193659cf9ee\") " pod="openshift-image-registry/image-registry-67fbb677d9-vf9k4" Jul 23 11:36:07.984097 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.981276 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f249368d-c49d-4bfe-906f-d1d12104dbd0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-j4sbd\" (UID: \"f249368d-c49d-4bfe-906f-d1d12104dbd0\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-j4sbd" Jul 23 11:36:07.984097 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:07.981378 2561 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 23 11:36:07.984097 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:07.981421 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f249368d-c49d-4bfe-906f-d1d12104dbd0-cluster-monitoring-operator-tls podName:f249368d-c49d-4bfe-906f-d1d12104dbd0 nodeName:}" failed. No retries permitted until 2026-07-23 11:36:08.481409294 +0000 UTC m=+35.553099282 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f249368d-c49d-4bfe-906f-d1d12104dbd0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-j4sbd" (UID: "f249368d-c49d-4bfe-906f-d1d12104dbd0") : secret "cluster-monitoring-operator-tls" not found Jul 23 11:36:07.984097 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.981653 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lp5f4_620c5e1f-97de-42e6-b752-50d951728a5d/konnectivity-agent/0.log" limit=1024 Jul 23 11:36:07.984097 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.982050 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b993c3b9-94b8-4c21-b391-6193659cf9ee-ca-trust-extracted\") pod \"image-registry-67fbb677d9-vf9k4\" (UID: \"b993c3b9-94b8-4c21-b391-6193659cf9ee\") " pod="openshift-image-registry/image-registry-67fbb677d9-vf9k4" Jul 23 11:36:07.984097 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.982337 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b64d0632-5844-419e-9f53-ac57a0495a8d-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-48j4r\" (UID: \"b64d0632-5844-419e-9f53-ac57a0495a8d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-48j4r" Jul 23 11:36:07.984661 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.982460 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/65247e1f-fc10-4b40-af67-5e646f352263-config\") pod \"console-operator-565b7b569d-7slpj\" (UID: \"65247e1f-fc10-4b40-af67-5e646f352263\") " pod="openshift-console-operator/console-operator-565b7b569d-7slpj" Jul 23 11:36:07.984661 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.983619 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f249368d-c49d-4bfe-906f-d1d12104dbd0-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-j4sbd\" (UID: \"f249368d-c49d-4bfe-906f-d1d12104dbd0\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-j4sbd" Jul 23 11:36:07.985773 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.985732 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/65247e1f-fc10-4b40-af67-5e646f352263-serving-cert\") pod \"console-operator-565b7b569d-7slpj\" (UID: \"65247e1f-fc10-4b40-af67-5e646f352263\") " pod="openshift-console-operator/console-operator-565b7b569d-7slpj" Jul 23 11:36:07.985850 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.985819 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b993c3b9-94b8-4c21-b391-6193659cf9ee-installation-pull-secrets\") pod \"image-registry-67fbb677d9-vf9k4\" (UID: \"b993c3b9-94b8-4c21-b391-6193659cf9ee\") " pod="openshift-image-registry/image-registry-67fbb677d9-vf9k4" Jul 23 11:36:07.985850 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.985838 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b993c3b9-94b8-4c21-b391-6193659cf9ee-image-registry-private-configuration\") pod \"image-registry-67fbb677d9-vf9k4\" (UID: \"b993c3b9-94b8-4c21-b391-6193659cf9ee\") " pod="openshift-image-registry/image-registry-67fbb677d9-vf9k4" Jul 23 11:36:07.986051 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.985877 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/c5179eda-de1f-4d89-9e22-afb9d63d098b-default-certificate\") pod \"router-default-78fdc88b76-5w9ql\" (UID: \"c5179eda-de1f-4d89-9e22-afb9d63d098b\") " pod="openshift-ingress/router-default-78fdc88b76-5w9ql" Jul 23 11:36:07.986051 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.985895 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/c5179eda-de1f-4d89-9e22-afb9d63d098b-stats-auth\") pod \"router-default-78fdc88b76-5w9ql\" (UID: \"c5179eda-de1f-4d89-9e22-afb9d63d098b\") " pod="openshift-ingress/router-default-78fdc88b76-5w9ql" Jul 23 11:36:07.987154 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.987129 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b64d0632-5844-419e-9f53-ac57a0495a8d-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-48j4r\" (UID: \"b64d0632-5844-419e-9f53-ac57a0495a8d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-48j4r" Jul 23 11:36:07.991067 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.990935 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vc6vp\" (UniqueName: \"kubernetes.io/projected/f249368d-c49d-4bfe-906f-d1d12104dbd0-kube-api-access-vc6vp\") pod \"cluster-monitoring-operator-76cf8966f8-j4sbd\" (UID: \"f249368d-c49d-4bfe-906f-d1d12104dbd0\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-j4sbd" Jul 23 11:36:07.991301 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.991279 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj2wf\" (UniqueName: \"kubernetes.io/projected/65247e1f-fc10-4b40-af67-5e646f352263-kube-api-access-zj2wf\") pod \"console-operator-565b7b569d-7slpj\" (UID: \"65247e1f-fc10-4b40-af67-5e646f352263\") " pod="openshift-console-operator/console-operator-565b7b569d-7slpj" Jul 23 11:36:07.991405 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.991371 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zdd4\" (UniqueName: \"kubernetes.io/projected/b64d0632-5844-419e-9f53-ac57a0495a8d-kube-api-access-4zdd4\") pod \"kube-storage-version-migrator-operator-7658d9b749-48j4r\" (UID: \"b64d0632-5844-419e-9f53-ac57a0495a8d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-48j4r" Jul 23 11:36:07.992049 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.991997 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zth9j\" (UniqueName: \"kubernetes.io/projected/c5179eda-de1f-4d89-9e22-afb9d63d098b-kube-api-access-zth9j\") pod \"router-default-78fdc88b76-5w9ql\" (UID: \"c5179eda-de1f-4d89-9e22-afb9d63d098b\") " pod="openshift-ingress/router-default-78fdc88b76-5w9ql" Jul 23 11:36:07.992166 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.992145 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlkp9\" (UniqueName: \"kubernetes.io/projected/b993c3b9-94b8-4c21-b391-6193659cf9ee-kube-api-access-dlkp9\") pod \"image-registry-67fbb677d9-vf9k4\" (UID: \"b993c3b9-94b8-4c21-b391-6193659cf9ee\") " pod="openshift-image-registry/image-registry-67fbb677d9-vf9k4" Jul 23 11:36:07.992721 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:07.992688 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b993c3b9-94b8-4c21-b391-6193659cf9ee-bound-sa-token\") pod \"image-registry-67fbb677d9-vf9k4\" (UID: \"b993c3b9-94b8-4c21-b391-6193659cf9ee\") " pod="openshift-image-registry/image-registry-67fbb677d9-vf9k4" Jul 23 11:36:08.082113 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:08.082071 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbvpg\" (UniqueName: \"kubernetes.io/projected/a46e320d-ae78-4a40-9314-075f03b0b6ce-kube-api-access-pbvpg\") pod \"ingress-canary-fww69\" (UID: \"a46e320d-ae78-4a40-9314-075f03b0b6ce\") " pod="openshift-ingress-canary/ingress-canary-fww69" Jul 23 11:36:08.082836 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:08.082126 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htc4j\" (UniqueName: \"kubernetes.io/projected/74038c52-71a3-4c88-b607-7f0f9ec11001-kube-api-access-htc4j\") pod \"dns-default-fkvjh\" (UID: \"74038c52-71a3-4c88-b607-7f0f9ec11001\") " pod="openshift-dns/dns-default-fkvjh" Jul 23 11:36:08.082836 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:08.082156 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/74038c52-71a3-4c88-b607-7f0f9ec11001-metrics-tls\") pod \"dns-default-fkvjh\" (UID: \"74038c52-71a3-4c88-b607-7f0f9ec11001\") " pod="openshift-dns/dns-default-fkvjh" Jul 23 11:36:08.082836 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:08.082183 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/98f50c4a-91e4-4708-be1a-cff80dfb6cc8-serving-cert\") pod \"service-ca-operator-5bcbc94c78-sr6b2\" (UID: \"98f50c4a-91e4-4708-be1a-cff80dfb6cc8\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-sr6b2" Jul 23 11:36:08.082836 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:08.082208 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21319854-f159-442b-bf88-4498cec86096-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-5txgl\" (UID: \"21319854-f159-442b-bf88-4498cec86096\") " pod="openshift-insights/insights-operator-555d9d7885-5txgl" Jul 23 11:36:08.082836 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:08.082233 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqqxx\" (UniqueName: \"kubernetes.io/projected/21319854-f159-442b-bf88-4498cec86096-kube-api-access-rqqxx\") pod \"insights-operator-555d9d7885-5txgl\" (UID: \"21319854-f159-442b-bf88-4498cec86096\") " pod="openshift-insights/insights-operator-555d9d7885-5txgl" Jul 23 11:36:08.082836 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:08.082300 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/21319854-f159-442b-bf88-4498cec86096-serving-cert\") pod \"insights-operator-555d9d7885-5txgl\" (UID: \"21319854-f159-442b-bf88-4498cec86096\") " pod="openshift-insights/insights-operator-555d9d7885-5txgl" Jul 23 11:36:08.082836 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:08.082351 2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 11:36:08.082836 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:08.082395 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksdjb\" (UniqueName: \"kubernetes.io/projected/52ccdb9c-03da-48cf-94f4-a916e42975ac-kube-api-access-ksdjb\") pod \"volume-data-source-validator-78cf4c6f6-8gtcv\" (UID: \"52ccdb9c-03da-48cf-94f4-a916e42975ac\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-8gtcv" Jul 23 11:36:08.082836 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:08.082431 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74038c52-71a3-4c88-b607-7f0f9ec11001-metrics-tls podName:74038c52-71a3-4c88-b607-7f0f9ec11001 nodeName:}" failed. No retries permitted until 2026-07-23 11:36:08.582411504 +0000 UTC m=+35.654101491 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/74038c52-71a3-4c88-b607-7f0f9ec11001-metrics-tls") pod "dns-default-fkvjh" (UID: "74038c52-71a3-4c88-b607-7f0f9ec11001") : secret "dns-default-metrics-tls" not found Jul 23 11:36:08.082836 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:08.082467 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74038c52-71a3-4c88-b607-7f0f9ec11001-tmp-dir\") pod \"dns-default-fkvjh\" (UID: \"74038c52-71a3-4c88-b607-7f0f9ec11001\") " pod="openshift-dns/dns-default-fkvjh" Jul 23 11:36:08.082836 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:08.082492 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/89ae8727-bb42-4860-a957-dd9a06933722-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-n2rn6\" (UID: \"89ae8727-bb42-4860-a957-dd9a06933722\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-n2rn6" Jul 23 11:36:08.082836 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:08.082528 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/74038c52-71a3-4c88-b607-7f0f9ec11001-config-volume\") pod \"dns-default-fkvjh\" (UID: \"74038c52-71a3-4c88-b607-7f0f9ec11001\") " pod="openshift-dns/dns-default-fkvjh" Jul 23 11:36:08.082836 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:08.082568 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/98f50c4a-91e4-4708-be1a-cff80dfb6cc8-config\") pod \"service-ca-operator-5bcbc94c78-sr6b2\" (UID: \"98f50c4a-91e4-4708-be1a-cff80dfb6cc8\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-sr6b2" Jul 23 11:36:08.082836 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:08.082592 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/21319854-f159-442b-bf88-4498cec86096-snapshots\") pod \"insights-operator-555d9d7885-5txgl\" (UID: \"21319854-f159-442b-bf88-4498cec86096\") " pod="openshift-insights/insights-operator-555d9d7885-5txgl" Jul 23 11:36:08.082836 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:08.082623 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch96j\" (UniqueName: \"kubernetes.io/projected/89ae8727-bb42-4860-a957-dd9a06933722-kube-api-access-ch96j\") pod \"cluster-samples-operator-79b7c55f87-n2rn6\" (UID: \"89ae8727-bb42-4860-a957-dd9a06933722\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-n2rn6" Jul 23 11:36:08.082836 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:08.082631 2561 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 23 11:36:08.082836 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:08.082650 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szfm8\" (UniqueName: \"kubernetes.io/projected/0ceda81a-92dd-4858-b012-f05506292db4-kube-api-access-szfm8\") pod \"network-check-source-7bc5b6bf85-5ss74\" (UID: \"0ceda81a-92dd-4858-b012-f05506292db4\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-5ss74" Jul 23 11:36:08.083689 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:08.082682 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a46e320d-ae78-4a40-9314-075f03b0b6ce-cert\") pod \"ingress-canary-fww69\" (UID: \"a46e320d-ae78-4a40-9314-075f03b0b6ce\") " pod="openshift-ingress-canary/ingress-canary-fww69" Jul 23 11:36:08.083689 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:08.082724 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21319854-f159-442b-bf88-4498cec86096-service-ca-bundle\") pod \"insights-operator-555d9d7885-5txgl\" (UID: \"21319854-f159-442b-bf88-4498cec86096\") " pod="openshift-insights/insights-operator-555d9d7885-5txgl" Jul 23 11:36:08.083689 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:08.082750 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwdp7\" (UniqueName: \"kubernetes.io/projected/98f50c4a-91e4-4708-be1a-cff80dfb6cc8-kube-api-access-bwdp7\") pod \"service-ca-operator-5bcbc94c78-sr6b2\" (UID: \"98f50c4a-91e4-4708-be1a-cff80dfb6cc8\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-sr6b2" Jul 23 11:36:08.083689 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:08.082783 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/21319854-f159-442b-bf88-4498cec86096-tmp\") pod \"insights-operator-555d9d7885-5txgl\" (UID: \"21319854-f159-442b-bf88-4498cec86096\") " pod="openshift-insights/insights-operator-555d9d7885-5txgl" Jul 23 11:36:08.083689 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:08.083024 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74038c52-71a3-4c88-b607-7f0f9ec11001-tmp-dir\") pod \"dns-default-fkvjh\" (UID: \"74038c52-71a3-4c88-b607-7f0f9ec11001\") " pod="openshift-dns/dns-default-fkvjh" Jul 23 11:36:08.083689 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:08.083163 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21319854-f159-442b-bf88-4498cec86096-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-5txgl\" (UID: \"21319854-f159-442b-bf88-4498cec86096\") " pod="openshift-insights/insights-operator-555d9d7885-5txgl" Jul 23 11:36:08.083689 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:08.083240 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/21319854-f159-442b-bf88-4498cec86096-tmp\") pod \"insights-operator-555d9d7885-5txgl\" (UID: \"21319854-f159-442b-bf88-4498cec86096\") " pod="openshift-insights/insights-operator-555d9d7885-5txgl" Jul 23 11:36:08.083689 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:08.083253 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/74038c52-71a3-4c88-b607-7f0f9ec11001-config-volume\") pod \"dns-default-fkvjh\" (UID: \"74038c52-71a3-4c88-b607-7f0f9ec11001\") " pod="openshift-dns/dns-default-fkvjh" Jul 23 11:36:08.083689 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:08.083257 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/21319854-f159-442b-bf88-4498cec86096-snapshots\") pod \"insights-operator-555d9d7885-5txgl\" (UID: \"21319854-f159-442b-bf88-4498cec86096\") " pod="openshift-insights/insights-operator-555d9d7885-5txgl" Jul 23 11:36:08.083689 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:08.083313 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89ae8727-bb42-4860-a957-dd9a06933722-samples-operator-tls podName:89ae8727-bb42-4860-a957-dd9a06933722 nodeName:}" failed. No retries permitted until 2026-07-23 11:36:08.583296927 +0000 UTC m=+35.654986897 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/89ae8727-bb42-4860-a957-dd9a06933722-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-n2rn6" (UID: "89ae8727-bb42-4860-a957-dd9a06933722") : secret "samples-operator-tls" not found Jul 23 11:36:08.084124 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:08.083695 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21319854-f159-442b-bf88-4498cec86096-service-ca-bundle\") pod \"insights-operator-555d9d7885-5txgl\" (UID: \"21319854-f159-442b-bf88-4498cec86096\") " pod="openshift-insights/insights-operator-555d9d7885-5txgl" Jul 23 11:36:08.084124 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:08.083757 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/98f50c4a-91e4-4708-be1a-cff80dfb6cc8-config\") pod \"service-ca-operator-5bcbc94c78-sr6b2\" (UID: \"98f50c4a-91e4-4708-be1a-cff80dfb6cc8\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-sr6b2" Jul 23 11:36:08.085729 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:08.085174 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/98f50c4a-91e4-4708-be1a-cff80dfb6cc8-serving-cert\") pod \"service-ca-operator-5bcbc94c78-sr6b2\" (UID: \"98f50c4a-91e4-4708-be1a-cff80dfb6cc8\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-sr6b2" Jul 23 11:36:08.085729 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:08.085300 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/21319854-f159-442b-bf88-4498cec86096-serving-cert\") pod \"insights-operator-555d9d7885-5txgl\" (UID: \"21319854-f159-442b-bf88-4498cec86096\") " pod="openshift-insights/insights-operator-555d9d7885-5txgl" Jul 23 11:36:08.085729 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:08.085694 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a46e320d-ae78-4a40-9314-075f03b0b6ce-cert\") pod \"ingress-canary-fww69\" (UID: \"a46e320d-ae78-4a40-9314-075f03b0b6ce\") " pod="openshift-ingress-canary/ingress-canary-fww69" Jul 23 11:36:08.088119 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:08.088099 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-7slpj" Jul 23 11:36:08.092974 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:08.092951 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwdp7\" (UniqueName: \"kubernetes.io/projected/98f50c4a-91e4-4708-be1a-cff80dfb6cc8-kube-api-access-bwdp7\") pod \"service-ca-operator-5bcbc94c78-sr6b2\" (UID: \"98f50c4a-91e4-4708-be1a-cff80dfb6cc8\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-sr6b2" Jul 23 11:36:08.093296 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:08.093275 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch96j\" (UniqueName: \"kubernetes.io/projected/89ae8727-bb42-4860-a957-dd9a06933722-kube-api-access-ch96j\") pod \"cluster-samples-operator-79b7c55f87-n2rn6\" (UID: \"89ae8727-bb42-4860-a957-dd9a06933722\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-n2rn6" Jul 23 11:36:08.094240 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:08.094188 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-htc4j\" (UniqueName: \"kubernetes.io/projected/74038c52-71a3-4c88-b607-7f0f9ec11001-kube-api-access-htc4j\") pod \"dns-default-fkvjh\" (UID: \"74038c52-71a3-4c88-b607-7f0f9ec11001\") " pod="openshift-dns/dns-default-fkvjh" Jul 23 11:36:08.094337 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:08.094270 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqqxx\" (UniqueName: \"kubernetes.io/projected/21319854-f159-442b-bf88-4498cec86096-kube-api-access-rqqxx\") pod \"insights-operator-555d9d7885-5txgl\" (UID: \"21319854-f159-442b-bf88-4498cec86096\") " pod="openshift-insights/insights-operator-555d9d7885-5txgl" Jul 23 11:36:08.094401 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:08.094370 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksdjb\" (UniqueName: \"kubernetes.io/projected/52ccdb9c-03da-48cf-94f4-a916e42975ac-kube-api-access-ksdjb\") pod \"volume-data-source-validator-78cf4c6f6-8gtcv\" (UID: \"52ccdb9c-03da-48cf-94f4-a916e42975ac\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-8gtcv" Jul 23 11:36:08.094401 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:08.094382 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbvpg\" (UniqueName: \"kubernetes.io/projected/a46e320d-ae78-4a40-9314-075f03b0b6ce-kube-api-access-pbvpg\") pod \"ingress-canary-fww69\" (UID: \"a46e320d-ae78-4a40-9314-075f03b0b6ce\") " pod="openshift-ingress-canary/ingress-canary-fww69" Jul 23 11:36:08.094525 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:08.094499 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-szfm8\" (UniqueName: \"kubernetes.io/projected/0ceda81a-92dd-4858-b012-f05506292db4-kube-api-access-szfm8\") pod \"network-check-source-7bc5b6bf85-5ss74\" (UID: \"0ceda81a-92dd-4858-b012-f05506292db4\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-5ss74" Jul 23 11:36:08.101634 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:08.101551 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lp5f4_620c5e1f-97de-42e6-b752-50d951728a5d/konnectivity-agent/0.log" limit=1024 Jul 23 11:36:08.115733 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:08.115713 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-48j4r" Jul 23 11:36:08.153702 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:08.153680 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-5ss74" Jul 23 11:36:08.160545 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:08.160522 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fww69" Jul 23 11:36:08.168184 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:08.168164 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-5txgl" Jul 23 11:36:08.174882 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:08.174861 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-8gtcv" Jul 23 11:36:08.183647 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:08.183628 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-sr6b2" Jul 23 11:36:08.449213 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:08.449181 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8df69" Jul 23 11:36:08.451879 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:08.451855 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 23 11:36:08.485663 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:08.485641 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f249368d-c49d-4bfe-906f-d1d12104dbd0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-j4sbd\" (UID: \"f249368d-c49d-4bfe-906f-d1d12104dbd0\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-j4sbd" Jul 23 11:36:08.485779 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:08.485702 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5179eda-de1f-4d89-9e22-afb9d63d098b-metrics-certs\") pod \"router-default-78fdc88b76-5w9ql\" (UID: \"c5179eda-de1f-4d89-9e22-afb9d63d098b\") " pod="openshift-ingress/router-default-78fdc88b76-5w9ql" Jul 23 11:36:08.485779 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:08.485739 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5179eda-de1f-4d89-9e22-afb9d63d098b-service-ca-bundle\") pod \"router-default-78fdc88b76-5w9ql\" (UID: \"c5179eda-de1f-4d89-9e22-afb9d63d098b\") " pod="openshift-ingress/router-default-78fdc88b76-5w9ql" Jul 23 11:36:08.485887 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:08.485792 2561 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 23 11:36:08.485887 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:08.485799 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b993c3b9-94b8-4c21-b391-6193659cf9ee-registry-tls\") pod \"image-registry-67fbb677d9-vf9k4\" (UID: \"b993c3b9-94b8-4c21-b391-6193659cf9ee\") " pod="openshift-image-registry/image-registry-67fbb677d9-vf9k4" Jul 23 11:36:08.485887 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:08.485860 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f249368d-c49d-4bfe-906f-d1d12104dbd0-cluster-monitoring-operator-tls podName:f249368d-c49d-4bfe-906f-d1d12104dbd0 nodeName:}" failed. No retries permitted until 2026-07-23 11:36:09.485839355 +0000 UTC m=+36.557529340 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f249368d-c49d-4bfe-906f-d1d12104dbd0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-j4sbd" (UID: "f249368d-c49d-4bfe-906f-d1d12104dbd0") : secret "cluster-monitoring-operator-tls" not found Jul 23 11:36:08.485887 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:08.485863 2561 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 23 11:36:08.486067 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:08.485890 2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 11:36:08.486067 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:08.485893 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c5179eda-de1f-4d89-9e22-afb9d63d098b-service-ca-bundle podName:c5179eda-de1f-4d89-9e22-afb9d63d098b nodeName:}" failed. No retries permitted until 2026-07-23 11:36:09.48587523 +0000 UTC m=+36.557565212 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c5179eda-de1f-4d89-9e22-afb9d63d098b-service-ca-bundle") pod "router-default-78fdc88b76-5w9ql" (UID: "c5179eda-de1f-4d89-9e22-afb9d63d098b") : configmap references non-existent config key: service-ca.crt Jul 23 11:36:08.486067 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:08.485904 2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67fbb677d9-vf9k4: secret "image-registry-tls" not found Jul 23 11:36:08.489501 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:08.486298 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5179eda-de1f-4d89-9e22-afb9d63d098b-metrics-certs podName:c5179eda-de1f-4d89-9e22-afb9d63d098b nodeName:}" failed. No retries permitted until 2026-07-23 11:36:09.485928875 +0000 UTC m=+36.557618848 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c5179eda-de1f-4d89-9e22-afb9d63d098b-metrics-certs") pod "router-default-78fdc88b76-5w9ql" (UID: "c5179eda-de1f-4d89-9e22-afb9d63d098b") : secret "router-metrics-certs-default" not found Jul 23 11:36:08.489501 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:08.486450 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b993c3b9-94b8-4c21-b391-6193659cf9ee-registry-tls podName:b993c3b9-94b8-4c21-b391-6193659cf9ee nodeName:}" failed. No retries permitted until 2026-07-23 11:36:09.486430125 +0000 UTC m=+36.558120119 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b993c3b9-94b8-4c21-b391-6193659cf9ee-registry-tls") pod "image-registry-67fbb677d9-vf9k4" (UID: "b993c3b9-94b8-4c21-b391-6193659cf9ee") : secret "image-registry-tls" not found Jul 23 11:36:08.516767 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:08.516733 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lp5f4_620c5e1f-97de-42e6-b752-50d951728a5d/konnectivity-agent/0.log" limit=1024 Jul 23 11:36:08.586699 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:08.586670 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/74038c52-71a3-4c88-b607-7f0f9ec11001-metrics-tls\") pod \"dns-default-fkvjh\" (UID: \"74038c52-71a3-4c88-b607-7f0f9ec11001\") " pod="openshift-dns/dns-default-fkvjh" Jul 23 11:36:08.586830 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:08.586736 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/89ae8727-bb42-4860-a957-dd9a06933722-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-n2rn6\" (UID: \"89ae8727-bb42-4860-a957-dd9a06933722\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-n2rn6" Jul 23 11:36:08.586898 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:08.586844 2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 11:36:08.586898 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:08.586880 2561 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 23 11:36:08.586998 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:08.586961 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89ae8727-bb42-4860-a957-dd9a06933722-samples-operator-tls podName:89ae8727-bb42-4860-a957-dd9a06933722 nodeName:}" failed. No retries permitted until 2026-07-23 11:36:09.586939559 +0000 UTC m=+36.658629549 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/89ae8727-bb42-4860-a957-dd9a06933722-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-n2rn6" (UID: "89ae8727-bb42-4860-a957-dd9a06933722") : secret "samples-operator-tls" not found Jul 23 11:36:08.586998 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:08.586981 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74038c52-71a3-4c88-b607-7f0f9ec11001-metrics-tls podName:74038c52-71a3-4c88-b607-7f0f9ec11001 nodeName:}" failed. No retries permitted until 2026-07-23 11:36:09.586971347 +0000 UTC m=+36.658661319 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/74038c52-71a3-4c88-b607-7f0f9ec11001-metrics-tls") pod "dns-default-fkvjh" (UID: "74038c52-71a3-4c88-b607-7f0f9ec11001") : secret "dns-default-metrics-tls" not found Jul 23 11:36:08.617120 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:08.617088 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lp5f4_620c5e1f-97de-42e6-b752-50d951728a5d/konnectivity-agent/0.log" limit=1024 Jul 23 11:36:08.993093 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:08.992833 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-7slpj"] Jul 23 11:36:08.998680 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:08.997273 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-8gtcv"] Jul 23 11:36:09.000534 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:09.000486 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fww69"] Jul 23 11:36:09.001845 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:09.001820 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-5ss74"] Jul 23 11:36:09.002612 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:09.002594 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-sr6b2"] Jul 23 11:36:09.003343 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:09.003326 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-5txgl"] Jul 23 11:36:09.019201 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:09.019180 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-48j4r"] Jul 23 11:36:09.024414 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:36:09.024381 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda46e320d_ae78_4a40_9314_075f03b0b6ce.slice/crio-58d14782974286f3e996729900b3afd38f8d70efc01cf110e47541a7d2d20c15 WatchSource:0}: Error finding container 58d14782974286f3e996729900b3afd38f8d70efc01cf110e47541a7d2d20c15: Status 404 returned error can't find the container with id 58d14782974286f3e996729900b3afd38f8d70efc01cf110e47541a7d2d20c15 Jul 23 11:36:09.025476 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:36:09.025439 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52ccdb9c_03da_48cf_94f4_a916e42975ac.slice/crio-c62bab0f834f0930d218048a1f9856d70857706a15648044fa4f8882cdadfc2e WatchSource:0}: Error finding container c62bab0f834f0930d218048a1f9856d70857706a15648044fa4f8882cdadfc2e: Status 404 returned error can't find the container with id c62bab0f834f0930d218048a1f9856d70857706a15648044fa4f8882cdadfc2e Jul 23 11:36:09.026298 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:36:09.026261 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ceda81a_92dd_4858_b012_f05506292db4.slice/crio-bc53eb84f0ac35676c2b4f2ee8a3f5bfdcb052aa1d75a0b5060ea48141e29474 WatchSource:0}: Error finding container bc53eb84f0ac35676c2b4f2ee8a3f5bfdcb052aa1d75a0b5060ea48141e29474: Status 404 returned error can't find the container with id bc53eb84f0ac35676c2b4f2ee8a3f5bfdcb052aa1d75a0b5060ea48141e29474 Jul 23 11:36:09.027567 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:36:09.026975 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98f50c4a_91e4_4708_be1a_cff80dfb6cc8.slice/crio-3f29cb7515f2aaaf99cdb291a4271d3ba0caa1dc606a71bf8dbdf260b379db82 WatchSource:0}: Error finding container 3f29cb7515f2aaaf99cdb291a4271d3ba0caa1dc606a71bf8dbdf260b379db82: Status 404 returned error can't find the container with id 3f29cb7515f2aaaf99cdb291a4271d3ba0caa1dc606a71bf8dbdf260b379db82 Jul 23 11:36:09.027997 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:36:09.027971 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65247e1f_fc10_4b40_af67_5e646f352263.slice/crio-647b0e75f9bff8fd4e1ddddc44d621277cbfa817573af7c771a63c28bac722a3 WatchSource:0}: Error finding container 647b0e75f9bff8fd4e1ddddc44d621277cbfa817573af7c771a63c28bac722a3: Status 404 returned error can't find the container with id 647b0e75f9bff8fd4e1ddddc44d621277cbfa817573af7c771a63c28bac722a3 Jul 23 11:36:09.031488 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:36:09.031464 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21319854_f159_442b_bf88_4498cec86096.slice/crio-5d63a6b004cd6e174455111d66a6a56ec11df7a3ad362035ddb792d40278fa69 WatchSource:0}: Error finding container 5d63a6b004cd6e174455111d66a6a56ec11df7a3ad362035ddb792d40278fa69: Status 404 returned error can't find the container with id 5d63a6b004cd6e174455111d66a6a56ec11df7a3ad362035ddb792d40278fa69 Jul 23 11:36:09.034146 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:36:09.033570 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb64d0632_5844_419e_9f53_ac57a0495a8d.slice/crio-39a30956a50d3c5fbf57221ffab1320751360c385c22bcb7e91b1f6f59f20174 WatchSource:0}: Error finding container 39a30956a50d3c5fbf57221ffab1320751360c385c22bcb7e91b1f6f59f20174: Status 404 returned error can't find the container with id 39a30956a50d3c5fbf57221ffab1320751360c385c22bcb7e91b1f6f59f20174 Jul 23 11:36:09.449602 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:09.449578 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7q2v" Jul 23 11:36:09.449998 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:09.449739 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fv772" Jul 23 11:36:09.454202 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:09.454185 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 23 11:36:09.454311 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:09.454187 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xtz4b\"" Jul 23 11:36:09.454311 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:09.454189 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7flkj\"" Jul 23 11:36:09.501131 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:09.501104 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f249368d-c49d-4bfe-906f-d1d12104dbd0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-j4sbd\" (UID: \"f249368d-c49d-4bfe-906f-d1d12104dbd0\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-j4sbd" Jul 23 11:36:09.501238 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:09.501187 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5179eda-de1f-4d89-9e22-afb9d63d098b-metrics-certs\") pod \"router-default-78fdc88b76-5w9ql\" (UID: \"c5179eda-de1f-4d89-9e22-afb9d63d098b\") " pod="openshift-ingress/router-default-78fdc88b76-5w9ql" Jul 23 11:36:09.501294 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:09.501272 2561 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 23 11:36:09.501356 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:09.501294 2561 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 23 11:36:09.501356 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:09.501336 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f249368d-c49d-4bfe-906f-d1d12104dbd0-cluster-monitoring-operator-tls podName:f249368d-c49d-4bfe-906f-d1d12104dbd0 nodeName:}" failed. No retries permitted until 2026-07-23 11:36:11.501316883 +0000 UTC m=+38.573006870 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f249368d-c49d-4bfe-906f-d1d12104dbd0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-j4sbd" (UID: "f249368d-c49d-4bfe-906f-d1d12104dbd0") : secret "cluster-monitoring-operator-tls" not found Jul 23 11:36:09.501356 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:09.501355 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5179eda-de1f-4d89-9e22-afb9d63d098b-metrics-certs podName:c5179eda-de1f-4d89-9e22-afb9d63d098b nodeName:}" failed. No retries permitted until 2026-07-23 11:36:11.501345819 +0000 UTC m=+38.573035787 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c5179eda-de1f-4d89-9e22-afb9d63d098b-metrics-certs") pod "router-default-78fdc88b76-5w9ql" (UID: "c5179eda-de1f-4d89-9e22-afb9d63d098b") : secret "router-metrics-certs-default" not found Jul 23 11:36:09.501560 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:09.501391 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5179eda-de1f-4d89-9e22-afb9d63d098b-service-ca-bundle\") pod \"router-default-78fdc88b76-5w9ql\" (UID: \"c5179eda-de1f-4d89-9e22-afb9d63d098b\") " pod="openshift-ingress/router-default-78fdc88b76-5w9ql" Jul 23 11:36:09.501560 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:09.501456 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b993c3b9-94b8-4c21-b391-6193659cf9ee-registry-tls\") pod \"image-registry-67fbb677d9-vf9k4\" (UID: \"b993c3b9-94b8-4c21-b391-6193659cf9ee\") " pod="openshift-image-registry/image-registry-67fbb677d9-vf9k4" Jul 23 11:36:09.501560 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:09.501543 2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 11:36:09.501560 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:09.501553 2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67fbb677d9-vf9k4: secret "image-registry-tls" not found Jul 23 11:36:09.501674 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:09.501578 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b993c3b9-94b8-4c21-b391-6193659cf9ee-registry-tls podName:b993c3b9-94b8-4c21-b391-6193659cf9ee nodeName:}" failed. No retries permitted until 2026-07-23 11:36:11.501571282 +0000 UTC m=+38.573261263 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b993c3b9-94b8-4c21-b391-6193659cf9ee-registry-tls") pod "image-registry-67fbb677d9-vf9k4" (UID: "b993c3b9-94b8-4c21-b391-6193659cf9ee") : secret "image-registry-tls" not found Jul 23 11:36:09.501674 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:09.501624 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c5179eda-de1f-4d89-9e22-afb9d63d098b-service-ca-bundle podName:c5179eda-de1f-4d89-9e22-afb9d63d098b nodeName:}" failed. No retries permitted until 2026-07-23 11:36:11.501618204 +0000 UTC m=+38.573308172 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c5179eda-de1f-4d89-9e22-afb9d63d098b-service-ca-bundle") pod "router-default-78fdc88b76-5w9ql" (UID: "c5179eda-de1f-4d89-9e22-afb9d63d098b") : configmap references non-existent config key: service-ca.crt Jul 23 11:36:09.602410 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:09.602378 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/74038c52-71a3-4c88-b607-7f0f9ec11001-metrics-tls\") pod \"dns-default-fkvjh\" (UID: \"74038c52-71a3-4c88-b607-7f0f9ec11001\") " pod="openshift-dns/dns-default-fkvjh" Jul 23 11:36:09.602571 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:09.602487 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/89ae8727-bb42-4860-a957-dd9a06933722-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-n2rn6\" (UID: \"89ae8727-bb42-4860-a957-dd9a06933722\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-n2rn6" Jul 23 11:36:09.602651 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:09.602632 2561 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 23 11:36:09.602710 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:09.602699 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89ae8727-bb42-4860-a957-dd9a06933722-samples-operator-tls podName:89ae8727-bb42-4860-a957-dd9a06933722 nodeName:}" failed. No retries permitted until 2026-07-23 11:36:11.602682341 +0000 UTC m=+38.674372312 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/89ae8727-bb42-4860-a957-dd9a06933722-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-n2rn6" (UID: "89ae8727-bb42-4860-a957-dd9a06933722") : secret "samples-operator-tls" not found Jul 23 11:36:09.604661 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:09.603149 2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 11:36:09.604661 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:09.603197 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74038c52-71a3-4c88-b607-7f0f9ec11001-metrics-tls podName:74038c52-71a3-4c88-b607-7f0f9ec11001 nodeName:}" failed. No retries permitted until 2026-07-23 11:36:11.603181048 +0000 UTC m=+38.674871019 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/74038c52-71a3-4c88-b607-7f0f9ec11001-metrics-tls") pod "dns-default-fkvjh" (UID: "74038c52-71a3-4c88-b607-7f0f9ec11001") : secret "dns-default-metrics-tls" not found Jul 23 11:36:09.648642 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:09.647884 2561 generic.go:356] "Generic (PLEG): container finished" podID="f898b85f-57f0-4973-9e99-117e9fd60fa8" containerID="2115f467ffb006f78eec810a0827d5c3cda48b3740bf16af3afbc0901a22298d" exitCode=0 Jul 23 11:36:09.648642 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:09.647976 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6jdp4" event={"ID":"f898b85f-57f0-4973-9e99-117e9fd60fa8","Type":"ContainerDied","Data":"2115f467ffb006f78eec810a0827d5c3cda48b3740bf16af3afbc0901a22298d"} Jul 23 11:36:09.652998 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:09.652889 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-sr6b2" event={"ID":"98f50c4a-91e4-4708-be1a-cff80dfb6cc8","Type":"ContainerStarted","Data":"3f29cb7515f2aaaf99cdb291a4271d3ba0caa1dc606a71bf8dbdf260b379db82"} Jul 23 11:36:09.656725 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:09.656683 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-5txgl" event={"ID":"21319854-f159-442b-bf88-4498cec86096","Type":"ContainerStarted","Data":"5d63a6b004cd6e174455111d66a6a56ec11df7a3ad362035ddb792d40278fa69"} Jul 23 11:36:09.661841 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:09.661775 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-5ss74" event={"ID":"0ceda81a-92dd-4858-b012-f05506292db4","Type":"ContainerStarted","Data":"bc53eb84f0ac35676c2b4f2ee8a3f5bfdcb052aa1d75a0b5060ea48141e29474"} Jul 23 11:36:09.664881 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:09.664827 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fww69" event={"ID":"a46e320d-ae78-4a40-9314-075f03b0b6ce","Type":"ContainerStarted","Data":"58d14782974286f3e996729900b3afd38f8d70efc01cf110e47541a7d2d20c15"} Jul 23 11:36:09.668477 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:09.668419 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-7slpj" event={"ID":"65247e1f-fc10-4b40-af67-5e646f352263","Type":"ContainerStarted","Data":"647b0e75f9bff8fd4e1ddddc44d621277cbfa817573af7c771a63c28bac722a3"} Jul 23 11:36:09.672603 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:09.672538 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-8gtcv" event={"ID":"52ccdb9c-03da-48cf-94f4-a916e42975ac","Type":"ContainerStarted","Data":"c62bab0f834f0930d218048a1f9856d70857706a15648044fa4f8882cdadfc2e"} Jul 23 11:36:09.674333 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:09.674278 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-48j4r" event={"ID":"b64d0632-5844-419e-9f53-ac57a0495a8d","Type":"ContainerStarted","Data":"39a30956a50d3c5fbf57221ffab1320751360c385c22bcb7e91b1f6f59f20174"} Jul 23 11:36:10.690462 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:10.690133 2561 generic.go:356] "Generic (PLEG): container finished" podID="f898b85f-57f0-4973-9e99-117e9fd60fa8" containerID="cbc7f09a865e16c09a898d2529b48a9405c33d5f9890b7d3e318f51eb2cd3f7a" exitCode=0 Jul 23 11:36:10.690462 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:10.690272 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6jdp4" event={"ID":"f898b85f-57f0-4973-9e99-117e9fd60fa8","Type":"ContainerDied","Data":"cbc7f09a865e16c09a898d2529b48a9405c33d5f9890b7d3e318f51eb2cd3f7a"} Jul 23 11:36:11.520078 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:11.520038 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f249368d-c49d-4bfe-906f-d1d12104dbd0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-j4sbd\" (UID: \"f249368d-c49d-4bfe-906f-d1d12104dbd0\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-j4sbd" Jul 23 11:36:11.520247 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:11.520140 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5179eda-de1f-4d89-9e22-afb9d63d098b-metrics-certs\") pod \"router-default-78fdc88b76-5w9ql\" (UID: \"c5179eda-de1f-4d89-9e22-afb9d63d098b\") " pod="openshift-ingress/router-default-78fdc88b76-5w9ql" Jul 23 11:36:11.520247 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:11.520185 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5179eda-de1f-4d89-9e22-afb9d63d098b-service-ca-bundle\") pod \"router-default-78fdc88b76-5w9ql\" (UID: \"c5179eda-de1f-4d89-9e22-afb9d63d098b\") " pod="openshift-ingress/router-default-78fdc88b76-5w9ql" Jul 23 11:36:11.520247 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:11.520244 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b993c3b9-94b8-4c21-b391-6193659cf9ee-registry-tls\") pod \"image-registry-67fbb677d9-vf9k4\" (UID: \"b993c3b9-94b8-4c21-b391-6193659cf9ee\") " pod="openshift-image-registry/image-registry-67fbb677d9-vf9k4" Jul 23 11:36:11.520412 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:11.520373 2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 11:36:11.520412 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:11.520387 2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67fbb677d9-vf9k4: secret "image-registry-tls" not found Jul 23 11:36:11.520514 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:11.520442 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b993c3b9-94b8-4c21-b391-6193659cf9ee-registry-tls podName:b993c3b9-94b8-4c21-b391-6193659cf9ee nodeName:}" failed. No retries permitted until 2026-07-23 11:36:15.520424911 +0000 UTC m=+42.592114883 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b993c3b9-94b8-4c21-b391-6193659cf9ee-registry-tls") pod "image-registry-67fbb677d9-vf9k4" (UID: "b993c3b9-94b8-4c21-b391-6193659cf9ee") : secret "image-registry-tls" not found Jul 23 11:36:11.520514 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:11.520514 2561 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 23 11:36:11.520627 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:11.520548 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f249368d-c49d-4bfe-906f-d1d12104dbd0-cluster-monitoring-operator-tls podName:f249368d-c49d-4bfe-906f-d1d12104dbd0 nodeName:}" failed. No retries permitted until 2026-07-23 11:36:15.520536833 +0000 UTC m=+42.592226818 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f249368d-c49d-4bfe-906f-d1d12104dbd0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-j4sbd" (UID: "f249368d-c49d-4bfe-906f-d1d12104dbd0") : secret "cluster-monitoring-operator-tls" not found Jul 23 11:36:11.520627 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:11.520598 2561 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 23 11:36:11.520627 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:11.520627 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5179eda-de1f-4d89-9e22-afb9d63d098b-metrics-certs podName:c5179eda-de1f-4d89-9e22-afb9d63d098b nodeName:}" failed. No retries permitted until 2026-07-23 11:36:15.520618224 +0000 UTC m=+42.592308194 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c5179eda-de1f-4d89-9e22-afb9d63d098b-metrics-certs") pod "router-default-78fdc88b76-5w9ql" (UID: "c5179eda-de1f-4d89-9e22-afb9d63d098b") : secret "router-metrics-certs-default" not found Jul 23 11:36:11.520771 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:11.520693 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c5179eda-de1f-4d89-9e22-afb9d63d098b-service-ca-bundle podName:c5179eda-de1f-4d89-9e22-afb9d63d098b nodeName:}" failed. No retries permitted until 2026-07-23 11:36:15.520682902 +0000 UTC m=+42.592372875 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c5179eda-de1f-4d89-9e22-afb9d63d098b-service-ca-bundle") pod "router-default-78fdc88b76-5w9ql" (UID: "c5179eda-de1f-4d89-9e22-afb9d63d098b") : configmap references non-existent config key: service-ca.crt Jul 23 11:36:11.621328 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:11.621297 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/74038c52-71a3-4c88-b607-7f0f9ec11001-metrics-tls\") pod \"dns-default-fkvjh\" (UID: \"74038c52-71a3-4c88-b607-7f0f9ec11001\") " pod="openshift-dns/dns-default-fkvjh" Jul 23 11:36:11.621511 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:11.621367 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/89ae8727-bb42-4860-a957-dd9a06933722-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-n2rn6\" (UID: \"89ae8727-bb42-4860-a957-dd9a06933722\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-n2rn6" Jul 23 11:36:11.621511 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:11.621502 2561 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 23 11:36:11.621623 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:11.621561 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89ae8727-bb42-4860-a957-dd9a06933722-samples-operator-tls podName:89ae8727-bb42-4860-a957-dd9a06933722 nodeName:}" failed. No retries permitted until 2026-07-23 11:36:15.621541964 +0000 UTC m=+42.693231957 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/89ae8727-bb42-4860-a957-dd9a06933722-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-n2rn6" (UID: "89ae8727-bb42-4860-a957-dd9a06933722") : secret "samples-operator-tls" not found Jul 23 11:36:11.621933 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:11.621894 2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 11:36:11.622037 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:11.621961 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74038c52-71a3-4c88-b607-7f0f9ec11001-metrics-tls podName:74038c52-71a3-4c88-b607-7f0f9ec11001 nodeName:}" failed. No retries permitted until 2026-07-23 11:36:15.621945767 +0000 UTC m=+42.693635738 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/74038c52-71a3-4c88-b607-7f0f9ec11001-metrics-tls") pod "dns-default-fkvjh" (UID: "74038c52-71a3-4c88-b607-7f0f9ec11001") : secret "dns-default-metrics-tls" not found Jul 23 11:36:11.700462 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:11.700425 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6jdp4" event={"ID":"f898b85f-57f0-4973-9e99-117e9fd60fa8","Type":"ContainerStarted","Data":"3eec54552816f26b99870a41d8ab7da49d98590e5b9ae505fc3dec2b65200021"} Jul 23 11:36:11.703349 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:11.702853 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-8gtcv" event={"ID":"52ccdb9c-03da-48cf-94f4-a916e42975ac","Type":"ContainerStarted","Data":"2a2ab4093ff7e703703e72e8f5d9c80945428526686bc3ac61fa8a9b0a5e510f"} Jul 23 11:36:11.725880 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:11.725775 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-6jdp4" podStartSLOduration=8.654910272 podStartE2EDuration="38.72575765s" totalImagesPullingTime="30.070847378s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 11:35:33 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:36:11.724287486 +0000 UTC m=+38.795977476" watchObservedRunningTime="2026-07-23 11:36:11.72575765 +0000 UTC m=+38.797447641" Jul 23 11:36:15.554502 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:15.554462 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f249368d-c49d-4bfe-906f-d1d12104dbd0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-j4sbd\" (UID: \"f249368d-c49d-4bfe-906f-d1d12104dbd0\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-j4sbd" Jul 23 11:36:15.555212 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:15.554568 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5179eda-de1f-4d89-9e22-afb9d63d098b-metrics-certs\") pod \"router-default-78fdc88b76-5w9ql\" (UID: \"c5179eda-de1f-4d89-9e22-afb9d63d098b\") " pod="openshift-ingress/router-default-78fdc88b76-5w9ql" Jul 23 11:36:15.555212 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:15.554622 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5179eda-de1f-4d89-9e22-afb9d63d098b-service-ca-bundle\") pod \"router-default-78fdc88b76-5w9ql\" (UID: \"c5179eda-de1f-4d89-9e22-afb9d63d098b\") " pod="openshift-ingress/router-default-78fdc88b76-5w9ql" Jul 23 11:36:15.555212 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:15.554632 2561 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 23 11:36:15.555212 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:15.554667 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b993c3b9-94b8-4c21-b391-6193659cf9ee-registry-tls\") pod \"image-registry-67fbb677d9-vf9k4\" (UID: \"b993c3b9-94b8-4c21-b391-6193659cf9ee\") " pod="openshift-image-registry/image-registry-67fbb677d9-vf9k4" Jul 23 11:36:15.555212 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:15.554707 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f249368d-c49d-4bfe-906f-d1d12104dbd0-cluster-monitoring-operator-tls podName:f249368d-c49d-4bfe-906f-d1d12104dbd0 nodeName:}" failed. No retries permitted until 2026-07-23 11:36:23.554684755 +0000 UTC m=+50.626374723 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f249368d-c49d-4bfe-906f-d1d12104dbd0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-j4sbd" (UID: "f249368d-c49d-4bfe-906f-d1d12104dbd0") : secret "cluster-monitoring-operator-tls" not found Jul 23 11:36:15.555212 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:15.554749 2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 11:36:15.555212 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:15.554759 2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67fbb677d9-vf9k4: secret "image-registry-tls" not found Jul 23 11:36:15.555212 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:15.554804 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b993c3b9-94b8-4c21-b391-6193659cf9ee-registry-tls podName:b993c3b9-94b8-4c21-b391-6193659cf9ee nodeName:}" failed. No retries permitted until 2026-07-23 11:36:23.554793069 +0000 UTC m=+50.626483038 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b993c3b9-94b8-4c21-b391-6193659cf9ee-registry-tls") pod "image-registry-67fbb677d9-vf9k4" (UID: "b993c3b9-94b8-4c21-b391-6193659cf9ee") : secret "image-registry-tls" not found Jul 23 11:36:15.555212 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:15.554841 2561 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 23 11:36:15.555212 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:15.554868 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5179eda-de1f-4d89-9e22-afb9d63d098b-metrics-certs podName:c5179eda-de1f-4d89-9e22-afb9d63d098b nodeName:}" failed. No retries permitted until 2026-07-23 11:36:23.554862223 +0000 UTC m=+50.626552191 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c5179eda-de1f-4d89-9e22-afb9d63d098b-metrics-certs") pod "router-default-78fdc88b76-5w9ql" (UID: "c5179eda-de1f-4d89-9e22-afb9d63d098b") : secret "router-metrics-certs-default" not found Jul 23 11:36:15.555212 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:15.554932 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c5179eda-de1f-4d89-9e22-afb9d63d098b-service-ca-bundle podName:c5179eda-de1f-4d89-9e22-afb9d63d098b nodeName:}" failed. No retries permitted until 2026-07-23 11:36:23.554905166 +0000 UTC m=+50.626595134 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c5179eda-de1f-4d89-9e22-afb9d63d098b-service-ca-bundle") pod "router-default-78fdc88b76-5w9ql" (UID: "c5179eda-de1f-4d89-9e22-afb9d63d098b") : configmap references non-existent config key: service-ca.crt Jul 23 11:36:15.655302 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:15.655272 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/74038c52-71a3-4c88-b607-7f0f9ec11001-metrics-tls\") pod \"dns-default-fkvjh\" (UID: \"74038c52-71a3-4c88-b607-7f0f9ec11001\") " pod="openshift-dns/dns-default-fkvjh" Jul 23 11:36:15.655432 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:15.655352 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/89ae8727-bb42-4860-a957-dd9a06933722-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-n2rn6\" (UID: \"89ae8727-bb42-4860-a957-dd9a06933722\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-n2rn6" Jul 23 11:36:15.655501 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:15.655449 2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 11:36:15.655501 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:15.655480 2561 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 23 11:36:15.655573 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:15.655524 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74038c52-71a3-4c88-b607-7f0f9ec11001-metrics-tls podName:74038c52-71a3-4c88-b607-7f0f9ec11001 nodeName:}" failed. No retries permitted until 2026-07-23 11:36:23.655501988 +0000 UTC m=+50.727191973 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/74038c52-71a3-4c88-b607-7f0f9ec11001-metrics-tls") pod "dns-default-fkvjh" (UID: "74038c52-71a3-4c88-b607-7f0f9ec11001") : secret "dns-default-metrics-tls" not found Jul 23 11:36:15.655573 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:15.655546 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89ae8727-bb42-4860-a957-dd9a06933722-samples-operator-tls podName:89ae8727-bb42-4860-a957-dd9a06933722 nodeName:}" failed. No retries permitted until 2026-07-23 11:36:23.655535644 +0000 UTC m=+50.727225615 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/89ae8727-bb42-4860-a957-dd9a06933722-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-n2rn6" (UID: "89ae8727-bb42-4860-a957-dd9a06933722") : secret "samples-operator-tls" not found Jul 23 11:36:16.720992 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:16.720899 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-sr6b2" event={"ID":"98f50c4a-91e4-4708-be1a-cff80dfb6cc8","Type":"ContainerStarted","Data":"a4f45d2edccb24164e03cac100ef2ecc9b6d46500dbe308b19875b72b90b9e21"} Jul 23 11:36:16.722943 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:16.722866 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-5txgl" event={"ID":"21319854-f159-442b-bf88-4498cec86096","Type":"ContainerStarted","Data":"0dbbd5387a70b304b338c702b8b7f441d0ad42e19732e93b3c2735128804c381"} Jul 23 11:36:16.724953 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:16.724696 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-5ss74" event={"ID":"0ceda81a-92dd-4858-b012-f05506292db4","Type":"ContainerStarted","Data":"94de40fd2ad7fd731fafdf9bdb45d0d597e742a156c292ef72bd132b3ab08010"} Jul 23 11:36:16.726779 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:16.726756 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fww69" event={"ID":"a46e320d-ae78-4a40-9314-075f03b0b6ce","Type":"ContainerStarted","Data":"c580097a1737ccc1df803a9e4629c6c4c40fdc789cde87ea70dfe7f7de116946"} Jul 23 11:36:16.728551 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:16.728533 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-7slpj_65247e1f-fc10-4b40-af67-5e646f352263/console-operator/0.log" Jul 23 11:36:16.728630 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:16.728566 2561 generic.go:356] "Generic (PLEG): container finished" podID="65247e1f-fc10-4b40-af67-5e646f352263" containerID="4fa0d042f9d49344f996e3335dd4f4e0ee9d986db6dbcb9102830692a9f229db" exitCode=255 Jul 23 11:36:16.728810 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:16.728771 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-7slpj" event={"ID":"65247e1f-fc10-4b40-af67-5e646f352263","Type":"ContainerDied","Data":"4fa0d042f9d49344f996e3335dd4f4e0ee9d986db6dbcb9102830692a9f229db"} Jul 23 11:36:16.728883 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:16.728852 2561 scope.go:117] "RemoveContainer" containerID="4fa0d042f9d49344f996e3335dd4f4e0ee9d986db6dbcb9102830692a9f229db" Jul 23 11:36:16.730287 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:16.730263 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-48j4r" event={"ID":"b64d0632-5844-419e-9f53-ac57a0495a8d","Type":"ContainerStarted","Data":"2b93f10485091aa9685dd137dfe9afc70a4933e8eb4f9a5283a8e84bd443a70c"} Jul 23 11:36:16.753356 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:16.753308 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-8gtcv" podStartSLOduration=17.974034895 podStartE2EDuration="19.753293551s" totalImagesPullingTime="1.779258656s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:35:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:36:11.74243636 +0000 UTC m=+38.814126350" watchObservedRunningTime="2026-07-23 11:36:16.753293551 +0000 UTC m=+43.824983522" Jul 23 11:36:16.753738 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:16.753713 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-sr6b2" podStartSLOduration=12.455796933 podStartE2EDuration="19.753706241s" totalImagesPullingTime="7.297909308s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:35:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:36:16.753048874 +0000 UTC m=+43.824738867" watchObservedRunningTime="2026-07-23 11:36:16.753706241 +0000 UTC m=+43.825396234" Jul 23 11:36:16.802297 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:16.802252 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-555d9d7885-5txgl" podStartSLOduration=12.505288912 podStartE2EDuration="19.802233635s" totalImagesPullingTime="7.296944723s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:35:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:36:16.800253294 +0000 UTC m=+43.871943284" watchObservedRunningTime="2026-07-23 11:36:16.802233635 +0000 UTC m=+43.873923618" Jul 23 11:36:16.850277 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:16.850233 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-48j4r" podStartSLOduration=12.555168369 podStartE2EDuration="19.850215584s" totalImagesPullingTime="7.295047215s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:35:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:36:16.826412737 +0000 UTC m=+43.898102731" watchObservedRunningTime="2026-07-23 11:36:16.850215584 +0000 UTC m=+43.921905575" Jul 23 11:36:16.850400 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:16.850326 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-5ss74" podStartSLOduration=11.542006589 podStartE2EDuration="18.850320122s" totalImagesPullingTime="7.308313533s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:35:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:36:16.848958517 +0000 UTC m=+43.920648502" watchObservedRunningTime="2026-07-23 11:36:16.850320122 +0000 UTC m=+43.922010113" Jul 23 11:36:17.735976 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:17.735865 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-7slpj_65247e1f-fc10-4b40-af67-5e646f352263/console-operator/1.log" Jul 23 11:36:17.736403 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:17.736341 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-7slpj_65247e1f-fc10-4b40-af67-5e646f352263/console-operator/0.log" Jul 23 11:36:17.736403 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:17.736383 2561 generic.go:356] "Generic (PLEG): container finished" podID="65247e1f-fc10-4b40-af67-5e646f352263" containerID="8df4d8334a61f9bc7e532e99b63420fbbca2afbd802c35635fef53a65d397939" exitCode=255 Jul 23 11:36:17.736679 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:17.736506 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-7slpj" event={"ID":"65247e1f-fc10-4b40-af67-5e646f352263","Type":"ContainerDied","Data":"8df4d8334a61f9bc7e532e99b63420fbbca2afbd802c35635fef53a65d397939"} Jul 23 11:36:17.736679 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:17.736557 2561 scope.go:117] "RemoveContainer" containerID="4fa0d042f9d49344f996e3335dd4f4e0ee9d986db6dbcb9102830692a9f229db" Jul 23 11:36:17.736867 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:17.736754 2561 scope.go:117] "RemoveContainer" containerID="8df4d8334a61f9bc7e532e99b63420fbbca2afbd802c35635fef53a65d397939" Jul 23 11:36:17.737373 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:17.736972 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-7slpj_openshift-console-operator(65247e1f-fc10-4b40-af67-5e646f352263)\"" pod="openshift-console-operator/console-operator-565b7b569d-7slpj" podUID="65247e1f-fc10-4b40-af67-5e646f352263" Jul 23 11:36:17.756469 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:17.756417 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-fww69" podStartSLOduration=3.452528075 podStartE2EDuration="10.75639633s" totalImagesPullingTime="7.303868255s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:36:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:36:16.871022533 +0000 UTC m=+43.942712523" watchObservedRunningTime="2026-07-23 11:36:17.75639633 +0000 UTC m=+44.828086319" Jul 23 11:36:18.088331 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:18.088306 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-7slpj" Jul 23 11:36:18.088488 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:18.088351 2561 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-565b7b569d-7slpj" Jul 23 11:36:18.163830 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:18.163797 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-zq4l7"] Jul 23 11:36:18.193617 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:18.193589 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-4rfbr"] Jul 23 11:36:18.193791 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:18.193727 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-zq4l7" Jul 23 11:36:18.196439 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:18.196420 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 23 11:36:18.196570 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:18.196466 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 23 11:36:18.196701 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:18.196685 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-wljrx\"" Jul 23 11:36:18.217280 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:18.217263 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-zq4l7"] Jul 23 11:36:18.217365 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:18.217292 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4rfbr"] Jul 23 11:36:18.217403 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:18.217371 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4rfbr" Jul 23 11:36:18.219946 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:18.219886 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 23 11:36:18.220069 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:18.220042 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 23 11:36:18.220157 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:18.220076 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jxxcm\"" Jul 23 11:36:18.278211 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:18.278189 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nhfj\" (UniqueName: \"kubernetes.io/projected/57491661-d002-4cf8-8437-77a7637d4a24-kube-api-access-4nhfj\") pod \"migrator-7cc9c7ffd7-zq4l7\" (UID: \"57491661-d002-4cf8-8437-77a7637d4a24\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-zq4l7" Jul 23 11:36:18.278320 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:18.278236 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1ae0a33f-4691-4a97-aea8-70267e2a8057-data-volume\") pod \"insights-runtime-extractor-4rfbr\" (UID: \"1ae0a33f-4691-4a97-aea8-70267e2a8057\") " pod="openshift-insights/insights-runtime-extractor-4rfbr" Jul 23 11:36:18.278320 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:18.278259 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1ae0a33f-4691-4a97-aea8-70267e2a8057-crio-socket\") pod \"insights-runtime-extractor-4rfbr\" (UID: \"1ae0a33f-4691-4a97-aea8-70267e2a8057\") " pod="openshift-insights/insights-runtime-extractor-4rfbr" Jul 23 11:36:18.278320 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:18.278311 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1ae0a33f-4691-4a97-aea8-70267e2a8057-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4rfbr\" (UID: \"1ae0a33f-4691-4a97-aea8-70267e2a8057\") " pod="openshift-insights/insights-runtime-extractor-4rfbr" Jul 23 11:36:18.278459 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:18.278334 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1ae0a33f-4691-4a97-aea8-70267e2a8057-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4rfbr\" (UID: \"1ae0a33f-4691-4a97-aea8-70267e2a8057\") " pod="openshift-insights/insights-runtime-extractor-4rfbr" Jul 23 11:36:18.278459 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:18.278389 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4shr\" (UniqueName: \"kubernetes.io/projected/1ae0a33f-4691-4a97-aea8-70267e2a8057-kube-api-access-g4shr\") pod \"insights-runtime-extractor-4rfbr\" (UID: \"1ae0a33f-4691-4a97-aea8-70267e2a8057\") " pod="openshift-insights/insights-runtime-extractor-4rfbr" Jul 23 11:36:18.379001 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:18.378949 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1ae0a33f-4691-4a97-aea8-70267e2a8057-data-volume\") pod \"insights-runtime-extractor-4rfbr\" (UID: \"1ae0a33f-4691-4a97-aea8-70267e2a8057\") " pod="openshift-insights/insights-runtime-extractor-4rfbr" Jul 23 11:36:18.379001 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:18.378977 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1ae0a33f-4691-4a97-aea8-70267e2a8057-crio-socket\") pod \"insights-runtime-extractor-4rfbr\" (UID: \"1ae0a33f-4691-4a97-aea8-70267e2a8057\") " pod="openshift-insights/insights-runtime-extractor-4rfbr" Jul 23 11:36:18.379166 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:18.379019 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1ae0a33f-4691-4a97-aea8-70267e2a8057-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4rfbr\" (UID: \"1ae0a33f-4691-4a97-aea8-70267e2a8057\") " pod="openshift-insights/insights-runtime-extractor-4rfbr" Jul 23 11:36:18.379166 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:18.379038 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1ae0a33f-4691-4a97-aea8-70267e2a8057-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4rfbr\" (UID: \"1ae0a33f-4691-4a97-aea8-70267e2a8057\") " pod="openshift-insights/insights-runtime-extractor-4rfbr" Jul 23 11:36:18.379166 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:18.379069 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4shr\" (UniqueName: \"kubernetes.io/projected/1ae0a33f-4691-4a97-aea8-70267e2a8057-kube-api-access-g4shr\") pod \"insights-runtime-extractor-4rfbr\" (UID: \"1ae0a33f-4691-4a97-aea8-70267e2a8057\") " pod="openshift-insights/insights-runtime-extractor-4rfbr" Jul 23 11:36:18.379166 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:18.379104 2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 23 11:36:18.379166 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:18.379147 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nhfj\" (UniqueName: \"kubernetes.io/projected/57491661-d002-4cf8-8437-77a7637d4a24-kube-api-access-4nhfj\") pod \"migrator-7cc9c7ffd7-zq4l7\" (UID: \"57491661-d002-4cf8-8437-77a7637d4a24\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-zq4l7" Jul 23 11:36:18.379166 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:18.379164 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1ae0a33f-4691-4a97-aea8-70267e2a8057-kube-rbac-proxy-cm podName:1ae0a33f-4691-4a97-aea8-70267e2a8057 nodeName:}" failed. No retries permitted until 2026-07-23 11:36:18.879144495 +0000 UTC m=+45.950834463 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1ae0a33f-4691-4a97-aea8-70267e2a8057-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4rfbr" (UID: "1ae0a33f-4691-4a97-aea8-70267e2a8057") : configmap "kube-rbac-proxy" not found Jul 23 11:36:18.379430 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:18.379206 2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 23 11:36:18.379430 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:18.379213 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1ae0a33f-4691-4a97-aea8-70267e2a8057-crio-socket\") pod \"insights-runtime-extractor-4rfbr\" (UID: \"1ae0a33f-4691-4a97-aea8-70267e2a8057\") " pod="openshift-insights/insights-runtime-extractor-4rfbr" Jul 23 11:36:18.379430 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:18.379253 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ae0a33f-4691-4a97-aea8-70267e2a8057-insights-runtime-extractor-tls podName:1ae0a33f-4691-4a97-aea8-70267e2a8057 nodeName:}" failed. No retries permitted until 2026-07-23 11:36:18.879238425 +0000 UTC m=+45.950928409 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1ae0a33f-4691-4a97-aea8-70267e2a8057-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4rfbr" (UID: "1ae0a33f-4691-4a97-aea8-70267e2a8057") : secret "insights-runtime-extractor-tls" not found Jul 23 11:36:18.379430 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:18.379324 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1ae0a33f-4691-4a97-aea8-70267e2a8057-data-volume\") pod \"insights-runtime-extractor-4rfbr\" (UID: \"1ae0a33f-4691-4a97-aea8-70267e2a8057\") " pod="openshift-insights/insights-runtime-extractor-4rfbr" Jul 23 11:36:18.397745 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:18.397720 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nhfj\" (UniqueName: \"kubernetes.io/projected/57491661-d002-4cf8-8437-77a7637d4a24-kube-api-access-4nhfj\") pod \"migrator-7cc9c7ffd7-zq4l7\" (UID: \"57491661-d002-4cf8-8437-77a7637d4a24\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-zq4l7" Jul 23 11:36:18.412942 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:18.412902 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4shr\" (UniqueName: \"kubernetes.io/projected/1ae0a33f-4691-4a97-aea8-70267e2a8057-kube-api-access-g4shr\") pod \"insights-runtime-extractor-4rfbr\" (UID: \"1ae0a33f-4691-4a97-aea8-70267e2a8057\") " pod="openshift-insights/insights-runtime-extractor-4rfbr" Jul 23 11:36:18.502302 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:18.502283 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-zq4l7" Jul 23 11:36:18.645698 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:18.645588 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-zq4l7"] Jul 23 11:36:18.650017 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:36:18.649987 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57491661_d002_4cf8_8437_77a7637d4a24.slice/crio-574c52e936a38ea5282bacf4a5766f03ce4329a3f70a9f1cbafc848867d1d878 WatchSource:0}: Error finding container 574c52e936a38ea5282bacf4a5766f03ce4329a3f70a9f1cbafc848867d1d878: Status 404 returned error can't find the container with id 574c52e936a38ea5282bacf4a5766f03ce4329a3f70a9f1cbafc848867d1d878 Jul 23 11:36:18.756384 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:18.756355 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-zq4l7" event={"ID":"57491661-d002-4cf8-8437-77a7637d4a24","Type":"ContainerStarted","Data":"574c52e936a38ea5282bacf4a5766f03ce4329a3f70a9f1cbafc848867d1d878"} Jul 23 11:36:18.757801 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:18.757782 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-7slpj_65247e1f-fc10-4b40-af67-5e646f352263/console-operator/1.log" Jul 23 11:36:18.758161 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:18.758139 2561 scope.go:117] "RemoveContainer" containerID="8df4d8334a61f9bc7e532e99b63420fbbca2afbd802c35635fef53a65d397939" Jul 23 11:36:18.758323 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:18.758303 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-7slpj_openshift-console-operator(65247e1f-fc10-4b40-af67-5e646f352263)\"" pod="openshift-console-operator/console-operator-565b7b569d-7slpj" podUID="65247e1f-fc10-4b40-af67-5e646f352263" Jul 23 11:36:18.885498 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:18.885472 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1ae0a33f-4691-4a97-aea8-70267e2a8057-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4rfbr\" (UID: \"1ae0a33f-4691-4a97-aea8-70267e2a8057\") " pod="openshift-insights/insights-runtime-extractor-4rfbr" Jul 23 11:36:18.885585 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:18.885500 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1ae0a33f-4691-4a97-aea8-70267e2a8057-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4rfbr\" (UID: \"1ae0a33f-4691-4a97-aea8-70267e2a8057\") " pod="openshift-insights/insights-runtime-extractor-4rfbr" Jul 23 11:36:18.885585 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:18.885578 2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 23 11:36:18.885649 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:18.885597 2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 23 11:36:18.885649 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:18.885636 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1ae0a33f-4691-4a97-aea8-70267e2a8057-kube-rbac-proxy-cm podName:1ae0a33f-4691-4a97-aea8-70267e2a8057 nodeName:}" failed. No retries permitted until 2026-07-23 11:36:19.885619513 +0000 UTC m=+46.957309495 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1ae0a33f-4691-4a97-aea8-70267e2a8057-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4rfbr" (UID: "1ae0a33f-4691-4a97-aea8-70267e2a8057") : configmap "kube-rbac-proxy" not found Jul 23 11:36:18.885649 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:18.885650 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ae0a33f-4691-4a97-aea8-70267e2a8057-insights-runtime-extractor-tls podName:1ae0a33f-4691-4a97-aea8-70267e2a8057 nodeName:}" failed. No retries permitted until 2026-07-23 11:36:19.885643855 +0000 UTC m=+46.957333823 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1ae0a33f-4691-4a97-aea8-70267e2a8057-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4rfbr" (UID: "1ae0a33f-4691-4a97-aea8-70267e2a8057") : secret "insights-runtime-extractor-tls" not found Jul 23 11:36:19.564783 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:19.564757 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-pkwm5_a17ee9c0-c387-4e9d-a6a4-481501f5b97b/dns-node-resolver/0.log" Jul 23 11:36:19.760740 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:19.760718 2561 scope.go:117] "RemoveContainer" containerID="8df4d8334a61f9bc7e532e99b63420fbbca2afbd802c35635fef53a65d397939" Jul 23 11:36:19.761081 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:19.760851 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-7slpj_openshift-console-operator(65247e1f-fc10-4b40-af67-5e646f352263)\"" pod="openshift-console-operator/console-operator-565b7b569d-7slpj" podUID="65247e1f-fc10-4b40-af67-5e646f352263" Jul 23 11:36:19.895123 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:19.895062 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1ae0a33f-4691-4a97-aea8-70267e2a8057-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4rfbr\" (UID: \"1ae0a33f-4691-4a97-aea8-70267e2a8057\") " pod="openshift-insights/insights-runtime-extractor-4rfbr" Jul 23 11:36:19.895123 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:19.895090 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1ae0a33f-4691-4a97-aea8-70267e2a8057-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4rfbr\" (UID: \"1ae0a33f-4691-4a97-aea8-70267e2a8057\") " pod="openshift-insights/insights-runtime-extractor-4rfbr" Jul 23 11:36:19.895261 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:19.895161 2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 23 11:36:19.895261 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:19.895204 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1ae0a33f-4691-4a97-aea8-70267e2a8057-kube-rbac-proxy-cm podName:1ae0a33f-4691-4a97-aea8-70267e2a8057 nodeName:}" failed. No retries permitted until 2026-07-23 11:36:21.895190622 +0000 UTC m=+48.966880594 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1ae0a33f-4691-4a97-aea8-70267e2a8057-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4rfbr" (UID: "1ae0a33f-4691-4a97-aea8-70267e2a8057") : configmap "kube-rbac-proxy" not found Jul 23 11:36:19.895261 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:19.895244 2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 23 11:36:19.895357 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:19.895287 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ae0a33f-4691-4a97-aea8-70267e2a8057-insights-runtime-extractor-tls podName:1ae0a33f-4691-4a97-aea8-70267e2a8057 nodeName:}" failed. No retries permitted until 2026-07-23 11:36:21.895275135 +0000 UTC m=+48.966965107 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1ae0a33f-4691-4a97-aea8-70267e2a8057-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4rfbr" (UID: "1ae0a33f-4691-4a97-aea8-70267e2a8057") : secret "insights-runtime-extractor-tls" not found Jul 23 11:36:20.335636 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:20.335610 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-58598695f7-jzxsm"] Jul 23 11:36:20.359242 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:20.359221 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-jzxsm"] Jul 23 11:36:20.359365 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:20.359293 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-jzxsm" Jul 23 11:36:20.368681 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:20.368658 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 23 11:36:20.368819 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:20.368757 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 23 11:36:20.369062 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:20.369046 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 23 11:36:20.369792 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:20.369780 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-jmccq\"" Jul 23 11:36:20.369842 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:20.369803 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 23 11:36:20.374375 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:20.374360 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6cbrx_aa374cc8-936e-41ca-a905-c0449df6825d/node-ca/0.log" Jul 23 11:36:20.398848 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:20.398827 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/05e7fd96-abbd-4a7f-8958-0e5f4b1592a9-signing-cabundle\") pod \"service-ca-58598695f7-jzxsm\" (UID: \"05e7fd96-abbd-4a7f-8958-0e5f4b1592a9\") " pod="openshift-service-ca/service-ca-58598695f7-jzxsm" Jul 23 11:36:20.398955 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:20.398879 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58dq2\" (UniqueName: \"kubernetes.io/projected/05e7fd96-abbd-4a7f-8958-0e5f4b1592a9-kube-api-access-58dq2\") pod \"service-ca-58598695f7-jzxsm\" (UID: \"05e7fd96-abbd-4a7f-8958-0e5f4b1592a9\") " pod="openshift-service-ca/service-ca-58598695f7-jzxsm" Jul 23 11:36:20.398999 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:20.398952 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/05e7fd96-abbd-4a7f-8958-0e5f4b1592a9-signing-key\") pod \"service-ca-58598695f7-jzxsm\" (UID: \"05e7fd96-abbd-4a7f-8958-0e5f4b1592a9\") " pod="openshift-service-ca/service-ca-58598695f7-jzxsm" Jul 23 11:36:20.499676 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:20.499650 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/05e7fd96-abbd-4a7f-8958-0e5f4b1592a9-signing-cabundle\") pod \"service-ca-58598695f7-jzxsm\" (UID: \"05e7fd96-abbd-4a7f-8958-0e5f4b1592a9\") " pod="openshift-service-ca/service-ca-58598695f7-jzxsm" Jul 23 11:36:20.499782 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:20.499700 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58dq2\" (UniqueName: \"kubernetes.io/projected/05e7fd96-abbd-4a7f-8958-0e5f4b1592a9-kube-api-access-58dq2\") pod \"service-ca-58598695f7-jzxsm\" (UID: \"05e7fd96-abbd-4a7f-8958-0e5f4b1592a9\") " pod="openshift-service-ca/service-ca-58598695f7-jzxsm" Jul 23 11:36:20.499782 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:20.499750 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/05e7fd96-abbd-4a7f-8958-0e5f4b1592a9-signing-key\") pod \"service-ca-58598695f7-jzxsm\" (UID: \"05e7fd96-abbd-4a7f-8958-0e5f4b1592a9\") " pod="openshift-service-ca/service-ca-58598695f7-jzxsm" Jul 23 11:36:20.500425 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:20.500369 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/05e7fd96-abbd-4a7f-8958-0e5f4b1592a9-signing-cabundle\") pod \"service-ca-58598695f7-jzxsm\" (UID: \"05e7fd96-abbd-4a7f-8958-0e5f4b1592a9\") " pod="openshift-service-ca/service-ca-58598695f7-jzxsm" Jul 23 11:36:20.502262 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:20.502243 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/05e7fd96-abbd-4a7f-8958-0e5f4b1592a9-signing-key\") pod \"service-ca-58598695f7-jzxsm\" (UID: \"05e7fd96-abbd-4a7f-8958-0e5f4b1592a9\") " pod="openshift-service-ca/service-ca-58598695f7-jzxsm" Jul 23 11:36:20.522100 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:20.522072 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58dq2\" (UniqueName: \"kubernetes.io/projected/05e7fd96-abbd-4a7f-8958-0e5f4b1592a9-kube-api-access-58dq2\") pod \"service-ca-58598695f7-jzxsm\" (UID: \"05e7fd96-abbd-4a7f-8958-0e5f4b1592a9\") " pod="openshift-service-ca/service-ca-58598695f7-jzxsm" Jul 23 11:36:20.667695 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:20.667618 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-jzxsm" Jul 23 11:36:20.823163 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:20.823134 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-jzxsm"] Jul 23 11:36:20.825947 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:36:20.825923 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05e7fd96_abbd_4a7f_8958_0e5f4b1592a9.slice/crio-ae963e0a94d9e9a6f1faf558662ed8cd71d480e76c37e866e72c0ec390dc83d5 WatchSource:0}: Error finding container ae963e0a94d9e9a6f1faf558662ed8cd71d480e76c37e866e72c0ec390dc83d5: Status 404 returned error can't find the container with id ae963e0a94d9e9a6f1faf558662ed8cd71d480e76c37e866e72c0ec390dc83d5 Jul 23 11:36:20.964390 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:20.964311 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-fww69_a46e320d-ae78-4a40-9314-075f03b0b6ce/serve-healthcheck-canary/0.log" Jul 23 11:36:21.770626 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:21.770545 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-zq4l7" event={"ID":"57491661-d002-4cf8-8437-77a7637d4a24","Type":"ContainerStarted","Data":"d2848385e8415a3c8aea9c0a09b48b904cc0837f812918997cdbf50f08ed12e7"} Jul 23 11:36:21.770626 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:21.770605 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-zq4l7" event={"ID":"57491661-d002-4cf8-8437-77a7637d4a24","Type":"ContainerStarted","Data":"8d8c055960111a31e6280ecf22fb59498de759c5bec5f30a690f46cd0346f6ee"} Jul 23 11:36:21.771936 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:21.771896 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-jzxsm" event={"ID":"05e7fd96-abbd-4a7f-8958-0e5f4b1592a9","Type":"ContainerStarted","Data":"650b0fd7f5794c4b6f7e907a34ebdf37297bcf0f16a965684c287f2459b3bf29"} Jul 23 11:36:21.772020 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:21.771957 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-jzxsm" event={"ID":"05e7fd96-abbd-4a7f-8958-0e5f4b1592a9","Type":"ContainerStarted","Data":"ae963e0a94d9e9a6f1faf558662ed8cd71d480e76c37e866e72c0ec390dc83d5"} Jul 23 11:36:21.805395 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:21.805331 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-zq4l7" podStartSLOduration=0.970948384 podStartE2EDuration="3.805310732s" totalImagesPullingTime="2.834362348s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:36:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:36:21.8039047 +0000 UTC m=+48.875594690" watchObservedRunningTime="2026-07-23 11:36:21.805310732 +0000 UTC m=+48.877000724" Jul 23 11:36:21.913808 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:21.913766 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1ae0a33f-4691-4a97-aea8-70267e2a8057-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4rfbr\" (UID: \"1ae0a33f-4691-4a97-aea8-70267e2a8057\") " pod="openshift-insights/insights-runtime-extractor-4rfbr" Jul 23 11:36:21.913808 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:21.913804 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1ae0a33f-4691-4a97-aea8-70267e2a8057-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4rfbr\" (UID: \"1ae0a33f-4691-4a97-aea8-70267e2a8057\") " pod="openshift-insights/insights-runtime-extractor-4rfbr" Jul 23 11:36:21.914353 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:21.913886 2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 23 11:36:21.914353 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:21.913928 2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 23 11:36:21.914353 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:21.913985 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ae0a33f-4691-4a97-aea8-70267e2a8057-insights-runtime-extractor-tls podName:1ae0a33f-4691-4a97-aea8-70267e2a8057 nodeName:}" failed. No retries permitted until 2026-07-23 11:36:25.9139679 +0000 UTC m=+52.985657885 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1ae0a33f-4691-4a97-aea8-70267e2a8057-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4rfbr" (UID: "1ae0a33f-4691-4a97-aea8-70267e2a8057") : secret "insights-runtime-extractor-tls" not found Jul 23 11:36:21.914353 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:21.914001 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1ae0a33f-4691-4a97-aea8-70267e2a8057-kube-rbac-proxy-cm podName:1ae0a33f-4691-4a97-aea8-70267e2a8057 nodeName:}" failed. No retries permitted until 2026-07-23 11:36:25.913992597 +0000 UTC m=+52.985682566 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1ae0a33f-4691-4a97-aea8-70267e2a8057-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4rfbr" (UID: "1ae0a33f-4691-4a97-aea8-70267e2a8057") : configmap "kube-rbac-proxy" not found Jul 23 11:36:23.630506 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:23.630479 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5179eda-de1f-4d89-9e22-afb9d63d098b-metrics-certs\") pod \"router-default-78fdc88b76-5w9ql\" (UID: \"c5179eda-de1f-4d89-9e22-afb9d63d098b\") " pod="openshift-ingress/router-default-78fdc88b76-5w9ql" Jul 23 11:36:23.630506 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:23.630518 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5179eda-de1f-4d89-9e22-afb9d63d098b-service-ca-bundle\") pod \"router-default-78fdc88b76-5w9ql\" (UID: \"c5179eda-de1f-4d89-9e22-afb9d63d098b\") " pod="openshift-ingress/router-default-78fdc88b76-5w9ql" Jul 23 11:36:23.630944 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:23.630555 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b993c3b9-94b8-4c21-b391-6193659cf9ee-registry-tls\") pod \"image-registry-67fbb677d9-vf9k4\" (UID: \"b993c3b9-94b8-4c21-b391-6193659cf9ee\") " pod="openshift-image-registry/image-registry-67fbb677d9-vf9k4" Jul 23 11:36:23.630944 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:23.630653 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c5179eda-de1f-4d89-9e22-afb9d63d098b-service-ca-bundle podName:c5179eda-de1f-4d89-9e22-afb9d63d098b nodeName:}" failed. No retries permitted until 2026-07-23 11:36:39.630636281 +0000 UTC m=+66.702326250 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c5179eda-de1f-4d89-9e22-afb9d63d098b-service-ca-bundle") pod "router-default-78fdc88b76-5w9ql" (UID: "c5179eda-de1f-4d89-9e22-afb9d63d098b") : configmap references non-existent config key: service-ca.crt Jul 23 11:36:23.630944 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:23.630656 2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 11:36:23.630944 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:23.630663 2561 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 23 11:36:23.630944 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:23.630673 2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67fbb677d9-vf9k4: secret "image-registry-tls" not found Jul 23 11:36:23.630944 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:23.630677 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f249368d-c49d-4bfe-906f-d1d12104dbd0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-j4sbd\" (UID: \"f249368d-c49d-4bfe-906f-d1d12104dbd0\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-j4sbd" Jul 23 11:36:23.630944 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:23.630710 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b993c3b9-94b8-4c21-b391-6193659cf9ee-registry-tls podName:b993c3b9-94b8-4c21-b391-6193659cf9ee nodeName:}" failed. No retries permitted until 2026-07-23 11:36:39.630698295 +0000 UTC m=+66.702388266 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b993c3b9-94b8-4c21-b391-6193659cf9ee-registry-tls") pod "image-registry-67fbb677d9-vf9k4" (UID: "b993c3b9-94b8-4c21-b391-6193659cf9ee") : secret "image-registry-tls" not found Jul 23 11:36:23.630944 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:23.630733 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5179eda-de1f-4d89-9e22-afb9d63d098b-metrics-certs podName:c5179eda-de1f-4d89-9e22-afb9d63d098b nodeName:}" failed. No retries permitted until 2026-07-23 11:36:39.630715822 +0000 UTC m=+66.702405805 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c5179eda-de1f-4d89-9e22-afb9d63d098b-metrics-certs") pod "router-default-78fdc88b76-5w9ql" (UID: "c5179eda-de1f-4d89-9e22-afb9d63d098b") : secret "router-metrics-certs-default" not found Jul 23 11:36:23.630944 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:23.630749 2561 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 23 11:36:23.630944 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:23.630778 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f249368d-c49d-4bfe-906f-d1d12104dbd0-cluster-monitoring-operator-tls podName:f249368d-c49d-4bfe-906f-d1d12104dbd0 nodeName:}" failed. No retries permitted until 2026-07-23 11:36:39.630771937 +0000 UTC m=+66.702461905 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f249368d-c49d-4bfe-906f-d1d12104dbd0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-j4sbd" (UID: "f249368d-c49d-4bfe-906f-d1d12104dbd0") : secret "cluster-monitoring-operator-tls" not found Jul 23 11:36:23.731940 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:23.731899 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/74038c52-71a3-4c88-b607-7f0f9ec11001-metrics-tls\") pod \"dns-default-fkvjh\" (UID: \"74038c52-71a3-4c88-b607-7f0f9ec11001\") " pod="openshift-dns/dns-default-fkvjh" Jul 23 11:36:23.732095 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:23.732019 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/89ae8727-bb42-4860-a957-dd9a06933722-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-n2rn6\" (UID: \"89ae8727-bb42-4860-a957-dd9a06933722\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-n2rn6" Jul 23 11:36:23.732095 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:23.732029 2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 11:36:23.732095 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:23.732073 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74038c52-71a3-4c88-b607-7f0f9ec11001-metrics-tls podName:74038c52-71a3-4c88-b607-7f0f9ec11001 nodeName:}" failed. No retries permitted until 2026-07-23 11:36:39.732060795 +0000 UTC m=+66.803750764 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/74038c52-71a3-4c88-b607-7f0f9ec11001-metrics-tls") pod "dns-default-fkvjh" (UID: "74038c52-71a3-4c88-b607-7f0f9ec11001") : secret "dns-default-metrics-tls" not found Jul 23 11:36:23.732247 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:23.732121 2561 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 23 11:36:23.732247 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:23.732167 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89ae8727-bb42-4860-a957-dd9a06933722-samples-operator-tls podName:89ae8727-bb42-4860-a957-dd9a06933722 nodeName:}" failed. No retries permitted until 2026-07-23 11:36:39.732153949 +0000 UTC m=+66.803843925 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/89ae8727-bb42-4860-a957-dd9a06933722-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-n2rn6" (UID: "89ae8727-bb42-4860-a957-dd9a06933722") : secret "samples-operator-tls" not found Jul 23 11:36:25.951096 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:25.951067 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1ae0a33f-4691-4a97-aea8-70267e2a8057-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4rfbr\" (UID: \"1ae0a33f-4691-4a97-aea8-70267e2a8057\") " pod="openshift-insights/insights-runtime-extractor-4rfbr" Jul 23 11:36:25.951096 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:25.951098 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1ae0a33f-4691-4a97-aea8-70267e2a8057-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4rfbr\" (UID: \"1ae0a33f-4691-4a97-aea8-70267e2a8057\") " pod="openshift-insights/insights-runtime-extractor-4rfbr" Jul 23 11:36:25.951597 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:25.951204 2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 23 11:36:25.951597 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:36:25.951257 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ae0a33f-4691-4a97-aea8-70267e2a8057-insights-runtime-extractor-tls podName:1ae0a33f-4691-4a97-aea8-70267e2a8057 nodeName:}" failed. No retries permitted until 2026-07-23 11:36:33.951242616 +0000 UTC m=+61.022932584 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1ae0a33f-4691-4a97-aea8-70267e2a8057-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4rfbr" (UID: "1ae0a33f-4691-4a97-aea8-70267e2a8057") : secret "insights-runtime-extractor-tls" not found Jul 23 11:36:25.951699 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:25.951677 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1ae0a33f-4691-4a97-aea8-70267e2a8057-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4rfbr\" (UID: \"1ae0a33f-4691-4a97-aea8-70267e2a8057\") " pod="openshift-insights/insights-runtime-extractor-4rfbr" Jul 23 11:36:33.645985 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:33.645959 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-92n98" Jul 23 11:36:33.683262 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:33.683214 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-58598695f7-jzxsm" podStartSLOduration=13.683199968 podStartE2EDuration="13.683199968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:36:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:36:21.830233651 +0000 UTC m=+48.901923641" watchObservedRunningTime="2026-07-23 11:36:33.683199968 +0000 UTC m=+60.754889957" Jul 23 11:36:34.012695 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:34.012636 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1ae0a33f-4691-4a97-aea8-70267e2a8057-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4rfbr\" (UID: \"1ae0a33f-4691-4a97-aea8-70267e2a8057\") " pod="openshift-insights/insights-runtime-extractor-4rfbr" Jul 23 11:36:34.014718 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:34.014701 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1ae0a33f-4691-4a97-aea8-70267e2a8057-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4rfbr\" (UID: \"1ae0a33f-4691-4a97-aea8-70267e2a8057\") " pod="openshift-insights/insights-runtime-extractor-4rfbr" Jul 23 11:36:34.129829 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:34.129811 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jxxcm\"" Jul 23 11:36:34.136434 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:34.136419 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4rfbr" Jul 23 11:36:34.262241 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:34.262220 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4rfbr"] Jul 23 11:36:34.264506 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:36:34.264444 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ae0a33f_4691_4a97_aea8_70267e2a8057.slice/crio-4ef789431114d21a6e0f1cc8069b0c9549c8967c4e54d70aa65755aa8f5dc988 WatchSource:0}: Error finding container 4ef789431114d21a6e0f1cc8069b0c9549c8967c4e54d70aa65755aa8f5dc988: Status 404 returned error can't find the container with id 4ef789431114d21a6e0f1cc8069b0c9549c8967c4e54d70aa65755aa8f5dc988 Jul 23 11:36:34.449637 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:34.449617 2561 scope.go:117] "RemoveContainer" containerID="8df4d8334a61f9bc7e532e99b63420fbbca2afbd802c35635fef53a65d397939" Jul 23 11:36:34.815781 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:34.814887 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4rfbr" event={"ID":"1ae0a33f-4691-4a97-aea8-70267e2a8057","Type":"ContainerStarted","Data":"4ef789431114d21a6e0f1cc8069b0c9549c8967c4e54d70aa65755aa8f5dc988"} Jul 23 11:36:34.819867 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:34.817983 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-7slpj_65247e1f-fc10-4b40-af67-5e646f352263/console-operator/1.log" Jul 23 11:36:34.819867 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:34.818030 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-7slpj" event={"ID":"65247e1f-fc10-4b40-af67-5e646f352263","Type":"ContainerStarted","Data":"dabf8a6031d3cc1cb71b41c675b51bd9a20fce3a374475a9422af983ef575af8"} Jul 23 11:36:34.820536 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:34.820104 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-7slpj" Jul 23 11:36:35.438977 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:35.438721 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-565b7b569d-7slpj" Jul 23 11:36:35.463537 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:35.458233 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-565b7b569d-7slpj" podStartSLOduration=31.161429497 podStartE2EDuration="38.458215623s" totalImagesPullingTime="7.296786126s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:35:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:36:34.842064312 +0000 UTC m=+61.913754302" watchObservedRunningTime="2026-07-23 11:36:35.458215623 +0000 UTC m=+62.529905614" Jul 23 11:36:39.163183 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:39.163160 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8886ab91-1d26-47c4-835e-a2905af1aa16-metrics-certs\") pod \"network-metrics-daemon-x7q2v\" (UID: \"8886ab91-1d26-47c4-835e-a2905af1aa16\") " pod="openshift-multus/network-metrics-daemon-x7q2v" Jul 23 11:36:39.163433 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:39.163216 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1e281f98-6114-4c25-9b31-79c65f9100e2-original-pull-secret\") pod \"global-pull-secret-syncer-8df69\" (UID: \"1e281f98-6114-4c25-9b31-79c65f9100e2\") " pod="kube-system/global-pull-secret-syncer-8df69" Jul 23 11:36:39.166193 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:39.166100 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 23 11:36:39.166193 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:39.166128 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 23 11:36:39.175487 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:39.175469 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1e281f98-6114-4c25-9b31-79c65f9100e2-original-pull-secret\") pod \"global-pull-secret-syncer-8df69\" (UID: \"1e281f98-6114-4c25-9b31-79c65f9100e2\") " pod="kube-system/global-pull-secret-syncer-8df69" Jul 23 11:36:39.175614 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:39.175589 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8886ab91-1d26-47c4-835e-a2905af1aa16-metrics-certs\") pod \"network-metrics-daemon-x7q2v\" (UID: \"8886ab91-1d26-47c4-835e-a2905af1aa16\") " pod="openshift-multus/network-metrics-daemon-x7q2v" Jul 23 11:36:39.263795 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:39.263770 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnlxz\" (UniqueName: \"kubernetes.io/projected/c92ad538-9642-4c08-b92c-886738d3c413-kube-api-access-gnlxz\") pod \"network-check-target-fv772\" (UID: \"c92ad538-9642-4c08-b92c-886738d3c413\") " pod="openshift-network-diagnostics/network-check-target-fv772" Jul 23 11:36:39.266115 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:39.266097 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnlxz\" (UniqueName: \"kubernetes.io/projected/c92ad538-9642-4c08-b92c-886738d3c413-kube-api-access-gnlxz\") pod \"network-check-target-fv772\" (UID: \"c92ad538-9642-4c08-b92c-886738d3c413\") " pod="openshift-network-diagnostics/network-check-target-fv772" Jul 23 11:36:39.359727 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:39.359703 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8df69" Jul 23 11:36:39.462114 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:39.462090 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7flkj\"" Jul 23 11:36:39.466667 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:39.466644 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xtz4b\"" Jul 23 11:36:39.469955 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:39.469900 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7q2v" Jul 23 11:36:39.475106 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:39.475087 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fv772" Jul 23 11:36:39.491621 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:39.491143 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8df69"] Jul 23 11:36:39.495385 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:36:39.495359 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e281f98_6114_4c25_9b31_79c65f9100e2.slice/crio-27f2dd99b6dd20605fcfa2a433b16e9269398f5da84276994218a806d4e55fc6 WatchSource:0}: Error finding container 27f2dd99b6dd20605fcfa2a433b16e9269398f5da84276994218a806d4e55fc6: Status 404 returned error can't find the container with id 27f2dd99b6dd20605fcfa2a433b16e9269398f5da84276994218a806d4e55fc6 Jul 23 11:36:39.633429 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:39.633365 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-x7q2v"] Jul 23 11:36:39.635869 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:36:39.635829 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8886ab91_1d26_47c4_835e_a2905af1aa16.slice/crio-41bfed58201b1c9a2b43ff517f7e9eabe8fffe0d24722ff9e0babc0d0a6da5c0 WatchSource:0}: Error finding container 41bfed58201b1c9a2b43ff517f7e9eabe8fffe0d24722ff9e0babc0d0a6da5c0: Status 404 returned error can't find the container with id 41bfed58201b1c9a2b43ff517f7e9eabe8fffe0d24722ff9e0babc0d0a6da5c0 Jul 23 11:36:39.668059 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:39.667960 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5179eda-de1f-4d89-9e22-afb9d63d098b-metrics-certs\") pod \"router-default-78fdc88b76-5w9ql\" (UID: \"c5179eda-de1f-4d89-9e22-afb9d63d098b\") " pod="openshift-ingress/router-default-78fdc88b76-5w9ql" Jul 23 11:36:39.668059 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:39.668026 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5179eda-de1f-4d89-9e22-afb9d63d098b-service-ca-bundle\") pod \"router-default-78fdc88b76-5w9ql\" (UID: \"c5179eda-de1f-4d89-9e22-afb9d63d098b\") " pod="openshift-ingress/router-default-78fdc88b76-5w9ql" Jul 23 11:36:39.668217 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:39.668082 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b993c3b9-94b8-4c21-b391-6193659cf9ee-registry-tls\") pod \"image-registry-67fbb677d9-vf9k4\" (UID: \"b993c3b9-94b8-4c21-b391-6193659cf9ee\") " pod="openshift-image-registry/image-registry-67fbb677d9-vf9k4" Jul 23 11:36:39.668271 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:39.668219 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f249368d-c49d-4bfe-906f-d1d12104dbd0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-j4sbd\" (UID: \"f249368d-c49d-4bfe-906f-d1d12104dbd0\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-j4sbd" Jul 23 11:36:39.668982 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:39.668931 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5179eda-de1f-4d89-9e22-afb9d63d098b-service-ca-bundle\") pod \"router-default-78fdc88b76-5w9ql\" (UID: \"c5179eda-de1f-4d89-9e22-afb9d63d098b\") " pod="openshift-ingress/router-default-78fdc88b76-5w9ql" Jul 23 11:36:39.670807 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:39.670788 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5179eda-de1f-4d89-9e22-afb9d63d098b-metrics-certs\") pod \"router-default-78fdc88b76-5w9ql\" (UID: \"c5179eda-de1f-4d89-9e22-afb9d63d098b\") " pod="openshift-ingress/router-default-78fdc88b76-5w9ql" Jul 23 11:36:39.670807 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:39.670795 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b993c3b9-94b8-4c21-b391-6193659cf9ee-registry-tls\") pod \"image-registry-67fbb677d9-vf9k4\" (UID: \"b993c3b9-94b8-4c21-b391-6193659cf9ee\") " pod="openshift-image-registry/image-registry-67fbb677d9-vf9k4" Jul 23 11:36:39.671062 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:39.671040 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f249368d-c49d-4bfe-906f-d1d12104dbd0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-j4sbd\" (UID: \"f249368d-c49d-4bfe-906f-d1d12104dbd0\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-j4sbd" Jul 23 11:36:39.684298 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:39.684279 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fv772"] Jul 23 11:36:39.687061 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:36:39.687035 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc92ad538_9642_4c08_b92c_886738d3c413.slice/crio-1c7226df61d6e5c1963fab767782271551e3ad38399dd0cc17e3e08ed145f5eb WatchSource:0}: Error finding container 1c7226df61d6e5c1963fab767782271551e3ad38399dd0cc17e3e08ed145f5eb: Status 404 returned error can't find the container with id 1c7226df61d6e5c1963fab767782271551e3ad38399dd0cc17e3e08ed145f5eb Jul 23 11:36:39.769189 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:39.769160 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/74038c52-71a3-4c88-b607-7f0f9ec11001-metrics-tls\") pod \"dns-default-fkvjh\" (UID: \"74038c52-71a3-4c88-b607-7f0f9ec11001\") " pod="openshift-dns/dns-default-fkvjh" Jul 23 11:36:39.769281 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:39.769230 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/89ae8727-bb42-4860-a957-dd9a06933722-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-n2rn6\" (UID: \"89ae8727-bb42-4860-a957-dd9a06933722\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-n2rn6" Jul 23 11:36:39.771756 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:39.771729 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/74038c52-71a3-4c88-b607-7f0f9ec11001-metrics-tls\") pod \"dns-default-fkvjh\" (UID: \"74038c52-71a3-4c88-b607-7f0f9ec11001\") " pod="openshift-dns/dns-default-fkvjh" Jul 23 11:36:39.772022 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:39.772004 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/89ae8727-bb42-4860-a957-dd9a06933722-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-n2rn6\" (UID: \"89ae8727-bb42-4860-a957-dd9a06933722\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-n2rn6" Jul 23 11:36:39.838252 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:39.838221 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fv772" event={"ID":"c92ad538-9642-4c08-b92c-886738d3c413","Type":"ContainerStarted","Data":"c75acdca3b3adac8abe8faba5c0e18caedfd35734c21806e66ffdc5032e3461e"} Jul 23 11:36:39.838400 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:39.838283 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fv772" event={"ID":"c92ad538-9642-4c08-b92c-886738d3c413","Type":"ContainerStarted","Data":"1c7226df61d6e5c1963fab767782271551e3ad38399dd0cc17e3e08ed145f5eb"} Jul 23 11:36:39.838466 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:39.838410 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-fv772" Jul 23 11:36:39.839854 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:39.839823 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4rfbr" event={"ID":"1ae0a33f-4691-4a97-aea8-70267e2a8057","Type":"ContainerStarted","Data":"83e492da1fcfc1e6fd24c1d029f057fd643139f32e24495298de9638f6f53694"} Jul 23 11:36:39.841260 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:39.841238 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-x7q2v" event={"ID":"8886ab91-1d26-47c4-835e-a2905af1aa16","Type":"ContainerStarted","Data":"41bfed58201b1c9a2b43ff517f7e9eabe8fffe0d24722ff9e0babc0d0a6da5c0"} Jul 23 11:36:39.842338 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:39.842316 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8df69" event={"ID":"1e281f98-6114-4c25-9b31-79c65f9100e2","Type":"ContainerStarted","Data":"27f2dd99b6dd20605fcfa2a433b16e9269398f5da84276994218a806d4e55fc6"} Jul 23 11:36:39.859956 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:39.859901 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-fv772" podStartSLOduration=66.859888313 podStartE2EDuration="1m6.859888313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:35:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:36:39.858132659 +0000 UTC m=+66.929822651" watchObservedRunningTime="2026-07-23 11:36:39.859888313 +0000 UTC m=+66.931578302" Jul 23 11:36:39.905582 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:39.905560 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-lfpvr\"" Jul 23 11:36:39.913731 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:39.913699 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-j4sbd" Jul 23 11:36:39.927640 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:39.927474 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-pmn7v\"" Jul 23 11:36:39.935600 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:39.935579 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67fbb677d9-vf9k4" Jul 23 11:36:39.939738 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:39.939719 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-j9zgf\"" Jul 23 11:36:39.947216 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:39.947170 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-78fdc88b76-5w9ql" Jul 23 11:36:39.950487 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:39.950467 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-qfq6r\"" Jul 23 11:36:39.957558 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:39.957538 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-n2rn6" Jul 23 11:36:39.992162 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:39.992077 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-67ffh\"" Jul 23 11:36:39.999518 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:39.999261 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fkvjh" Jul 23 11:36:40.110106 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:40.106418 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-j4sbd"] Jul 23 11:36:40.189698 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:40.189472 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-67fbb677d9-vf9k4"] Jul 23 11:36:40.194179 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:36:40.193969 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb993c3b9_94b8_4c21_b391_6193659cf9ee.slice/crio-70035628e2207ba158d90a1b31f6e44113814bd7c336e4553a9985ed98ea6d4a WatchSource:0}: Error finding container 70035628e2207ba158d90a1b31f6e44113814bd7c336e4553a9985ed98ea6d4a: Status 404 returned error can't find the container with id 70035628e2207ba158d90a1b31f6e44113814bd7c336e4553a9985ed98ea6d4a Jul 23 11:36:40.215968 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:40.215570 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-78fdc88b76-5w9ql"] Jul 23 11:36:40.220459 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:36:40.220420 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5179eda_de1f_4d89_9e22_afb9d63d098b.slice/crio-88f66b408e90608e31675d8ce591b9e9acfe5bf62668bde166c5c0b6446c283c WatchSource:0}: Error finding container 88f66b408e90608e31675d8ce591b9e9acfe5bf62668bde166c5c0b6446c283c: Status 404 returned error can't find the container with id 88f66b408e90608e31675d8ce591b9e9acfe5bf62668bde166c5c0b6446c283c Jul 23 11:36:40.242861 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:40.242833 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-n2rn6"] Jul 23 11:36:40.262019 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:40.260763 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fkvjh"] Jul 23 11:36:40.273244 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:36:40.273215 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74038c52_71a3_4c88_b607_7f0f9ec11001.slice/crio-8c7135ae7a9670da277121d1eb28bdb89e685c3b0fec878cfb7e693a48687701 WatchSource:0}: Error finding container 8c7135ae7a9670da277121d1eb28bdb89e685c3b0fec878cfb7e693a48687701: Status 404 returned error can't find the container with id 8c7135ae7a9670da277121d1eb28bdb89e685c3b0fec878cfb7e693a48687701 Jul 23 11:36:40.850832 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:40.850732 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-n2rn6" event={"ID":"89ae8727-bb42-4860-a957-dd9a06933722","Type":"ContainerStarted","Data":"d312334eeff76a86fcb92e51cd7e9656209bc991d3d8f9b322820352470c7bbd"} Jul 23 11:36:40.854564 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:40.853972 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-78fdc88b76-5w9ql" event={"ID":"c5179eda-de1f-4d89-9e22-afb9d63d098b","Type":"ContainerStarted","Data":"7f2884aa75d9222676d5da1d4114723058d2b4da83c886e13933026c7a8adfe2"} Jul 23 11:36:40.854564 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:40.854023 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-78fdc88b76-5w9ql" event={"ID":"c5179eda-de1f-4d89-9e22-afb9d63d098b","Type":"ContainerStarted","Data":"88f66b408e90608e31675d8ce591b9e9acfe5bf62668bde166c5c0b6446c283c"} Jul 23 11:36:40.856025 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:40.855890 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67fbb677d9-vf9k4" event={"ID":"b993c3b9-94b8-4c21-b391-6193659cf9ee","Type":"ContainerStarted","Data":"47f2dba388a36ddcff377d4f21a85aeb1cbecbd3bc072c13453aafeb992f92c4"} Jul 23 11:36:40.856025 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:40.855951 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67fbb677d9-vf9k4" event={"ID":"b993c3b9-94b8-4c21-b391-6193659cf9ee","Type":"ContainerStarted","Data":"70035628e2207ba158d90a1b31f6e44113814bd7c336e4553a9985ed98ea6d4a"} Jul 23 11:36:40.856323 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:40.856289 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-67fbb677d9-vf9k4" Jul 23 11:36:40.857390 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:40.857336 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-j4sbd" event={"ID":"f249368d-c49d-4bfe-906f-d1d12104dbd0","Type":"ContainerStarted","Data":"f34da0b4a0e9d23f036e81659fd43cf39e104bf6a3f6f833feccb47edf1380c3"} Jul 23 11:36:40.859890 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:40.859543 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fkvjh" event={"ID":"74038c52-71a3-4c88-b607-7f0f9ec11001","Type":"ContainerStarted","Data":"8c7135ae7a9670da277121d1eb28bdb89e685c3b0fec878cfb7e693a48687701"} Jul 23 11:36:40.886923 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:40.885409 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-78fdc88b76-5w9ql" podStartSLOduration=43.885388534 podStartE2EDuration="43.885388534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:35:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:36:40.884110186 +0000 UTC m=+67.955800176" watchObservedRunningTime="2026-07-23 11:36:40.885388534 +0000 UTC m=+67.957078525" Jul 23 11:36:40.919639 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:40.919576 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-67fbb677d9-vf9k4" podStartSLOduration=67.919554864 podStartE2EDuration="1m7.919554864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:35:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:36:40.919122891 +0000 UTC m=+67.990812886" watchObservedRunningTime="2026-07-23 11:36:40.919554864 +0000 UTC m=+67.991244857" Jul 23 11:36:40.947858 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:40.947830 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-78fdc88b76-5w9ql" Jul 23 11:36:40.951314 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:40.951136 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-78fdc88b76-5w9ql" Jul 23 11:36:41.867014 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:41.866957 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-78fdc88b76-5w9ql" Jul 23 11:36:41.868450 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:41.868427 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-78fdc88b76-5w9ql" Jul 23 11:36:48.898410 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:48.898374 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-j4sbd" event={"ID":"f249368d-c49d-4bfe-906f-d1d12104dbd0","Type":"ContainerStarted","Data":"f399a76bff03f55b330c8de185256384be29a96b9e1a76e5d4f72f0fca5d9467"} Jul 23 11:36:48.900174 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:48.900137 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-x7q2v" event={"ID":"8886ab91-1d26-47c4-835e-a2905af1aa16","Type":"ContainerStarted","Data":"27b21425241f1ec3eac0c4781ad51c318a376d1234beb21b68ac6463d233035b"} Jul 23 11:36:48.900290 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:48.900183 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-x7q2v" event={"ID":"8886ab91-1d26-47c4-835e-a2905af1aa16","Type":"ContainerStarted","Data":"5da3a79186b125ea316b1bf9b4258b99367930f616941f48ed1fd2fcfc834d09"} Jul 23 11:36:48.901648 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:48.901623 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fkvjh" event={"ID":"74038c52-71a3-4c88-b607-7f0f9ec11001","Type":"ContainerStarted","Data":"13bf36dda7e24c9c4bfb1a01d8be3a97a8962c6716df9ce27aa9a31d70d4d20c"} Jul 23 11:36:48.901791 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:48.901672 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fkvjh" event={"ID":"74038c52-71a3-4c88-b607-7f0f9ec11001","Type":"ContainerStarted","Data":"7d64693b41a487fcea069a15024ccff3c95edba3619828eae7b5d886d95e8e08"} Jul 23 11:36:48.901791 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:48.901747 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-fkvjh" Jul 23 11:36:48.903314 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:48.903293 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4rfbr" event={"ID":"1ae0a33f-4691-4a97-aea8-70267e2a8057","Type":"ContainerStarted","Data":"6b5de0ae3007cb6bb7cad65b5620d843a8caebb7fabf2312e61cacdc9f4bd526"} Jul 23 11:36:48.904653 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:48.904630 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8df69" event={"ID":"1e281f98-6114-4c25-9b31-79c65f9100e2","Type":"ContainerStarted","Data":"2a15811b91ede0ea2752d0eada9c6e3d63b9d1651403c36945076d556a21a48c"} Jul 23 11:36:48.906233 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:48.906215 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-n2rn6" event={"ID":"89ae8727-bb42-4860-a957-dd9a06933722","Type":"ContainerStarted","Data":"a2e93fe998b2202e7fb0a6d2d237716cadbddd68e52f7868024e7d77fe253b6d"} Jul 23 11:36:48.906317 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:48.906247 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-n2rn6" event={"ID":"89ae8727-bb42-4860-a957-dd9a06933722","Type":"ContainerStarted","Data":"24471bc44cca29935c77be9b22016dc3ef495a8469bece556c8fce4cf13c4f51"} Jul 23 11:36:48.917394 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:48.917354 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-j4sbd" podStartSLOduration=44.009877481 podStartE2EDuration="51.917344146s" totalImagesPullingTime="7.907466665s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:35:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:36:48.91641555 +0000 UTC m=+75.988105541" watchObservedRunningTime="2026-07-23 11:36:48.917344146 +0000 UTC m=+75.989034136" Jul 23 11:36:48.939699 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:48.939664 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-fkvjh" podStartSLOduration=34.170158386 podStartE2EDuration="41.939648145s" totalImagesPullingTime="7.769489759s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:36:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:36:48.938886369 +0000 UTC m=+76.010576360" watchObservedRunningTime="2026-07-23 11:36:48.939648145 +0000 UTC m=+76.011338113" Jul 23 11:36:48.954841 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:48.954800 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-8df69" podStartSLOduration=67.403990274 podStartE2EDuration="1m15.954783741s" totalImagesPullingTime="8.550793467s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 11:35:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:36:48.953430683 +0000 UTC m=+76.025120675" watchObservedRunningTime="2026-07-23 11:36:48.954783741 +0000 UTC m=+76.026473732" Jul 23 11:36:48.971186 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:48.971151 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-n2rn6" podStartSLOduration=44.245600175 podStartE2EDuration="51.971140327s" totalImagesPullingTime="7.725540152s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:35:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:36:48.969801368 +0000 UTC m=+76.041491359" watchObservedRunningTime="2026-07-23 11:36:48.971140327 +0000 UTC m=+76.042830317" Jul 23 11:36:48.990707 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:48.990660 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-x7q2v" podStartSLOduration=67.739636827 podStartE2EDuration="1m15.990649143s" totalImagesPullingTime="8.251012316s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:35:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:36:48.989686377 +0000 UTC m=+76.061376369" watchObservedRunningTime="2026-07-23 11:36:48.990649143 +0000 UTC m=+76.062339143" Jul 23 11:36:51.920548 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:51.920523 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4rfbr" event={"ID":"1ae0a33f-4691-4a97-aea8-70267e2a8057","Type":"ContainerStarted","Data":"6792a8b42bf209ac16439d4491dd0924e8443b87892e1b971ed0ab5bc96033ab"} Jul 23 11:36:51.941560 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:51.941511 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-4rfbr" podStartSLOduration=16.575413037 podStartE2EDuration="33.941496667s" totalImagesPullingTime="17.36608363s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 11:36:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:36:51.940095492 +0000 UTC m=+79.011785498" watchObservedRunningTime="2026-07-23 11:36:51.941496667 +0000 UTC m=+79.013186657" Jul 23 11:36:58.912867 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:36:58.912740 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-fkvjh" Jul 23 11:37:01.293427 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:01.293394 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-ptljk"] Jul 23 11:37:01.297024 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:01.297002 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ptljk" Jul 23 11:37:01.300207 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:01.300188 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 23 11:37:01.300767 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:01.300752 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 23 11:37:01.305867 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:01.305844 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 23 11:37:01.306024 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:01.305846 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 23 11:37:01.310720 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:01.310700 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-mhh7m\"" Jul 23 11:37:01.431330 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:01.431303 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bd1684db-f985-4971-b821-eb4a6b9a1b50-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ptljk\" (UID: \"bd1684db-f985-4971-b821-eb4a6b9a1b50\") " pod="openshift-monitoring/node-exporter-ptljk" Jul 23 11:37:01.431427 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:01.431345 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bd1684db-f985-4971-b821-eb4a6b9a1b50-node-exporter-accelerators-collector-config\") pod \"node-exporter-ptljk\" (UID: \"bd1684db-f985-4971-b821-eb4a6b9a1b50\") " pod="openshift-monitoring/node-exporter-ptljk" Jul 23 11:37:01.431427 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:01.431369 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsgl7\" (UniqueName: \"kubernetes.io/projected/bd1684db-f985-4971-b821-eb4a6b9a1b50-kube-api-access-fsgl7\") pod \"node-exporter-ptljk\" (UID: \"bd1684db-f985-4971-b821-eb4a6b9a1b50\") " pod="openshift-monitoring/node-exporter-ptljk" Jul 23 11:37:01.431427 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:01.431418 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bd1684db-f985-4971-b821-eb4a6b9a1b50-node-exporter-tls\") pod \"node-exporter-ptljk\" (UID: \"bd1684db-f985-4971-b821-eb4a6b9a1b50\") " pod="openshift-monitoring/node-exporter-ptljk" Jul 23 11:37:01.431572 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:01.431449 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bd1684db-f985-4971-b821-eb4a6b9a1b50-metrics-client-ca\") pod \"node-exporter-ptljk\" (UID: \"bd1684db-f985-4971-b821-eb4a6b9a1b50\") " pod="openshift-monitoring/node-exporter-ptljk" Jul 23 11:37:01.431572 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:01.431468 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bd1684db-f985-4971-b821-eb4a6b9a1b50-sys\") pod \"node-exporter-ptljk\" (UID: \"bd1684db-f985-4971-b821-eb4a6b9a1b50\") " pod="openshift-monitoring/node-exporter-ptljk" Jul 23 11:37:01.431572 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:01.431494 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bd1684db-f985-4971-b821-eb4a6b9a1b50-node-exporter-textfile\") pod \"node-exporter-ptljk\" (UID: \"bd1684db-f985-4971-b821-eb4a6b9a1b50\") " pod="openshift-monitoring/node-exporter-ptljk" Jul 23 11:37:01.431572 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:01.431525 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bd1684db-f985-4971-b821-eb4a6b9a1b50-node-exporter-wtmp\") pod \"node-exporter-ptljk\" (UID: \"bd1684db-f985-4971-b821-eb4a6b9a1b50\") " pod="openshift-monitoring/node-exporter-ptljk" Jul 23 11:37:01.431717 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:01.431591 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bd1684db-f985-4971-b821-eb4a6b9a1b50-root\") pod \"node-exporter-ptljk\" (UID: \"bd1684db-f985-4971-b821-eb4a6b9a1b50\") " pod="openshift-monitoring/node-exporter-ptljk" Jul 23 11:37:01.532249 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:01.532218 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bd1684db-f985-4971-b821-eb4a6b9a1b50-node-exporter-wtmp\") pod \"node-exporter-ptljk\" (UID: \"bd1684db-f985-4971-b821-eb4a6b9a1b50\") " pod="openshift-monitoring/node-exporter-ptljk" Jul 23 11:37:01.532338 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:01.532275 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bd1684db-f985-4971-b821-eb4a6b9a1b50-root\") pod \"node-exporter-ptljk\" (UID: \"bd1684db-f985-4971-b821-eb4a6b9a1b50\") " pod="openshift-monitoring/node-exporter-ptljk" Jul 23 11:37:01.532338 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:01.532310 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bd1684db-f985-4971-b821-eb4a6b9a1b50-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ptljk\" (UID: \"bd1684db-f985-4971-b821-eb4a6b9a1b50\") " pod="openshift-monitoring/node-exporter-ptljk" Jul 23 11:37:01.532444 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:01.532338 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bd1684db-f985-4971-b821-eb4a6b9a1b50-node-exporter-accelerators-collector-config\") pod \"node-exporter-ptljk\" (UID: \"bd1684db-f985-4971-b821-eb4a6b9a1b50\") " pod="openshift-monitoring/node-exporter-ptljk" Jul 23 11:37:01.532444 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:01.532361 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsgl7\" (UniqueName: \"kubernetes.io/projected/bd1684db-f985-4971-b821-eb4a6b9a1b50-kube-api-access-fsgl7\") pod \"node-exporter-ptljk\" (UID: \"bd1684db-f985-4971-b821-eb4a6b9a1b50\") " pod="openshift-monitoring/node-exporter-ptljk" Jul 23 11:37:01.532444 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:01.532387 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bd1684db-f985-4971-b821-eb4a6b9a1b50-node-exporter-tls\") pod \"node-exporter-ptljk\" (UID: \"bd1684db-f985-4971-b821-eb4a6b9a1b50\") " pod="openshift-monitoring/node-exporter-ptljk" Jul 23 11:37:01.532444 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:01.532389 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bd1684db-f985-4971-b821-eb4a6b9a1b50-node-exporter-wtmp\") pod \"node-exporter-ptljk\" (UID: \"bd1684db-f985-4971-b821-eb4a6b9a1b50\") " pod="openshift-monitoring/node-exporter-ptljk" Jul 23 11:37:01.532444 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:01.532403 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bd1684db-f985-4971-b821-eb4a6b9a1b50-root\") pod \"node-exporter-ptljk\" (UID: \"bd1684db-f985-4971-b821-eb4a6b9a1b50\") " pod="openshift-monitoring/node-exporter-ptljk" Jul 23 11:37:01.532444 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:01.532420 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bd1684db-f985-4971-b821-eb4a6b9a1b50-metrics-client-ca\") pod \"node-exporter-ptljk\" (UID: \"bd1684db-f985-4971-b821-eb4a6b9a1b50\") " pod="openshift-monitoring/node-exporter-ptljk" Jul 23 11:37:01.532703 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:01.532458 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bd1684db-f985-4971-b821-eb4a6b9a1b50-sys\") pod \"node-exporter-ptljk\" (UID: \"bd1684db-f985-4971-b821-eb4a6b9a1b50\") " pod="openshift-monitoring/node-exporter-ptljk" Jul 23 11:37:01.532703 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:01.532489 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bd1684db-f985-4971-b821-eb4a6b9a1b50-node-exporter-textfile\") pod \"node-exporter-ptljk\" (UID: \"bd1684db-f985-4971-b821-eb4a6b9a1b50\") " pod="openshift-monitoring/node-exporter-ptljk" Jul 23 11:37:01.532703 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:01.532608 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bd1684db-f985-4971-b821-eb4a6b9a1b50-sys\") pod \"node-exporter-ptljk\" (UID: \"bd1684db-f985-4971-b821-eb4a6b9a1b50\") " pod="openshift-monitoring/node-exporter-ptljk" Jul 23 11:37:01.533021 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:01.532971 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bd1684db-f985-4971-b821-eb4a6b9a1b50-node-exporter-accelerators-collector-config\") pod \"node-exporter-ptljk\" (UID: \"bd1684db-f985-4971-b821-eb4a6b9a1b50\") " pod="openshift-monitoring/node-exporter-ptljk" Jul 23 11:37:01.533021 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:01.532986 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bd1684db-f985-4971-b821-eb4a6b9a1b50-metrics-client-ca\") pod \"node-exporter-ptljk\" (UID: \"bd1684db-f985-4971-b821-eb4a6b9a1b50\") " pod="openshift-monitoring/node-exporter-ptljk" Jul 23 11:37:01.533321 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:01.533026 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bd1684db-f985-4971-b821-eb4a6b9a1b50-node-exporter-textfile\") pod \"node-exporter-ptljk\" (UID: \"bd1684db-f985-4971-b821-eb4a6b9a1b50\") " pod="openshift-monitoring/node-exporter-ptljk" Jul 23 11:37:01.534738 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:01.534719 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bd1684db-f985-4971-b821-eb4a6b9a1b50-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ptljk\" (UID: \"bd1684db-f985-4971-b821-eb4a6b9a1b50\") " pod="openshift-monitoring/node-exporter-ptljk" Jul 23 11:37:01.534882 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:01.534862 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bd1684db-f985-4971-b821-eb4a6b9a1b50-node-exporter-tls\") pod \"node-exporter-ptljk\" (UID: \"bd1684db-f985-4971-b821-eb4a6b9a1b50\") " pod="openshift-monitoring/node-exporter-ptljk" Jul 23 11:37:01.542538 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:01.542513 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsgl7\" (UniqueName: \"kubernetes.io/projected/bd1684db-f985-4971-b821-eb4a6b9a1b50-kube-api-access-fsgl7\") pod \"node-exporter-ptljk\" (UID: \"bd1684db-f985-4971-b821-eb4a6b9a1b50\") " pod="openshift-monitoring/node-exporter-ptljk" Jul 23 11:37:01.606447 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:01.606430 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ptljk" Jul 23 11:37:01.616529 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:37:01.616508 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd1684db_f985_4971_b821_eb4a6b9a1b50.slice/crio-469bbc2d4e31715b2698b3e09fb9117d269abff50f48fb0e782b424aa339db62 WatchSource:0}: Error finding container 469bbc2d4e31715b2698b3e09fb9117d269abff50f48fb0e782b424aa339db62: Status 404 returned error can't find the container with id 469bbc2d4e31715b2698b3e09fb9117d269abff50f48fb0e782b424aa339db62 Jul 23 11:37:01.871161 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:01.871089 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-67fbb677d9-vf9k4" Jul 23 11:37:01.956130 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:01.956093 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ptljk" event={"ID":"bd1684db-f985-4971-b821-eb4a6b9a1b50","Type":"ContainerStarted","Data":"469bbc2d4e31715b2698b3e09fb9117d269abff50f48fb0e782b424aa339db62"} Jul 23 11:37:02.961177 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:02.961113 2561 generic.go:356] "Generic (PLEG): container finished" podID="bd1684db-f985-4971-b821-eb4a6b9a1b50" containerID="5e8637f413b271672387b993094290a72623b433029bae6b179616d0a362800f" exitCode=0 Jul 23 11:37:02.961177 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:02.961166 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ptljk" event={"ID":"bd1684db-f985-4971-b821-eb4a6b9a1b50","Type":"ContainerDied","Data":"5e8637f413b271672387b993094290a72623b433029bae6b179616d0a362800f"} Jul 23 11:37:03.473936 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:03.473883 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-67fbb677d9-vf9k4"] Jul 23 11:37:03.967002 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:03.966969 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ptljk" event={"ID":"bd1684db-f985-4971-b821-eb4a6b9a1b50","Type":"ContainerStarted","Data":"f182e94fa155bdd1374d120c981fc7aabdc6dd3aa36e42f12ac2346ff642f948"} Jul 23 11:37:03.967356 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:03.967022 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ptljk" event={"ID":"bd1684db-f985-4971-b821-eb4a6b9a1b50","Type":"ContainerStarted","Data":"134d204f5f11627b76ba8b0a31a5417f83e427b1a572bbfa3ae9f7ac3089ff78"} Jul 23 11:37:04.003567 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:04.003519 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-ptljk" podStartSLOduration=1.9086742509999999 podStartE2EDuration="3.003504986s" totalImagesPullingTime="1.026264735s" totalInitContainerRuntime="68.566ms" isStatefulPod=true podCreationTimestamp="2026-07-23 11:37:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:37:04.002560777 +0000 UTC m=+91.074250767" watchObservedRunningTime="2026-07-23 11:37:04.003504986 +0000 UTC m=+91.075194979" Jul 23 11:37:10.861375 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:10.861347 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-fv772" Jul 23 11:37:28.497386 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:28.497346 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-67fbb677d9-vf9k4" podUID="b993c3b9-94b8-4c21-b391-6193659cf9ee" containerName="registry" containerID="cri-o://47f2dba388a36ddcff377d4f21a85aeb1cbecbd3bc072c13453aafeb992f92c4" gracePeriod=30 Jul 23 11:37:28.730779 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:28.730758 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67fbb677d9-vf9k4" Jul 23 11:37:28.819395 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:28.819372 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b993c3b9-94b8-4c21-b391-6193659cf9ee-image-registry-private-configuration\") pod \"b993c3b9-94b8-4c21-b391-6193659cf9ee\" (UID: \"b993c3b9-94b8-4c21-b391-6193659cf9ee\") " Jul 23 11:37:28.819553 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:28.819412 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b993c3b9-94b8-4c21-b391-6193659cf9ee-installation-pull-secrets\") pod \"b993c3b9-94b8-4c21-b391-6193659cf9ee\" (UID: \"b993c3b9-94b8-4c21-b391-6193659cf9ee\") " Jul 23 11:37:28.819553 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:28.819440 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b993c3b9-94b8-4c21-b391-6193659cf9ee-registry-tls\") pod \"b993c3b9-94b8-4c21-b391-6193659cf9ee\" (UID: \"b993c3b9-94b8-4c21-b391-6193659cf9ee\") " Jul 23 11:37:28.819553 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:28.819459 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dlkp9\" (UniqueName: \"kubernetes.io/projected/b993c3b9-94b8-4c21-b391-6193659cf9ee-kube-api-access-dlkp9\") pod \"b993c3b9-94b8-4c21-b391-6193659cf9ee\" (UID: \"b993c3b9-94b8-4c21-b391-6193659cf9ee\") " Jul 23 11:37:28.819553 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:28.819473 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b993c3b9-94b8-4c21-b391-6193659cf9ee-trusted-ca\") pod \"b993c3b9-94b8-4c21-b391-6193659cf9ee\" (UID: \"b993c3b9-94b8-4c21-b391-6193659cf9ee\") " Jul 23 11:37:28.819553 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:28.819491 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b993c3b9-94b8-4c21-b391-6193659cf9ee-ca-trust-extracted\") pod \"b993c3b9-94b8-4c21-b391-6193659cf9ee\" (UID: \"b993c3b9-94b8-4c21-b391-6193659cf9ee\") " Jul 23 11:37:28.819553 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:28.819515 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b993c3b9-94b8-4c21-b391-6193659cf9ee-bound-sa-token\") pod \"b993c3b9-94b8-4c21-b391-6193659cf9ee\" (UID: \"b993c3b9-94b8-4c21-b391-6193659cf9ee\") " Jul 23 11:37:28.819854 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:28.819769 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b993c3b9-94b8-4c21-b391-6193659cf9ee-registry-certificates\") pod \"b993c3b9-94b8-4c21-b391-6193659cf9ee\" (UID: \"b993c3b9-94b8-4c21-b391-6193659cf9ee\") " Jul 23 11:37:28.820045 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:28.820010 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b993c3b9-94b8-4c21-b391-6193659cf9ee-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "b993c3b9-94b8-4c21-b391-6193659cf9ee" (UID: "b993c3b9-94b8-4c21-b391-6193659cf9ee"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 11:37:28.820301 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:28.820277 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b993c3b9-94b8-4c21-b391-6193659cf9ee-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "b993c3b9-94b8-4c21-b391-6193659cf9ee" (UID: "b993c3b9-94b8-4c21-b391-6193659cf9ee"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 11:37:28.822667 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:28.822633 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b993c3b9-94b8-4c21-b391-6193659cf9ee-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "b993c3b9-94b8-4c21-b391-6193659cf9ee" (UID: "b993c3b9-94b8-4c21-b391-6193659cf9ee"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 11:37:28.822765 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:28.822678 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b993c3b9-94b8-4c21-b391-6193659cf9ee-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "b993c3b9-94b8-4c21-b391-6193659cf9ee" (UID: "b993c3b9-94b8-4c21-b391-6193659cf9ee"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 11:37:28.822765 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:28.822699 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b993c3b9-94b8-4c21-b391-6193659cf9ee-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "b993c3b9-94b8-4c21-b391-6193659cf9ee" (UID: "b993c3b9-94b8-4c21-b391-6193659cf9ee"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 11:37:28.822863 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:28.822770 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b993c3b9-94b8-4c21-b391-6193659cf9ee-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "b993c3b9-94b8-4c21-b391-6193659cf9ee" (UID: "b993c3b9-94b8-4c21-b391-6193659cf9ee"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 11:37:28.822863 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:28.822815 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b993c3b9-94b8-4c21-b391-6193659cf9ee-kube-api-access-dlkp9" (OuterVolumeSpecName: "kube-api-access-dlkp9") pod "b993c3b9-94b8-4c21-b391-6193659cf9ee" (UID: "b993c3b9-94b8-4c21-b391-6193659cf9ee"). InnerVolumeSpecName "kube-api-access-dlkp9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 11:37:28.828645 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:28.828624 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b993c3b9-94b8-4c21-b391-6193659cf9ee-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "b993c3b9-94b8-4c21-b391-6193659cf9ee" (UID: "b993c3b9-94b8-4c21-b391-6193659cf9ee"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 11:37:28.920265 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:28.920241 2561 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b993c3b9-94b8-4c21-b391-6193659cf9ee-bound-sa-token\") on node \"ip-10-0-134-236.ec2.internal\" DevicePath \"\"" Jul 23 11:37:28.920379 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:28.920265 2561 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b993c3b9-94b8-4c21-b391-6193659cf9ee-registry-certificates\") on node \"ip-10-0-134-236.ec2.internal\" DevicePath \"\"" Jul 23 11:37:28.920379 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:28.920280 2561 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b993c3b9-94b8-4c21-b391-6193659cf9ee-image-registry-private-configuration\") on node \"ip-10-0-134-236.ec2.internal\" DevicePath \"\"" Jul 23 11:37:28.920379 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:28.920292 2561 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b993c3b9-94b8-4c21-b391-6193659cf9ee-installation-pull-secrets\") on node \"ip-10-0-134-236.ec2.internal\" DevicePath \"\"" Jul 23 11:37:28.920379 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:28.920305 2561 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b993c3b9-94b8-4c21-b391-6193659cf9ee-registry-tls\") on node \"ip-10-0-134-236.ec2.internal\" DevicePath \"\"" Jul 23 11:37:28.920379 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:28.920317 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlkp9\" (UniqueName: \"kubernetes.io/projected/b993c3b9-94b8-4c21-b391-6193659cf9ee-kube-api-access-dlkp9\") on node \"ip-10-0-134-236.ec2.internal\" DevicePath \"\"" Jul 23 11:37:28.920379 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:28.920329 2561 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b993c3b9-94b8-4c21-b391-6193659cf9ee-trusted-ca\") on node \"ip-10-0-134-236.ec2.internal\" DevicePath \"\"" Jul 23 11:37:28.920379 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:28.920340 2561 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b993c3b9-94b8-4c21-b391-6193659cf9ee-ca-trust-extracted\") on node \"ip-10-0-134-236.ec2.internal\" DevicePath \"\"" Jul 23 11:37:29.055496 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:29.055471 2561 generic.go:356] "Generic (PLEG): container finished" podID="b993c3b9-94b8-4c21-b391-6193659cf9ee" containerID="47f2dba388a36ddcff377d4f21a85aeb1cbecbd3bc072c13453aafeb992f92c4" exitCode=0 Jul 23 11:37:29.055576 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:29.055530 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67fbb677d9-vf9k4" Jul 23 11:37:29.055576 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:29.055554 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67fbb677d9-vf9k4" event={"ID":"b993c3b9-94b8-4c21-b391-6193659cf9ee","Type":"ContainerDied","Data":"47f2dba388a36ddcff377d4f21a85aeb1cbecbd3bc072c13453aafeb992f92c4"} Jul 23 11:37:29.055659 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:29.055620 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67fbb677d9-vf9k4" event={"ID":"b993c3b9-94b8-4c21-b391-6193659cf9ee","Type":"ContainerDied","Data":"70035628e2207ba158d90a1b31f6e44113814bd7c336e4553a9985ed98ea6d4a"} Jul 23 11:37:29.055659 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:29.055640 2561 scope.go:117] "RemoveContainer" containerID="47f2dba388a36ddcff377d4f21a85aeb1cbecbd3bc072c13453aafeb992f92c4" Jul 23 11:37:29.063421 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:29.063406 2561 scope.go:117] "RemoveContainer" containerID="47f2dba388a36ddcff377d4f21a85aeb1cbecbd3bc072c13453aafeb992f92c4" Jul 23 11:37:29.063697 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:37:29.063672 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47f2dba388a36ddcff377d4f21a85aeb1cbecbd3bc072c13453aafeb992f92c4\": container with ID starting with 47f2dba388a36ddcff377d4f21a85aeb1cbecbd3bc072c13453aafeb992f92c4 not found: ID does not exist" containerID="47f2dba388a36ddcff377d4f21a85aeb1cbecbd3bc072c13453aafeb992f92c4" Jul 23 11:37:29.063783 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:29.063702 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47f2dba388a36ddcff377d4f21a85aeb1cbecbd3bc072c13453aafeb992f92c4"} err="failed to get container status \"47f2dba388a36ddcff377d4f21a85aeb1cbecbd3bc072c13453aafeb992f92c4\": rpc error: code = NotFound desc = could not find container \"47f2dba388a36ddcff377d4f21a85aeb1cbecbd3bc072c13453aafeb992f92c4\": container with ID starting with 47f2dba388a36ddcff377d4f21a85aeb1cbecbd3bc072c13453aafeb992f92c4 not found: ID does not exist" Jul 23 11:37:29.075085 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:29.075041 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-67fbb677d9-vf9k4"] Jul 23 11:37:29.079629 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:29.079611 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-67fbb677d9-vf9k4"] Jul 23 11:37:29.453496 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:29.453420 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b993c3b9-94b8-4c21-b391-6193659cf9ee" path="/var/lib/kubelet/pods/b993c3b9-94b8-4c21-b391-6193659cf9ee/volumes" Jul 23 11:37:33.070508 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:33.070477 2561 generic.go:356] "Generic (PLEG): container finished" podID="21319854-f159-442b-bf88-4498cec86096" containerID="0dbbd5387a70b304b338c702b8b7f441d0ad42e19732e93b3c2735128804c381" exitCode=0 Jul 23 11:37:33.070845 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:33.070537 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-5txgl" event={"ID":"21319854-f159-442b-bf88-4498cec86096","Type":"ContainerDied","Data":"0dbbd5387a70b304b338c702b8b7f441d0ad42e19732e93b3c2735128804c381"} Jul 23 11:37:33.070845 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:33.070808 2561 scope.go:117] "RemoveContainer" containerID="0dbbd5387a70b304b338c702b8b7f441d0ad42e19732e93b3c2735128804c381" Jul 23 11:37:34.075709 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:34.075681 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-5txgl" event={"ID":"21319854-f159-442b-bf88-4498cec86096","Type":"ContainerStarted","Data":"81afde53ee6d68ed6e2b93f7323c20cd0d1e2eb81305b68e9e3367898cea9686"} Jul 23 11:37:38.092604 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:38.092574 2561 generic.go:356] "Generic (PLEG): container finished" podID="98f50c4a-91e4-4708-be1a-cff80dfb6cc8" containerID="a4f45d2edccb24164e03cac100ef2ecc9b6d46500dbe308b19875b72b90b9e21" exitCode=0 Jul 23 11:37:38.093071 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:38.092650 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-sr6b2" event={"ID":"98f50c4a-91e4-4708-be1a-cff80dfb6cc8","Type":"ContainerDied","Data":"a4f45d2edccb24164e03cac100ef2ecc9b6d46500dbe308b19875b72b90b9e21"} Jul 23 11:37:38.093071 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:38.093019 2561 scope.go:117] "RemoveContainer" containerID="a4f45d2edccb24164e03cac100ef2ecc9b6d46500dbe308b19875b72b90b9e21" Jul 23 11:37:38.094273 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:38.094167 2561 generic.go:356] "Generic (PLEG): container finished" podID="b64d0632-5844-419e-9f53-ac57a0495a8d" containerID="2b93f10485091aa9685dd137dfe9afc70a4933e8eb4f9a5283a8e84bd443a70c" exitCode=0 Jul 23 11:37:38.094273 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:38.094193 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-48j4r" event={"ID":"b64d0632-5844-419e-9f53-ac57a0495a8d","Type":"ContainerDied","Data":"2b93f10485091aa9685dd137dfe9afc70a4933e8eb4f9a5283a8e84bd443a70c"} Jul 23 11:37:38.094445 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:38.094432 2561 scope.go:117] "RemoveContainer" containerID="2b93f10485091aa9685dd137dfe9afc70a4933e8eb4f9a5283a8e84bd443a70c" Jul 23 11:37:39.099806 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:39.099776 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-48j4r" event={"ID":"b64d0632-5844-419e-9f53-ac57a0495a8d","Type":"ContainerStarted","Data":"1baecdd160903da179d415c232a96ac20f0048415b07c40b614b2a9f295df40e"} Jul 23 11:37:39.101553 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:37:39.101531 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-sr6b2" event={"ID":"98f50c4a-91e4-4708-be1a-cff80dfb6cc8","Type":"ContainerStarted","Data":"34008d43d6da82def339de2fade90a6ebedaf91d757cf7c624824d195b8466e8"} Jul 23 11:40:15.155158 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:15.155125 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5786849f7-lktgl"] Jul 23 11:40:15.155562 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:15.155394 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b993c3b9-94b8-4c21-b391-6193659cf9ee" containerName="registry" Jul 23 11:40:15.155562 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:15.155404 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b993c3b9-94b8-4c21-b391-6193659cf9ee" containerName="registry" Jul 23 11:40:15.155562 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:15.155456 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b993c3b9-94b8-4c21-b391-6193659cf9ee" containerName="registry" Jul 23 11:40:15.158231 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:15.158215 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5786849f7-lktgl" Jul 23 11:40:15.161069 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:15.161045 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 23 11:40:15.162231 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:15.162211 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 23 11:40:15.162335 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:15.162236 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 23 11:40:15.162335 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:15.162212 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 23 11:40:15.168089 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:15.168069 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5786849f7-lktgl"] Jul 23 11:40:15.224925 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:15.224885 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d86efba8-a403-4e05-bb0d-8e56aa97e8ea-tmp\") pod \"klusterlet-addon-workmgr-5786849f7-lktgl\" (UID: \"d86efba8-a403-4e05-bb0d-8e56aa97e8ea\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5786849f7-lktgl" Jul 23 11:40:15.225014 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:15.224927 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/d86efba8-a403-4e05-bb0d-8e56aa97e8ea-klusterlet-config\") pod \"klusterlet-addon-workmgr-5786849f7-lktgl\" (UID: \"d86efba8-a403-4e05-bb0d-8e56aa97e8ea\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5786849f7-lktgl" Jul 23 11:40:15.225014 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:15.224969 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nggnr\" (UniqueName: \"kubernetes.io/projected/d86efba8-a403-4e05-bb0d-8e56aa97e8ea-kube-api-access-nggnr\") pod \"klusterlet-addon-workmgr-5786849f7-lktgl\" (UID: \"d86efba8-a403-4e05-bb0d-8e56aa97e8ea\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5786849f7-lktgl" Jul 23 11:40:15.325340 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:15.325318 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nggnr\" (UniqueName: \"kubernetes.io/projected/d86efba8-a403-4e05-bb0d-8e56aa97e8ea-kube-api-access-nggnr\") pod \"klusterlet-addon-workmgr-5786849f7-lktgl\" (UID: \"d86efba8-a403-4e05-bb0d-8e56aa97e8ea\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5786849f7-lktgl" Jul 23 11:40:15.325421 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:15.325353 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d86efba8-a403-4e05-bb0d-8e56aa97e8ea-tmp\") pod \"klusterlet-addon-workmgr-5786849f7-lktgl\" (UID: \"d86efba8-a403-4e05-bb0d-8e56aa97e8ea\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5786849f7-lktgl" Jul 23 11:40:15.325421 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:15.325378 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/d86efba8-a403-4e05-bb0d-8e56aa97e8ea-klusterlet-config\") pod \"klusterlet-addon-workmgr-5786849f7-lktgl\" (UID: \"d86efba8-a403-4e05-bb0d-8e56aa97e8ea\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5786849f7-lktgl" Jul 23 11:40:15.325642 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:15.325627 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d86efba8-a403-4e05-bb0d-8e56aa97e8ea-tmp\") pod \"klusterlet-addon-workmgr-5786849f7-lktgl\" (UID: \"d86efba8-a403-4e05-bb0d-8e56aa97e8ea\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5786849f7-lktgl" Jul 23 11:40:15.327713 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:15.327685 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/d86efba8-a403-4e05-bb0d-8e56aa97e8ea-klusterlet-config\") pod \"klusterlet-addon-workmgr-5786849f7-lktgl\" (UID: \"d86efba8-a403-4e05-bb0d-8e56aa97e8ea\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5786849f7-lktgl" Jul 23 11:40:15.334373 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:15.334350 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nggnr\" (UniqueName: \"kubernetes.io/projected/d86efba8-a403-4e05-bb0d-8e56aa97e8ea-kube-api-access-nggnr\") pod \"klusterlet-addon-workmgr-5786849f7-lktgl\" (UID: \"d86efba8-a403-4e05-bb0d-8e56aa97e8ea\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5786849f7-lktgl" Jul 23 11:40:15.467083 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:15.467033 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5786849f7-lktgl" Jul 23 11:40:15.588270 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:15.588250 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5786849f7-lktgl"] Jul 23 11:40:15.590859 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:40:15.590829 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd86efba8_a403_4e05_bb0d_8e56aa97e8ea.slice/crio-53f9b277bc645b1947c18239e20be9de319d1948118668d18919a571ab3ae079 WatchSource:0}: Error finding container 53f9b277bc645b1947c18239e20be9de319d1948118668d18919a571ab3ae079: Status 404 returned error can't find the container with id 53f9b277bc645b1947c18239e20be9de319d1948118668d18919a571ab3ae079 Jul 23 11:40:15.624897 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:15.624869 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5786849f7-lktgl" event={"ID":"d86efba8-a403-4e05-bb0d-8e56aa97e8ea","Type":"ContainerStarted","Data":"53f9b277bc645b1947c18239e20be9de319d1948118668d18919a571ab3ae079"} Jul 23 11:40:19.643009 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:19.642973 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5786849f7-lktgl" event={"ID":"d86efba8-a403-4e05-bb0d-8e56aa97e8ea","Type":"ContainerStarted","Data":"9863b9b99b9310ccb5f4ff9b54a128f98677bb57bf60f6c9e666da3278d1fb61"} Jul 23 11:40:19.643397 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:19.643153 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5786849f7-lktgl" Jul 23 11:40:19.644599 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:19.644581 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5786849f7-lktgl" Jul 23 11:40:19.666434 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:19.666393 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5786849f7-lktgl" podStartSLOduration=1.423968326 podStartE2EDuration="4.666379762s" totalImagesPullingTime="3.242411436s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:40:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:40:19.665723021 +0000 UTC m=+286.737413004" watchObservedRunningTime="2026-07-23 11:40:19.666379762 +0000 UTC m=+286.738069750" Jul 23 11:40:25.833787 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:25.833760 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lxtv9"] Jul 23 11:40:25.836983 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:25.836967 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lxtv9" Jul 23 11:40:25.839929 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:25.839887 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 23 11:40:25.840185 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:25.840169 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 23 11:40:25.840959 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:25.840943 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-rfftq\"" Jul 23 11:40:25.843436 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:25.843284 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lxtv9"] Jul 23 11:40:25.901207 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:25.901186 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6077b289-4069-4931-aa08-0265c7d981f3-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lxtv9\" (UID: \"6077b289-4069-4931-aa08-0265c7d981f3\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lxtv9" Jul 23 11:40:25.901296 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:25.901221 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6077b289-4069-4931-aa08-0265c7d981f3-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lxtv9\" (UID: \"6077b289-4069-4931-aa08-0265c7d981f3\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lxtv9" Jul 23 11:40:25.901296 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:25.901249 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkjg7\" (UniqueName: \"kubernetes.io/projected/6077b289-4069-4931-aa08-0265c7d981f3-kube-api-access-jkjg7\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lxtv9\" (UID: \"6077b289-4069-4931-aa08-0265c7d981f3\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lxtv9" Jul 23 11:40:26.001638 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:26.001610 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6077b289-4069-4931-aa08-0265c7d981f3-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lxtv9\" (UID: \"6077b289-4069-4931-aa08-0265c7d981f3\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lxtv9" Jul 23 11:40:26.001724 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:26.001650 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkjg7\" (UniqueName: \"kubernetes.io/projected/6077b289-4069-4931-aa08-0265c7d981f3-kube-api-access-jkjg7\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lxtv9\" (UID: \"6077b289-4069-4931-aa08-0265c7d981f3\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lxtv9" Jul 23 11:40:26.001724 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:26.001703 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6077b289-4069-4931-aa08-0265c7d981f3-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lxtv9\" (UID: \"6077b289-4069-4931-aa08-0265c7d981f3\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lxtv9" Jul 23 11:40:26.002031 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:26.002013 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6077b289-4069-4931-aa08-0265c7d981f3-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lxtv9\" (UID: \"6077b289-4069-4931-aa08-0265c7d981f3\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lxtv9" Jul 23 11:40:26.002031 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:26.002026 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6077b289-4069-4931-aa08-0265c7d981f3-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lxtv9\" (UID: \"6077b289-4069-4931-aa08-0265c7d981f3\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lxtv9" Jul 23 11:40:26.010481 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:26.010462 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkjg7\" (UniqueName: \"kubernetes.io/projected/6077b289-4069-4931-aa08-0265c7d981f3-kube-api-access-jkjg7\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lxtv9\" (UID: \"6077b289-4069-4931-aa08-0265c7d981f3\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lxtv9" Jul 23 11:40:26.146004 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:26.145962 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lxtv9" Jul 23 11:40:26.264377 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:26.264349 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lxtv9"] Jul 23 11:40:26.267204 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:40:26.267179 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6077b289_4069_4931_aa08_0265c7d981f3.slice/crio-c83e0bbaf99bab4e4e03e33dd38d09047492c56278d87f344964475bcfcce8e2 WatchSource:0}: Error finding container c83e0bbaf99bab4e4e03e33dd38d09047492c56278d87f344964475bcfcce8e2: Status 404 returned error can't find the container with id c83e0bbaf99bab4e4e03e33dd38d09047492c56278d87f344964475bcfcce8e2 Jul 23 11:40:26.664596 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:26.664569 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lxtv9" event={"ID":"6077b289-4069-4931-aa08-0265c7d981f3","Type":"ContainerStarted","Data":"c83e0bbaf99bab4e4e03e33dd38d09047492c56278d87f344964475bcfcce8e2"} Jul 23 11:40:33.317184 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:33.317162 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-7slpj_65247e1f-fc10-4b40-af67-5e646f352263/console-operator/1.log" Jul 23 11:40:33.319584 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:33.319559 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-7slpj_65247e1f-fc10-4b40-af67-5e646f352263/console-operator/1.log" Jul 23 11:40:33.326741 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:33.326716 2561 kubelet.go:1648] "Image garbage collection succeeded" Jul 23 11:40:33.690970 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:33.690896 2561 generic.go:356] "Generic (PLEG): container finished" podID="6077b289-4069-4931-aa08-0265c7d981f3" containerID="a8d19d12686c4672bc81f5c6e3905fed35b3f3a5092594fd54a48b1181f014ef" exitCode=0 Jul 23 11:40:33.693047 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:33.690986 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lxtv9" event={"ID":"6077b289-4069-4931-aa08-0265c7d981f3","Type":"ContainerDied","Data":"a8d19d12686c4672bc81f5c6e3905fed35b3f3a5092594fd54a48b1181f014ef"} Jul 23 11:40:35.701165 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:35.701132 2561 generic.go:356] "Generic (PLEG): container finished" podID="6077b289-4069-4931-aa08-0265c7d981f3" containerID="56915e934acb0765f7167e33e4d47a6fc06a54eb489dba8139d5656f8365e94d" exitCode=0 Jul 23 11:40:35.701560 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:35.701174 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lxtv9" event={"ID":"6077b289-4069-4931-aa08-0265c7d981f3","Type":"ContainerDied","Data":"56915e934acb0765f7167e33e4d47a6fc06a54eb489dba8139d5656f8365e94d"} Jul 23 11:40:35.702118 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:35.702101 2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 11:40:42.729531 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:42.729495 2561 generic.go:356] "Generic (PLEG): container finished" podID="6077b289-4069-4931-aa08-0265c7d981f3" containerID="ef00f9141a5d8c1d3fa9bc17ecf1f32e7f55165c2ce077b1b4b4ef3950fa946f" exitCode=0 Jul 23 11:40:42.729880 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:42.729541 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lxtv9" event={"ID":"6077b289-4069-4931-aa08-0265c7d981f3","Type":"ContainerDied","Data":"ef00f9141a5d8c1d3fa9bc17ecf1f32e7f55165c2ce077b1b4b4ef3950fa946f"} Jul 23 11:40:43.853678 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:43.853657 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lxtv9" Jul 23 11:40:43.934680 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:43.934658 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6077b289-4069-4931-aa08-0265c7d981f3-bundle\") pod \"6077b289-4069-4931-aa08-0265c7d981f3\" (UID: \"6077b289-4069-4931-aa08-0265c7d981f3\") " Jul 23 11:40:43.934779 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:43.934704 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6077b289-4069-4931-aa08-0265c7d981f3-util\") pod \"6077b289-4069-4931-aa08-0265c7d981f3\" (UID: \"6077b289-4069-4931-aa08-0265c7d981f3\") " Jul 23 11:40:43.934779 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:43.934728 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jkjg7\" (UniqueName: \"kubernetes.io/projected/6077b289-4069-4931-aa08-0265c7d981f3-kube-api-access-jkjg7\") pod \"6077b289-4069-4931-aa08-0265c7d981f3\" (UID: \"6077b289-4069-4931-aa08-0265c7d981f3\") " Jul 23 11:40:43.935301 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:43.935275 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6077b289-4069-4931-aa08-0265c7d981f3-bundle" (OuterVolumeSpecName: "bundle") pod "6077b289-4069-4931-aa08-0265c7d981f3" (UID: "6077b289-4069-4931-aa08-0265c7d981f3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 11:40:43.936814 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:43.936792 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6077b289-4069-4931-aa08-0265c7d981f3-kube-api-access-jkjg7" (OuterVolumeSpecName: "kube-api-access-jkjg7") pod "6077b289-4069-4931-aa08-0265c7d981f3" (UID: "6077b289-4069-4931-aa08-0265c7d981f3"). InnerVolumeSpecName "kube-api-access-jkjg7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 11:40:43.938671 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:43.938651 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6077b289-4069-4931-aa08-0265c7d981f3-util" (OuterVolumeSpecName: "util") pod "6077b289-4069-4931-aa08-0265c7d981f3" (UID: "6077b289-4069-4931-aa08-0265c7d981f3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 11:40:44.035292 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:44.035247 2561 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6077b289-4069-4931-aa08-0265c7d981f3-bundle\") on node \"ip-10-0-134-236.ec2.internal\" DevicePath \"\"" Jul 23 11:40:44.035292 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:44.035266 2561 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6077b289-4069-4931-aa08-0265c7d981f3-util\") on node \"ip-10-0-134-236.ec2.internal\" DevicePath \"\"" Jul 23 11:40:44.035292 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:44.035275 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkjg7\" (UniqueName: \"kubernetes.io/projected/6077b289-4069-4931-aa08-0265c7d981f3-kube-api-access-jkjg7\") on node \"ip-10-0-134-236.ec2.internal\" DevicePath \"\"" Jul 23 11:40:44.738860 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:44.738841 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lxtv9" Jul 23 11:40:44.738860 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:44.738841 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lxtv9" event={"ID":"6077b289-4069-4931-aa08-0265c7d981f3","Type":"ContainerDied","Data":"c83e0bbaf99bab4e4e03e33dd38d09047492c56278d87f344964475bcfcce8e2"} Jul 23 11:40:44.739015 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:44.738880 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c83e0bbaf99bab4e4e03e33dd38d09047492c56278d87f344964475bcfcce8e2" Jul 23 11:40:53.099299 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:53.099269 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbrc5d"] Jul 23 11:40:53.099664 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:53.099524 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6077b289-4069-4931-aa08-0265c7d981f3" containerName="util" Jul 23 11:40:53.099664 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:53.099534 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="6077b289-4069-4931-aa08-0265c7d981f3" containerName="util" Jul 23 11:40:53.099664 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:53.099544 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6077b289-4069-4931-aa08-0265c7d981f3" containerName="pull" Jul 23 11:40:53.099664 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:53.099549 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="6077b289-4069-4931-aa08-0265c7d981f3" containerName="pull" Jul 23 11:40:53.099664 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:53.099556 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6077b289-4069-4931-aa08-0265c7d981f3" containerName="extract" Jul 23 11:40:53.099664 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:53.099561 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="6077b289-4069-4931-aa08-0265c7d981f3" containerName="extract" Jul 23 11:40:53.099664 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:53.099608 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="6077b289-4069-4931-aa08-0265c7d981f3" containerName="extract" Jul 23 11:40:53.101774 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:53.101758 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbrc5d" Jul 23 11:40:53.106194 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:53.106176 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-rfftq\"" Jul 23 11:40:53.106979 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:53.106963 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 23 11:40:53.108307 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:53.108291 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 23 11:40:53.133153 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:53.133133 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbrc5d"] Jul 23 11:40:53.199976 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:53.199952 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8cd0a243-73e8-442a-9f82-3b70fea5292c-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbrc5d\" (UID: \"8cd0a243-73e8-442a-9f82-3b70fea5292c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbrc5d" Jul 23 11:40:53.200074 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:53.199980 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj89k\" (UniqueName: \"kubernetes.io/projected/8cd0a243-73e8-442a-9f82-3b70fea5292c-kube-api-access-vj89k\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbrc5d\" (UID: \"8cd0a243-73e8-442a-9f82-3b70fea5292c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbrc5d" Jul 23 11:40:53.200074 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:53.200012 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8cd0a243-73e8-442a-9f82-3b70fea5292c-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbrc5d\" (UID: \"8cd0a243-73e8-442a-9f82-3b70fea5292c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbrc5d" Jul 23 11:40:53.301024 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:53.301005 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8cd0a243-73e8-442a-9f82-3b70fea5292c-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbrc5d\" (UID: \"8cd0a243-73e8-442a-9f82-3b70fea5292c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbrc5d" Jul 23 11:40:53.301116 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:53.301030 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj89k\" (UniqueName: \"kubernetes.io/projected/8cd0a243-73e8-442a-9f82-3b70fea5292c-kube-api-access-vj89k\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbrc5d\" (UID: \"8cd0a243-73e8-442a-9f82-3b70fea5292c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbrc5d" Jul 23 11:40:53.301116 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:53.301062 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8cd0a243-73e8-442a-9f82-3b70fea5292c-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbrc5d\" (UID: \"8cd0a243-73e8-442a-9f82-3b70fea5292c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbrc5d" Jul 23 11:40:53.301342 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:53.301327 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8cd0a243-73e8-442a-9f82-3b70fea5292c-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbrc5d\" (UID: \"8cd0a243-73e8-442a-9f82-3b70fea5292c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbrc5d" Jul 23 11:40:53.301410 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:53.301387 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8cd0a243-73e8-442a-9f82-3b70fea5292c-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbrc5d\" (UID: \"8cd0a243-73e8-442a-9f82-3b70fea5292c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbrc5d" Jul 23 11:40:53.318324 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:53.318297 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj89k\" (UniqueName: \"kubernetes.io/projected/8cd0a243-73e8-442a-9f82-3b70fea5292c-kube-api-access-vj89k\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbrc5d\" (UID: \"8cd0a243-73e8-442a-9f82-3b70fea5292c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbrc5d" Jul 23 11:40:53.409930 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:53.409858 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbrc5d" Jul 23 11:40:53.550653 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:53.550630 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbrc5d"] Jul 23 11:40:53.552477 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:40:53.552455 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cd0a243_73e8_442a_9f82_3b70fea5292c.slice/crio-ed5f0790de9f7dcc9b1b7b60f9dc16ac8cdbf767c3b62e0be06165ef50a623e2 WatchSource:0}: Error finding container ed5f0790de9f7dcc9b1b7b60f9dc16ac8cdbf767c3b62e0be06165ef50a623e2: Status 404 returned error can't find the container with id ed5f0790de9f7dcc9b1b7b60f9dc16ac8cdbf767c3b62e0be06165ef50a623e2 Jul 23 11:40:53.769869 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:53.769813 2561 generic.go:356] "Generic (PLEG): container finished" podID="8cd0a243-73e8-442a-9f82-3b70fea5292c" containerID="cd92f2e625928561fbf4ccbc0494000fde838b12a1a28a952c016decdb416d7d" exitCode=0 Jul 23 11:40:53.769988 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:53.769870 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbrc5d" event={"ID":"8cd0a243-73e8-442a-9f82-3b70fea5292c","Type":"ContainerDied","Data":"cd92f2e625928561fbf4ccbc0494000fde838b12a1a28a952c016decdb416d7d"} Jul 23 11:40:53.769988 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:53.769902 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbrc5d" event={"ID":"8cd0a243-73e8-442a-9f82-3b70fea5292c","Type":"ContainerStarted","Data":"ed5f0790de9f7dcc9b1b7b60f9dc16ac8cdbf767c3b62e0be06165ef50a623e2"} Jul 23 11:40:56.782486 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:56.782459 2561 generic.go:356] "Generic (PLEG): container finished" podID="8cd0a243-73e8-442a-9f82-3b70fea5292c" containerID="2db0039bad55f22ee8175e35fff72f2f84ff892bf61d0a274c7435e52b64521e" exitCode=0 Jul 23 11:40:56.782794 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:56.782526 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbrc5d" event={"ID":"8cd0a243-73e8-442a-9f82-3b70fea5292c","Type":"ContainerDied","Data":"2db0039bad55f22ee8175e35fff72f2f84ff892bf61d0a274c7435e52b64521e"} Jul 23 11:40:57.239342 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:57.239311 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-6hhr5"] Jul 23 11:40:57.242370 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:57.242353 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-6hhr5" Jul 23 11:40:57.244741 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:57.244722 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 23 11:40:57.244834 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:57.244800 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-nq4xz\"" Jul 23 11:40:57.244896 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:57.244875 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 23 11:40:57.251227 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:57.251208 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-6hhr5"] Jul 23 11:40:57.328554 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:57.328531 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvwnz\" (UniqueName: \"kubernetes.io/projected/0aa616be-9982-442b-8743-c521346a7b19-kube-api-access-zvwnz\") pod \"cert-manager-cainjector-559cdd8c87-6hhr5\" (UID: \"0aa616be-9982-442b-8743-c521346a7b19\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-6hhr5" Jul 23 11:40:57.328643 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:57.328560 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0aa616be-9982-442b-8743-c521346a7b19-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-6hhr5\" (UID: \"0aa616be-9982-442b-8743-c521346a7b19\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-6hhr5" Jul 23 11:40:57.429483 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:57.429462 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0aa616be-9982-442b-8743-c521346a7b19-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-6hhr5\" (UID: \"0aa616be-9982-442b-8743-c521346a7b19\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-6hhr5" Jul 23 11:40:57.429553 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:57.429524 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvwnz\" (UniqueName: \"kubernetes.io/projected/0aa616be-9982-442b-8743-c521346a7b19-kube-api-access-zvwnz\") pod \"cert-manager-cainjector-559cdd8c87-6hhr5\" (UID: \"0aa616be-9982-442b-8743-c521346a7b19\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-6hhr5" Jul 23 11:40:57.442040 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:57.442018 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0aa616be-9982-442b-8743-c521346a7b19-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-6hhr5\" (UID: \"0aa616be-9982-442b-8743-c521346a7b19\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-6hhr5" Jul 23 11:40:57.444452 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:57.444435 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvwnz\" (UniqueName: \"kubernetes.io/projected/0aa616be-9982-442b-8743-c521346a7b19-kube-api-access-zvwnz\") pod \"cert-manager-cainjector-559cdd8c87-6hhr5\" (UID: \"0aa616be-9982-442b-8743-c521346a7b19\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-6hhr5" Jul 23 11:40:57.561213 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:57.561169 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-6hhr5" Jul 23 11:40:57.685882 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:57.685847 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-6hhr5"] Jul 23 11:40:57.688422 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:40:57.688399 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0aa616be_9982_442b_8743_c521346a7b19.slice/crio-cef11319bb4d24b764f78aeb7f7003da7807d0ca0967e332210b0ce45f10be47 WatchSource:0}: Error finding container cef11319bb4d24b764f78aeb7f7003da7807d0ca0967e332210b0ce45f10be47: Status 404 returned error can't find the container with id cef11319bb4d24b764f78aeb7f7003da7807d0ca0967e332210b0ce45f10be47 Jul 23 11:40:57.786819 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:57.786792 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-6hhr5" event={"ID":"0aa616be-9982-442b-8743-c521346a7b19","Type":"ContainerStarted","Data":"cef11319bb4d24b764f78aeb7f7003da7807d0ca0967e332210b0ce45f10be47"} Jul 23 11:40:57.788590 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:57.788571 2561 generic.go:356] "Generic (PLEG): container finished" podID="8cd0a243-73e8-442a-9f82-3b70fea5292c" containerID="00b646ecd14d83bcef47112e3d47a6d6dbfe8bac6862f70d8a6aad356adb2eee" exitCode=0 Jul 23 11:40:57.788668 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:57.788620 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbrc5d" event={"ID":"8cd0a243-73e8-442a-9f82-3b70fea5292c","Type":"ContainerDied","Data":"00b646ecd14d83bcef47112e3d47a6d6dbfe8bac6862f70d8a6aad356adb2eee"} Jul 23 11:40:58.931670 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:58.931650 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbrc5d" Jul 23 11:40:59.037995 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:59.037963 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8cd0a243-73e8-442a-9f82-3b70fea5292c-bundle\") pod \"8cd0a243-73e8-442a-9f82-3b70fea5292c\" (UID: \"8cd0a243-73e8-442a-9f82-3b70fea5292c\") " Jul 23 11:40:59.038146 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:59.038038 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vj89k\" (UniqueName: \"kubernetes.io/projected/8cd0a243-73e8-442a-9f82-3b70fea5292c-kube-api-access-vj89k\") pod \"8cd0a243-73e8-442a-9f82-3b70fea5292c\" (UID: \"8cd0a243-73e8-442a-9f82-3b70fea5292c\") " Jul 23 11:40:59.038146 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:59.038087 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8cd0a243-73e8-442a-9f82-3b70fea5292c-util\") pod \"8cd0a243-73e8-442a-9f82-3b70fea5292c\" (UID: \"8cd0a243-73e8-442a-9f82-3b70fea5292c\") " Jul 23 11:40:59.038589 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:59.038508 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cd0a243-73e8-442a-9f82-3b70fea5292c-bundle" (OuterVolumeSpecName: "bundle") pod "8cd0a243-73e8-442a-9f82-3b70fea5292c" (UID: "8cd0a243-73e8-442a-9f82-3b70fea5292c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 11:40:59.040447 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:59.040421 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cd0a243-73e8-442a-9f82-3b70fea5292c-kube-api-access-vj89k" (OuterVolumeSpecName: "kube-api-access-vj89k") pod "8cd0a243-73e8-442a-9f82-3b70fea5292c" (UID: "8cd0a243-73e8-442a-9f82-3b70fea5292c"). InnerVolumeSpecName "kube-api-access-vj89k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 11:40:59.043760 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:59.043696 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cd0a243-73e8-442a-9f82-3b70fea5292c-util" (OuterVolumeSpecName: "util") pod "8cd0a243-73e8-442a-9f82-3b70fea5292c" (UID: "8cd0a243-73e8-442a-9f82-3b70fea5292c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 11:40:59.138937 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:59.138899 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vj89k\" (UniqueName: \"kubernetes.io/projected/8cd0a243-73e8-442a-9f82-3b70fea5292c-kube-api-access-vj89k\") on node \"ip-10-0-134-236.ec2.internal\" DevicePath \"\"" Jul 23 11:40:59.139043 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:59.138939 2561 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8cd0a243-73e8-442a-9f82-3b70fea5292c-util\") on node \"ip-10-0-134-236.ec2.internal\" DevicePath \"\"" Jul 23 11:40:59.139043 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:59.138967 2561 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8cd0a243-73e8-442a-9f82-3b70fea5292c-bundle\") on node \"ip-10-0-134-236.ec2.internal\" DevicePath \"\"" Jul 23 11:40:59.799297 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:59.799182 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbrc5d" event={"ID":"8cd0a243-73e8-442a-9f82-3b70fea5292c","Type":"ContainerDied","Data":"ed5f0790de9f7dcc9b1b7b60f9dc16ac8cdbf767c3b62e0be06165ef50a623e2"} Jul 23 11:40:59.799297 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:59.799216 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbrc5d" Jul 23 11:40:59.799297 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:40:59.799233 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed5f0790de9f7dcc9b1b7b60f9dc16ac8cdbf767c3b62e0be06165ef50a623e2" Jul 23 11:41:00.804076 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:00.804049 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-6hhr5" event={"ID":"0aa616be-9982-442b-8743-c521346a7b19","Type":"ContainerStarted","Data":"918463e15bd4bad8d959b41ef28e1e8c88150b75f6d85a2bbab8e5f8021791ab"} Jul 23 11:41:00.826466 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:00.826423 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-6hhr5" podStartSLOduration=1.268186371 podStartE2EDuration="3.826408616s" totalImagesPullingTime="2.558222245s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:40:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:41:00.825174904 +0000 UTC m=+327.896864892" watchObservedRunningTime="2026-07-23 11:41:00.826408616 +0000 UTC m=+327.898098604" Jul 23 11:41:03.604166 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:03.604141 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-bmxv6"] Jul 23 11:41:03.604498 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:03.604399 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cd0a243-73e8-442a-9f82-3b70fea5292c" containerName="util" Jul 23 11:41:03.604498 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:03.604411 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cd0a243-73e8-442a-9f82-3b70fea5292c" containerName="util" Jul 23 11:41:03.604498 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:03.604428 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cd0a243-73e8-442a-9f82-3b70fea5292c" containerName="pull" Jul 23 11:41:03.604498 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:03.604432 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cd0a243-73e8-442a-9f82-3b70fea5292c" containerName="pull" Jul 23 11:41:03.604498 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:03.604443 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cd0a243-73e8-442a-9f82-3b70fea5292c" containerName="extract" Jul 23 11:41:03.604498 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:03.604452 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cd0a243-73e8-442a-9f82-3b70fea5292c" containerName="extract" Jul 23 11:41:03.604498 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:03.604497 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8cd0a243-73e8-442a-9f82-3b70fea5292c" containerName="extract" Jul 23 11:41:03.607333 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:03.607319 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-bmxv6" Jul 23 11:41:03.611737 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:03.611720 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-x4tdr\"" Jul 23 11:41:03.618666 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:03.618647 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-bmxv6"] Jul 23 11:41:03.772121 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:03.772095 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr72r\" (UniqueName: \"kubernetes.io/projected/302929ed-91f2-42d7-a2ef-39be56693a41-kube-api-access-gr72r\") pod \"cert-manager-55b4d954d5-bmxv6\" (UID: \"302929ed-91f2-42d7-a2ef-39be56693a41\") " pod="cert-manager/cert-manager-55b4d954d5-bmxv6" Jul 23 11:41:03.772243 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:03.772162 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/302929ed-91f2-42d7-a2ef-39be56693a41-bound-sa-token\") pod \"cert-manager-55b4d954d5-bmxv6\" (UID: \"302929ed-91f2-42d7-a2ef-39be56693a41\") " pod="cert-manager/cert-manager-55b4d954d5-bmxv6" Jul 23 11:41:03.872884 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:03.872826 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr72r\" (UniqueName: \"kubernetes.io/projected/302929ed-91f2-42d7-a2ef-39be56693a41-kube-api-access-gr72r\") pod \"cert-manager-55b4d954d5-bmxv6\" (UID: \"302929ed-91f2-42d7-a2ef-39be56693a41\") " pod="cert-manager/cert-manager-55b4d954d5-bmxv6" Jul 23 11:41:03.873000 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:03.872893 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/302929ed-91f2-42d7-a2ef-39be56693a41-bound-sa-token\") pod \"cert-manager-55b4d954d5-bmxv6\" (UID: \"302929ed-91f2-42d7-a2ef-39be56693a41\") " pod="cert-manager/cert-manager-55b4d954d5-bmxv6" Jul 23 11:41:03.882190 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:03.882171 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr72r\" (UniqueName: \"kubernetes.io/projected/302929ed-91f2-42d7-a2ef-39be56693a41-kube-api-access-gr72r\") pod \"cert-manager-55b4d954d5-bmxv6\" (UID: \"302929ed-91f2-42d7-a2ef-39be56693a41\") " pod="cert-manager/cert-manager-55b4d954d5-bmxv6" Jul 23 11:41:03.882190 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:03.882178 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/302929ed-91f2-42d7-a2ef-39be56693a41-bound-sa-token\") pod \"cert-manager-55b4d954d5-bmxv6\" (UID: \"302929ed-91f2-42d7-a2ef-39be56693a41\") " pod="cert-manager/cert-manager-55b4d954d5-bmxv6" Jul 23 11:41:03.916177 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:03.916158 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-bmxv6" Jul 23 11:41:04.036695 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:04.036593 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-bmxv6"] Jul 23 11:41:04.039046 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:41:04.039017 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod302929ed_91f2_42d7_a2ef_39be56693a41.slice/crio-828b8550506bfd1577eaadaeff6fdf9278c2974ba2529672bcf8d20d01734a4d WatchSource:0}: Error finding container 828b8550506bfd1577eaadaeff6fdf9278c2974ba2529672bcf8d20d01734a4d: Status 404 returned error can't find the container with id 828b8550506bfd1577eaadaeff6fdf9278c2974ba2529672bcf8d20d01734a4d Jul 23 11:41:04.819357 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:04.819329 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-bmxv6" event={"ID":"302929ed-91f2-42d7-a2ef-39be56693a41","Type":"ContainerStarted","Data":"d306a9597d93b0759e7d7f5c8a6cab0db577f5fb407859bfecf45311ba47f4a9"} Jul 23 11:41:04.819357 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:04.819367 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-bmxv6" event={"ID":"302929ed-91f2-42d7-a2ef-39be56693a41","Type":"ContainerStarted","Data":"828b8550506bfd1577eaadaeff6fdf9278c2974ba2529672bcf8d20d01734a4d"} Jul 23 11:41:04.835880 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:04.835833 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-bmxv6" podStartSLOduration=1.835819158 podStartE2EDuration="1.835819158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:41:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:41:04.834282264 +0000 UTC m=+331.905972254" watchObservedRunningTime="2026-07-23 11:41:04.835819158 +0000 UTC m=+331.907509146" Jul 23 11:41:12.735038 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:12.735007 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdgtsb"] Jul 23 11:41:12.816055 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:12.816026 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdgtsb"] Jul 23 11:41:12.816199 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:12.816121 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdgtsb" Jul 23 11:41:12.819030 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:12.818992 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 23 11:41:12.819030 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:12.819006 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 23 11:41:12.820054 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:12.820027 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-rfftq\"" Jul 23 11:41:12.928862 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:12.928837 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8950f507-4292-408b-a2d2-39cdd7a94200-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdgtsb\" (UID: \"8950f507-4292-408b-a2d2-39cdd7a94200\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdgtsb" Jul 23 11:41:12.928986 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:12.928877 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8950f507-4292-408b-a2d2-39cdd7a94200-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdgtsb\" (UID: \"8950f507-4292-408b-a2d2-39cdd7a94200\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdgtsb" Jul 23 11:41:12.928986 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:12.928899 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlq6b\" (UniqueName: \"kubernetes.io/projected/8950f507-4292-408b-a2d2-39cdd7a94200-kube-api-access-nlq6b\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdgtsb\" (UID: \"8950f507-4292-408b-a2d2-39cdd7a94200\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdgtsb" Jul 23 11:41:13.029218 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:13.029167 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8950f507-4292-408b-a2d2-39cdd7a94200-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdgtsb\" (UID: \"8950f507-4292-408b-a2d2-39cdd7a94200\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdgtsb" Jul 23 11:41:13.029218 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:13.029201 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlq6b\" (UniqueName: \"kubernetes.io/projected/8950f507-4292-408b-a2d2-39cdd7a94200-kube-api-access-nlq6b\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdgtsb\" (UID: \"8950f507-4292-408b-a2d2-39cdd7a94200\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdgtsb" Jul 23 11:41:13.029374 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:13.029231 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8950f507-4292-408b-a2d2-39cdd7a94200-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdgtsb\" (UID: \"8950f507-4292-408b-a2d2-39cdd7a94200\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdgtsb" Jul 23 11:41:13.029557 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:13.029537 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8950f507-4292-408b-a2d2-39cdd7a94200-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdgtsb\" (UID: \"8950f507-4292-408b-a2d2-39cdd7a94200\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdgtsb" Jul 23 11:41:13.029615 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:13.029542 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8950f507-4292-408b-a2d2-39cdd7a94200-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdgtsb\" (UID: \"8950f507-4292-408b-a2d2-39cdd7a94200\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdgtsb" Jul 23 11:41:13.041594 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:13.041579 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlq6b\" (UniqueName: \"kubernetes.io/projected/8950f507-4292-408b-a2d2-39cdd7a94200-kube-api-access-nlq6b\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdgtsb\" (UID: \"8950f507-4292-408b-a2d2-39cdd7a94200\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdgtsb" Jul 23 11:41:13.124970 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:13.124951 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdgtsb" Jul 23 11:41:13.263657 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:13.263584 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdgtsb"] Jul 23 11:41:13.265744 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:41:13.265719 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8950f507_4292_408b_a2d2_39cdd7a94200.slice/crio-14911fc176289dad4a87aaaf4cf6541db28e7c91805afbee8f1219f3d170798b WatchSource:0}: Error finding container 14911fc176289dad4a87aaaf4cf6541db28e7c91805afbee8f1219f3d170798b: Status 404 returned error can't find the container with id 14911fc176289dad4a87aaaf4cf6541db28e7c91805afbee8f1219f3d170798b Jul 23 11:41:13.852983 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:13.852953 2561 generic.go:356] "Generic (PLEG): container finished" podID="8950f507-4292-408b-a2d2-39cdd7a94200" containerID="b586257b24766f71fa4a36af6e6233ff23480a538d5fba414bcd601ad1560e78" exitCode=0 Jul 23 11:41:13.853262 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:13.853018 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdgtsb" event={"ID":"8950f507-4292-408b-a2d2-39cdd7a94200","Type":"ContainerDied","Data":"b586257b24766f71fa4a36af6e6233ff23480a538d5fba414bcd601ad1560e78"} Jul 23 11:41:13.853262 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:13.853053 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdgtsb" event={"ID":"8950f507-4292-408b-a2d2-39cdd7a94200","Type":"ContainerStarted","Data":"14911fc176289dad4a87aaaf4cf6541db28e7c91805afbee8f1219f3d170798b"} Jul 23 11:41:14.858407 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:14.858384 2561 generic.go:356] "Generic (PLEG): container finished" podID="8950f507-4292-408b-a2d2-39cdd7a94200" containerID="97200c67732d097f9537c37be9b692762470d6b898e950dbb2b4743946df2f23" exitCode=0 Jul 23 11:41:14.858702 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:14.858470 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdgtsb" event={"ID":"8950f507-4292-408b-a2d2-39cdd7a94200","Type":"ContainerDied","Data":"97200c67732d097f9537c37be9b692762470d6b898e950dbb2b4743946df2f23"} Jul 23 11:41:15.863824 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:15.863800 2561 generic.go:356] "Generic (PLEG): container finished" podID="8950f507-4292-408b-a2d2-39cdd7a94200" containerID="908adf0f09cdc0febbaeae30717d0ef752c7f63a0e7df05b2afba93bdc42dd04" exitCode=0 Jul 23 11:41:15.864139 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:15.863880 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdgtsb" event={"ID":"8950f507-4292-408b-a2d2-39cdd7a94200","Type":"ContainerDied","Data":"908adf0f09cdc0febbaeae30717d0ef752c7f63a0e7df05b2afba93bdc42dd04"} Jul 23 11:41:16.979856 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:16.979834 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdgtsb" Jul 23 11:41:17.058313 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:17.058291 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8950f507-4292-408b-a2d2-39cdd7a94200-util\") pod \"8950f507-4292-408b-a2d2-39cdd7a94200\" (UID: \"8950f507-4292-408b-a2d2-39cdd7a94200\") " Jul 23 11:41:17.058426 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:17.058321 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8950f507-4292-408b-a2d2-39cdd7a94200-bundle\") pod \"8950f507-4292-408b-a2d2-39cdd7a94200\" (UID: \"8950f507-4292-408b-a2d2-39cdd7a94200\") " Jul 23 11:41:17.058426 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:17.058366 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nlq6b\" (UniqueName: \"kubernetes.io/projected/8950f507-4292-408b-a2d2-39cdd7a94200-kube-api-access-nlq6b\") pod \"8950f507-4292-408b-a2d2-39cdd7a94200\" (UID: \"8950f507-4292-408b-a2d2-39cdd7a94200\") " Jul 23 11:41:17.058975 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:17.058948 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8950f507-4292-408b-a2d2-39cdd7a94200-bundle" (OuterVolumeSpecName: "bundle") pod "8950f507-4292-408b-a2d2-39cdd7a94200" (UID: "8950f507-4292-408b-a2d2-39cdd7a94200"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 11:41:17.060334 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:17.060309 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8950f507-4292-408b-a2d2-39cdd7a94200-kube-api-access-nlq6b" (OuterVolumeSpecName: "kube-api-access-nlq6b") pod "8950f507-4292-408b-a2d2-39cdd7a94200" (UID: "8950f507-4292-408b-a2d2-39cdd7a94200"). InnerVolumeSpecName "kube-api-access-nlq6b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 11:41:17.063515 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:17.063492 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8950f507-4292-408b-a2d2-39cdd7a94200-util" (OuterVolumeSpecName: "util") pod "8950f507-4292-408b-a2d2-39cdd7a94200" (UID: "8950f507-4292-408b-a2d2-39cdd7a94200"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 11:41:17.159782 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:17.159739 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlq6b\" (UniqueName: \"kubernetes.io/projected/8950f507-4292-408b-a2d2-39cdd7a94200-kube-api-access-nlq6b\") on node \"ip-10-0-134-236.ec2.internal\" DevicePath \"\"" Jul 23 11:41:17.159782 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:17.159756 2561 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8950f507-4292-408b-a2d2-39cdd7a94200-util\") on node \"ip-10-0-134-236.ec2.internal\" DevicePath \"\"" Jul 23 11:41:17.159782 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:17.159764 2561 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8950f507-4292-408b-a2d2-39cdd7a94200-bundle\") on node \"ip-10-0-134-236.ec2.internal\" DevicePath \"\"" Jul 23 11:41:17.872280 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:17.872259 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdgtsb" Jul 23 11:41:17.872376 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:17.872295 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdgtsb" event={"ID":"8950f507-4292-408b-a2d2-39cdd7a94200","Type":"ContainerDied","Data":"14911fc176289dad4a87aaaf4cf6541db28e7c91805afbee8f1219f3d170798b"} Jul 23 11:41:17.872376 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:17.872339 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14911fc176289dad4a87aaaf4cf6541db28e7c91805afbee8f1219f3d170798b" Jul 23 11:41:27.238541 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:27.238470 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-544c65c659-j4mgd"] Jul 23 11:41:27.238881 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:27.238744 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8950f507-4292-408b-a2d2-39cdd7a94200" containerName="util" Jul 23 11:41:27.238881 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:27.238753 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8950f507-4292-408b-a2d2-39cdd7a94200" containerName="util" Jul 23 11:41:27.238881 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:27.238769 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8950f507-4292-408b-a2d2-39cdd7a94200" containerName="pull" Jul 23 11:41:27.238881 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:27.238774 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8950f507-4292-408b-a2d2-39cdd7a94200" containerName="pull" Jul 23 11:41:27.238881 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:27.238780 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8950f507-4292-408b-a2d2-39cdd7a94200" containerName="extract" Jul 23 11:41:27.238881 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:27.238785 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8950f507-4292-408b-a2d2-39cdd7a94200" containerName="extract" Jul 23 11:41:27.238881 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:27.238830 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8950f507-4292-408b-a2d2-39cdd7a94200" containerName="extract" Jul 23 11:41:27.240513 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:27.240499 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-544c65c659-j4mgd" Jul 23 11:41:27.245397 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:27.245375 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 23 11:41:27.247484 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:27.247466 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 23 11:41:27.259577 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:27.259555 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 23 11:41:27.259790 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:27.259776 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 23 11:41:27.260067 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:27.260053 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 23 11:41:27.260272 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:27.260255 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-r8vzg\"" Jul 23 11:41:27.307483 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:27.307459 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-544c65c659-j4mgd"] Jul 23 11:41:27.322665 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:27.322643 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9c120a07-26e4-46b8-a3d3-d031c4a0c7d3-manager-config\") pod \"lws-controller-manager-544c65c659-j4mgd\" (UID: \"9c120a07-26e4-46b8-a3d3-d031c4a0c7d3\") " pod="openshift-lws-operator/lws-controller-manager-544c65c659-j4mgd" Jul 23 11:41:27.322754 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:27.322670 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9c120a07-26e4-46b8-a3d3-d031c4a0c7d3-cert\") pod \"lws-controller-manager-544c65c659-j4mgd\" (UID: \"9c120a07-26e4-46b8-a3d3-d031c4a0c7d3\") " pod="openshift-lws-operator/lws-controller-manager-544c65c659-j4mgd" Jul 23 11:41:27.322754 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:27.322686 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9c120a07-26e4-46b8-a3d3-d031c4a0c7d3-metrics-cert\") pod \"lws-controller-manager-544c65c659-j4mgd\" (UID: \"9c120a07-26e4-46b8-a3d3-d031c4a0c7d3\") " pod="openshift-lws-operator/lws-controller-manager-544c65c659-j4mgd" Jul 23 11:41:27.322832 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:27.322787 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9ptt\" (UniqueName: \"kubernetes.io/projected/9c120a07-26e4-46b8-a3d3-d031c4a0c7d3-kube-api-access-d9ptt\") pod \"lws-controller-manager-544c65c659-j4mgd\" (UID: \"9c120a07-26e4-46b8-a3d3-d031c4a0c7d3\") " pod="openshift-lws-operator/lws-controller-manager-544c65c659-j4mgd" Jul 23 11:41:27.423872 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:27.423845 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9c120a07-26e4-46b8-a3d3-d031c4a0c7d3-manager-config\") pod \"lws-controller-manager-544c65c659-j4mgd\" (UID: \"9c120a07-26e4-46b8-a3d3-d031c4a0c7d3\") " pod="openshift-lws-operator/lws-controller-manager-544c65c659-j4mgd" Jul 23 11:41:27.423977 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:27.423877 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9c120a07-26e4-46b8-a3d3-d031c4a0c7d3-cert\") pod \"lws-controller-manager-544c65c659-j4mgd\" (UID: \"9c120a07-26e4-46b8-a3d3-d031c4a0c7d3\") " pod="openshift-lws-operator/lws-controller-manager-544c65c659-j4mgd" Jul 23 11:41:27.423977 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:27.423894 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9c120a07-26e4-46b8-a3d3-d031c4a0c7d3-metrics-cert\") pod \"lws-controller-manager-544c65c659-j4mgd\" (UID: \"9c120a07-26e4-46b8-a3d3-d031c4a0c7d3\") " pod="openshift-lws-operator/lws-controller-manager-544c65c659-j4mgd" Jul 23 11:41:27.423977 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:27.423965 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9ptt\" (UniqueName: \"kubernetes.io/projected/9c120a07-26e4-46b8-a3d3-d031c4a0c7d3-kube-api-access-d9ptt\") pod \"lws-controller-manager-544c65c659-j4mgd\" (UID: \"9c120a07-26e4-46b8-a3d3-d031c4a0c7d3\") " pod="openshift-lws-operator/lws-controller-manager-544c65c659-j4mgd" Jul 23 11:41:27.424521 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:27.424504 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9c120a07-26e4-46b8-a3d3-d031c4a0c7d3-manager-config\") pod \"lws-controller-manager-544c65c659-j4mgd\" (UID: \"9c120a07-26e4-46b8-a3d3-d031c4a0c7d3\") " pod="openshift-lws-operator/lws-controller-manager-544c65c659-j4mgd" Jul 23 11:41:27.426268 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:27.426248 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9c120a07-26e4-46b8-a3d3-d031c4a0c7d3-cert\") pod \"lws-controller-manager-544c65c659-j4mgd\" (UID: \"9c120a07-26e4-46b8-a3d3-d031c4a0c7d3\") " pod="openshift-lws-operator/lws-controller-manager-544c65c659-j4mgd" Jul 23 11:41:27.426403 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:27.426386 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9c120a07-26e4-46b8-a3d3-d031c4a0c7d3-metrics-cert\") pod \"lws-controller-manager-544c65c659-j4mgd\" (UID: \"9c120a07-26e4-46b8-a3d3-d031c4a0c7d3\") " pod="openshift-lws-operator/lws-controller-manager-544c65c659-j4mgd" Jul 23 11:41:27.440109 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:27.440090 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9ptt\" (UniqueName: \"kubernetes.io/projected/9c120a07-26e4-46b8-a3d3-d031c4a0c7d3-kube-api-access-d9ptt\") pod \"lws-controller-manager-544c65c659-j4mgd\" (UID: \"9c120a07-26e4-46b8-a3d3-d031c4a0c7d3\") " pod="openshift-lws-operator/lws-controller-manager-544c65c659-j4mgd" Jul 23 11:41:27.548669 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:27.548623 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-544c65c659-j4mgd" Jul 23 11:41:27.721503 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:27.721477 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-544c65c659-j4mgd"] Jul 23 11:41:27.722818 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:41:27.722783 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c120a07_26e4_46b8_a3d3_d031c4a0c7d3.slice/crio-417ea1e6afa55386e5c3cf88d9579f9d0c540e70bfd15ba424e2b7f561859749 WatchSource:0}: Error finding container 417ea1e6afa55386e5c3cf88d9579f9d0c540e70bfd15ba424e2b7f561859749: Status 404 returned error can't find the container with id 417ea1e6afa55386e5c3cf88d9579f9d0c540e70bfd15ba424e2b7f561859749 Jul 23 11:41:27.907568 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:27.907542 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-544c65c659-j4mgd" event={"ID":"9c120a07-26e4-46b8-a3d3-d031c4a0c7d3","Type":"ContainerStarted","Data":"417ea1e6afa55386e5c3cf88d9579f9d0c540e70bfd15ba424e2b7f561859749"} Jul 23 11:41:29.918203 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:29.918175 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-544c65c659-j4mgd" event={"ID":"9c120a07-26e4-46b8-a3d3-d031c4a0c7d3","Type":"ContainerStarted","Data":"e496caf44856673e73239a7e73fb64017a2f39c1bcfafa50e7299657dbb02c8b"} Jul 23 11:41:29.918586 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:29.918307 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-544c65c659-j4mgd" Jul 23 11:41:29.939884 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:29.939826 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-544c65c659-j4mgd" podStartSLOduration=1.659872253 podStartE2EDuration="2.939813404s" totalImagesPullingTime="1.279941151s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:41:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:41:29.939540512 +0000 UTC m=+357.011230497" watchObservedRunningTime="2026-07-23 11:41:29.939813404 +0000 UTC m=+357.011503392" Jul 23 11:41:30.649554 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:30.649517 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7c7646c6cc-xnlnm"] Jul 23 11:41:30.651695 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:30.651680 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7c7646c6cc-xnlnm" Jul 23 11:41:30.657470 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:30.657224 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 23 11:41:30.658920 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:30.658886 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 23 11:41:30.659011 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:30.658979 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-rwjxf\"" Jul 23 11:41:30.659834 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:30.659817 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 23 11:41:30.659895 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:30.659856 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 23 11:41:30.751674 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:30.751652 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d2b3b67b-5cd8-4428-88f1-486ae20e1ead-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7c7646c6cc-xnlnm\" (UID: \"d2b3b67b-5cd8-4428-88f1-486ae20e1ead\") " pod="opendatahub/opendatahub-operator-controller-manager-7c7646c6cc-xnlnm" Jul 23 11:41:30.751760 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:30.751713 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d2b3b67b-5cd8-4428-88f1-486ae20e1ead-webhook-cert\") pod \"opendatahub-operator-controller-manager-7c7646c6cc-xnlnm\" (UID: \"d2b3b67b-5cd8-4428-88f1-486ae20e1ead\") " pod="opendatahub/opendatahub-operator-controller-manager-7c7646c6cc-xnlnm" Jul 23 11:41:30.751760 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:30.751748 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d2b3b67b-5cd8-4428-88f1-486ae20e1ead-manifests\") pod \"opendatahub-operator-controller-manager-7c7646c6cc-xnlnm\" (UID: \"d2b3b67b-5cd8-4428-88f1-486ae20e1ead\") " pod="opendatahub/opendatahub-operator-controller-manager-7c7646c6cc-xnlnm" Jul 23 11:41:30.751826 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:30.751779 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d2b3b67b-5cd8-4428-88f1-486ae20e1ead-tmp\") pod \"opendatahub-operator-controller-manager-7c7646c6cc-xnlnm\" (UID: \"d2b3b67b-5cd8-4428-88f1-486ae20e1ead\") " pod="opendatahub/opendatahub-operator-controller-manager-7c7646c6cc-xnlnm" Jul 23 11:41:30.751885 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:30.751860 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzwcs\" (UniqueName: \"kubernetes.io/projected/d2b3b67b-5cd8-4428-88f1-486ae20e1ead-kube-api-access-qzwcs\") pod \"opendatahub-operator-controller-manager-7c7646c6cc-xnlnm\" (UID: \"d2b3b67b-5cd8-4428-88f1-486ae20e1ead\") " pod="opendatahub/opendatahub-operator-controller-manager-7c7646c6cc-xnlnm" Jul 23 11:41:30.754626 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:30.754607 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7c7646c6cc-xnlnm"] Jul 23 11:41:30.852654 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:30.852626 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzwcs\" (UniqueName: \"kubernetes.io/projected/d2b3b67b-5cd8-4428-88f1-486ae20e1ead-kube-api-access-qzwcs\") pod \"opendatahub-operator-controller-manager-7c7646c6cc-xnlnm\" (UID: \"d2b3b67b-5cd8-4428-88f1-486ae20e1ead\") " pod="opendatahub/opendatahub-operator-controller-manager-7c7646c6cc-xnlnm" Jul 23 11:41:30.852735 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:30.852667 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d2b3b67b-5cd8-4428-88f1-486ae20e1ead-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7c7646c6cc-xnlnm\" (UID: \"d2b3b67b-5cd8-4428-88f1-486ae20e1ead\") " pod="opendatahub/opendatahub-operator-controller-manager-7c7646c6cc-xnlnm" Jul 23 11:41:30.852735 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:30.852683 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d2b3b67b-5cd8-4428-88f1-486ae20e1ead-webhook-cert\") pod \"opendatahub-operator-controller-manager-7c7646c6cc-xnlnm\" (UID: \"d2b3b67b-5cd8-4428-88f1-486ae20e1ead\") " pod="opendatahub/opendatahub-operator-controller-manager-7c7646c6cc-xnlnm" Jul 23 11:41:30.852735 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:30.852705 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d2b3b67b-5cd8-4428-88f1-486ae20e1ead-manifests\") pod \"opendatahub-operator-controller-manager-7c7646c6cc-xnlnm\" (UID: \"d2b3b67b-5cd8-4428-88f1-486ae20e1ead\") " pod="opendatahub/opendatahub-operator-controller-manager-7c7646c6cc-xnlnm" Jul 23 11:41:30.852735 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:30.852728 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d2b3b67b-5cd8-4428-88f1-486ae20e1ead-tmp\") pod \"opendatahub-operator-controller-manager-7c7646c6cc-xnlnm\" (UID: \"d2b3b67b-5cd8-4428-88f1-486ae20e1ead\") " pod="opendatahub/opendatahub-operator-controller-manager-7c7646c6cc-xnlnm" Jul 23 11:41:30.853097 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:30.853079 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d2b3b67b-5cd8-4428-88f1-486ae20e1ead-manifests\") pod \"opendatahub-operator-controller-manager-7c7646c6cc-xnlnm\" (UID: \"d2b3b67b-5cd8-4428-88f1-486ae20e1ead\") " pod="opendatahub/opendatahub-operator-controller-manager-7c7646c6cc-xnlnm" Jul 23 11:41:30.855032 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:30.855014 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d2b3b67b-5cd8-4428-88f1-486ae20e1ead-tmp\") pod \"opendatahub-operator-controller-manager-7c7646c6cc-xnlnm\" (UID: \"d2b3b67b-5cd8-4428-88f1-486ae20e1ead\") " pod="opendatahub/opendatahub-operator-controller-manager-7c7646c6cc-xnlnm" Jul 23 11:41:30.855352 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:30.855333 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d2b3b67b-5cd8-4428-88f1-486ae20e1ead-webhook-cert\") pod \"opendatahub-operator-controller-manager-7c7646c6cc-xnlnm\" (UID: \"d2b3b67b-5cd8-4428-88f1-486ae20e1ead\") " pod="opendatahub/opendatahub-operator-controller-manager-7c7646c6cc-xnlnm" Jul 23 11:41:30.855413 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:30.855333 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d2b3b67b-5cd8-4428-88f1-486ae20e1ead-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7c7646c6cc-xnlnm\" (UID: \"d2b3b67b-5cd8-4428-88f1-486ae20e1ead\") " pod="opendatahub/opendatahub-operator-controller-manager-7c7646c6cc-xnlnm" Jul 23 11:41:30.877928 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:30.877896 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzwcs\" (UniqueName: \"kubernetes.io/projected/d2b3b67b-5cd8-4428-88f1-486ae20e1ead-kube-api-access-qzwcs\") pod \"opendatahub-operator-controller-manager-7c7646c6cc-xnlnm\" (UID: \"d2b3b67b-5cd8-4428-88f1-486ae20e1ead\") " pod="opendatahub/opendatahub-operator-controller-manager-7c7646c6cc-xnlnm" Jul 23 11:41:30.960707 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:30.960655 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7c7646c6cc-xnlnm" Jul 23 11:41:31.142538 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:31.142512 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7c7646c6cc-xnlnm"] Jul 23 11:41:31.142764 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:41:31.142744 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2b3b67b_5cd8_4428_88f1_486ae20e1ead.slice/crio-fb63162e96d7b6d384e9ce278abafee0efc5d3e276f44540c24a5f175948dfb9 WatchSource:0}: Error finding container fb63162e96d7b6d384e9ce278abafee0efc5d3e276f44540c24a5f175948dfb9: Status 404 returned error can't find the container with id fb63162e96d7b6d384e9ce278abafee0efc5d3e276f44540c24a5f175948dfb9 Jul 23 11:41:31.928543 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:31.928509 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7c7646c6cc-xnlnm" event={"ID":"d2b3b67b-5cd8-4428-88f1-486ae20e1ead","Type":"ContainerStarted","Data":"fb63162e96d7b6d384e9ce278abafee0efc5d3e276f44540c24a5f175948dfb9"} Jul 23 11:41:33.937998 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:33.937967 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7c7646c6cc-xnlnm" event={"ID":"d2b3b67b-5cd8-4428-88f1-486ae20e1ead","Type":"ContainerStarted","Data":"a264a23d177f3fda9e75c696f9ddc6469ac248ade0b791323877d0aac8edac72"} Jul 23 11:41:34.944014 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:34.943987 2561 generic.go:356] "Generic (PLEG): container finished" podID="d2b3b67b-5cd8-4428-88f1-486ae20e1ead" containerID="a264a23d177f3fda9e75c696f9ddc6469ac248ade0b791323877d0aac8edac72" exitCode=0 Jul 23 11:41:34.944313 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:34.944032 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7c7646c6cc-xnlnm" event={"ID":"d2b3b67b-5cd8-4428-88f1-486ae20e1ead","Type":"ContainerDied","Data":"a264a23d177f3fda9e75c696f9ddc6469ac248ade0b791323877d0aac8edac72"} Jul 23 11:41:35.951629 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:35.951593 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7c7646c6cc-xnlnm" event={"ID":"d2b3b67b-5cd8-4428-88f1-486ae20e1ead","Type":"ContainerStarted","Data":"71b7e7a10176f9cf37e7f6fec07981f3816770e99107cc6165d578eede359366"} Jul 23 11:41:35.952030 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:35.951725 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7c7646c6cc-xnlnm" Jul 23 11:41:35.977221 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:35.977177 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7c7646c6cc-xnlnm" podStartSLOduration=2.000395451 podStartE2EDuration="5.977164345s" totalImagesPullingTime="3.052872943s" totalInitContainerRuntime="923.895951ms" isStatefulPod=false podCreationTimestamp="2026-07-23 11:41:30 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:41:35.976793345 +0000 UTC m=+363.048483334" watchObservedRunningTime="2026-07-23 11:41:35.977164345 +0000 UTC m=+363.048854336" Jul 23 11:41:40.924435 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:40.924408 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-544c65c659-j4mgd" Jul 23 11:41:46.958391 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:46.958358 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7c7646c6cc-xnlnm" Jul 23 11:41:58.734884 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:58.734845 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-d67vc"] Jul 23 11:41:58.738281 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:58.738259 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-d67vc" Jul 23 11:41:58.741138 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:58.741114 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 23 11:41:58.741620 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:58.741600 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 23 11:41:58.741696 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:58.741625 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-sfhtx\"" Jul 23 11:41:58.741696 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:58.741606 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 23 11:41:58.741889 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:58.741876 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 23 11:41:58.758971 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:58.758946 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-d67vc"] Jul 23 11:41:58.761132 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:58.761102 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ceaf1c1b-b54f-48f9-bcaf-58a505669805-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-d67vc\" (UID: \"ceaf1c1b-b54f-48f9-bcaf-58a505669805\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-d67vc" Jul 23 11:41:58.761240 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:58.761134 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ceaf1c1b-b54f-48f9-bcaf-58a505669805-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-d67vc\" (UID: \"ceaf1c1b-b54f-48f9-bcaf-58a505669805\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-d67vc" Jul 23 11:41:58.761240 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:58.761161 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ceaf1c1b-b54f-48f9-bcaf-58a505669805-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-d67vc\" (UID: \"ceaf1c1b-b54f-48f9-bcaf-58a505669805\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-d67vc" Jul 23 11:41:58.761240 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:58.761235 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mql4\" (UniqueName: \"kubernetes.io/projected/ceaf1c1b-b54f-48f9-bcaf-58a505669805-kube-api-access-2mql4\") pod \"istiod-openshift-gateway-94bb8fbfd-d67vc\" (UID: \"ceaf1c1b-b54f-48f9-bcaf-58a505669805\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-d67vc" Jul 23 11:41:58.761376 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:58.761273 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ceaf1c1b-b54f-48f9-bcaf-58a505669805-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-d67vc\" (UID: \"ceaf1c1b-b54f-48f9-bcaf-58a505669805\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-d67vc" Jul 23 11:41:58.761376 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:58.761313 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ceaf1c1b-b54f-48f9-bcaf-58a505669805-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-d67vc\" (UID: \"ceaf1c1b-b54f-48f9-bcaf-58a505669805\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-d67vc" Jul 23 11:41:58.761376 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:58.761342 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ceaf1c1b-b54f-48f9-bcaf-58a505669805-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-d67vc\" (UID: \"ceaf1c1b-b54f-48f9-bcaf-58a505669805\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-d67vc" Jul 23 11:41:58.862259 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:58.862222 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mql4\" (UniqueName: \"kubernetes.io/projected/ceaf1c1b-b54f-48f9-bcaf-58a505669805-kube-api-access-2mql4\") pod \"istiod-openshift-gateway-94bb8fbfd-d67vc\" (UID: \"ceaf1c1b-b54f-48f9-bcaf-58a505669805\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-d67vc" Jul 23 11:41:58.862456 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:58.862266 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ceaf1c1b-b54f-48f9-bcaf-58a505669805-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-d67vc\" (UID: \"ceaf1c1b-b54f-48f9-bcaf-58a505669805\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-d67vc" Jul 23 11:41:58.862456 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:58.862286 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ceaf1c1b-b54f-48f9-bcaf-58a505669805-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-d67vc\" (UID: \"ceaf1c1b-b54f-48f9-bcaf-58a505669805\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-d67vc" Jul 23 11:41:58.862456 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:58.862311 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ceaf1c1b-b54f-48f9-bcaf-58a505669805-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-d67vc\" (UID: \"ceaf1c1b-b54f-48f9-bcaf-58a505669805\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-d67vc" Jul 23 11:41:58.862456 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:58.862336 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ceaf1c1b-b54f-48f9-bcaf-58a505669805-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-d67vc\" (UID: \"ceaf1c1b-b54f-48f9-bcaf-58a505669805\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-d67vc" Jul 23 11:41:58.862456 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:58.862359 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ceaf1c1b-b54f-48f9-bcaf-58a505669805-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-d67vc\" (UID: \"ceaf1c1b-b54f-48f9-bcaf-58a505669805\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-d67vc" Jul 23 11:41:58.862456 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:58.862381 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ceaf1c1b-b54f-48f9-bcaf-58a505669805-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-d67vc\" (UID: \"ceaf1c1b-b54f-48f9-bcaf-58a505669805\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-d67vc" Jul 23 11:41:58.863037 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:58.862994 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ceaf1c1b-b54f-48f9-bcaf-58a505669805-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-d67vc\" (UID: \"ceaf1c1b-b54f-48f9-bcaf-58a505669805\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-d67vc" Jul 23 11:41:58.864763 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:58.864735 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ceaf1c1b-b54f-48f9-bcaf-58a505669805-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-d67vc\" (UID: \"ceaf1c1b-b54f-48f9-bcaf-58a505669805\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-d67vc" Jul 23 11:41:58.864875 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:58.864818 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ceaf1c1b-b54f-48f9-bcaf-58a505669805-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-d67vc\" (UID: \"ceaf1c1b-b54f-48f9-bcaf-58a505669805\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-d67vc" Jul 23 11:41:58.864875 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:58.864840 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ceaf1c1b-b54f-48f9-bcaf-58a505669805-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-d67vc\" (UID: \"ceaf1c1b-b54f-48f9-bcaf-58a505669805\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-d67vc" Jul 23 11:41:58.865008 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:58.864891 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ceaf1c1b-b54f-48f9-bcaf-58a505669805-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-d67vc\" (UID: \"ceaf1c1b-b54f-48f9-bcaf-58a505669805\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-d67vc" Jul 23 11:41:58.871603 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:58.871580 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mql4\" (UniqueName: \"kubernetes.io/projected/ceaf1c1b-b54f-48f9-bcaf-58a505669805-kube-api-access-2mql4\") pod \"istiod-openshift-gateway-94bb8fbfd-d67vc\" (UID: \"ceaf1c1b-b54f-48f9-bcaf-58a505669805\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-d67vc" Jul 23 11:41:58.871742 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:58.871724 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ceaf1c1b-b54f-48f9-bcaf-58a505669805-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-d67vc\" (UID: \"ceaf1c1b-b54f-48f9-bcaf-58a505669805\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-d67vc" Jul 23 11:41:59.047505 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:59.047424 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-d67vc" Jul 23 11:41:59.220429 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:41:59.220396 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-d67vc"] Jul 23 11:41:59.223805 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:41:59.223764 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceaf1c1b_b54f_48f9_bcaf_58a505669805.slice/crio-6f8772ed73c48c3dd2f3b0d640c286d6934264e5ffeeb2e82ea8365ce427ec53 WatchSource:0}: Error finding container 6f8772ed73c48c3dd2f3b0d640c286d6934264e5ffeeb2e82ea8365ce427ec53: Status 404 returned error can't find the container with id 6f8772ed73c48c3dd2f3b0d640c286d6934264e5ffeeb2e82ea8365ce427ec53 Jul 23 11:42:00.064438 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:42:00.064405 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-d67vc" event={"ID":"ceaf1c1b-b54f-48f9-bcaf-58a505669805","Type":"ContainerStarted","Data":"6f8772ed73c48c3dd2f3b0d640c286d6934264e5ffeeb2e82ea8365ce427ec53"} Jul 23 11:42:03.026939 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:42:03.026886 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 11:42:03.027250 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:42:03.026967 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 23 11:42:04.082707 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:42:04.082671 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-d67vc" event={"ID":"ceaf1c1b-b54f-48f9-bcaf-58a505669805","Type":"ContainerStarted","Data":"baa5223cceebb9839aaaa46f586258777fe217a5726fac4eb94ad06c66b7e465"} Jul 23 11:42:04.083160 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:42:04.082827 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-d67vc" Jul 23 11:42:04.084468 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:42:04.084442 2561 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-d67vc container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 23 11:42:04.084561 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:42:04.084488 2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-d67vc" podUID="ceaf1c1b-b54f-48f9-bcaf-58a505669805" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 23 11:42:04.145029 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:42:04.144982 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-d67vc" podStartSLOduration=2.344148455 podStartE2EDuration="6.144964465s" totalImagesPullingTime="3.80081601s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:41:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:42:04.142672566 +0000 UTC m=+391.214362556" watchObservedRunningTime="2026-07-23 11:42:04.144964465 +0000 UTC m=+391.216654456" Jul 23 11:42:05.087199 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:42:05.087169 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-d67vc" Jul 23 11:43:07.881817 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:07.881742 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73b2dck"] Jul 23 11:43:07.884319 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:07.884303 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73b2dck" Jul 23 11:43:07.887002 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:07.886975 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 23 11:43:07.887890 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:07.887835 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 23 11:43:07.887890 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:07.887883 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-gqsgb\"" Jul 23 11:43:07.894169 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:07.894147 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73b2dck"] Jul 23 11:43:07.899946 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:07.899925 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0748219c-2587-496d-a3c1-b218b6748b7c-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73b2dck\" (UID: \"0748219c-2587-496d-a3c1-b218b6748b7c\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73b2dck" Jul 23 11:43:07.900036 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:07.899957 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4rlg\" (UniqueName: \"kubernetes.io/projected/0748219c-2587-496d-a3c1-b218b6748b7c-kube-api-access-h4rlg\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73b2dck\" (UID: \"0748219c-2587-496d-a3c1-b218b6748b7c\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73b2dck" Jul 23 11:43:07.900036 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:07.899985 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0748219c-2587-496d-a3c1-b218b6748b7c-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73b2dck\" (UID: \"0748219c-2587-496d-a3c1-b218b6748b7c\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73b2dck" Jul 23 11:43:08.000631 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:08.000609 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0748219c-2587-496d-a3c1-b218b6748b7c-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73b2dck\" (UID: \"0748219c-2587-496d-a3c1-b218b6748b7c\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73b2dck" Jul 23 11:43:08.000728 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:08.000639 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4rlg\" (UniqueName: \"kubernetes.io/projected/0748219c-2587-496d-a3c1-b218b6748b7c-kube-api-access-h4rlg\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73b2dck\" (UID: \"0748219c-2587-496d-a3c1-b218b6748b7c\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73b2dck" Jul 23 11:43:08.000728 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:08.000666 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0748219c-2587-496d-a3c1-b218b6748b7c-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73b2dck\" (UID: \"0748219c-2587-496d-a3c1-b218b6748b7c\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73b2dck" Jul 23 11:43:08.000969 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:08.000951 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0748219c-2587-496d-a3c1-b218b6748b7c-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73b2dck\" (UID: \"0748219c-2587-496d-a3c1-b218b6748b7c\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73b2dck" Jul 23 11:43:08.001030 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:08.001014 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0748219c-2587-496d-a3c1-b218b6748b7c-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73b2dck\" (UID: \"0748219c-2587-496d-a3c1-b218b6748b7c\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73b2dck" Jul 23 11:43:08.010120 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:08.010091 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4rlg\" (UniqueName: \"kubernetes.io/projected/0748219c-2587-496d-a3c1-b218b6748b7c-kube-api-access-h4rlg\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73b2dck\" (UID: \"0748219c-2587-496d-a3c1-b218b6748b7c\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73b2dck" Jul 23 11:43:08.193445 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:08.193393 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73b2dck" Jul 23 11:43:08.322296 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:08.322275 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73b2dck"] Jul 23 11:43:08.324478 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:43:08.324453 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0748219c_2587_496d_a3c1_b218b6748b7c.slice/crio-f9703a2d7131bdcd69cfc34418b065d8780a169fd8ffe5d646fadffd408c4ace WatchSource:0}: Error finding container f9703a2d7131bdcd69cfc34418b065d8780a169fd8ffe5d646fadffd408c4ace: Status 404 returned error can't find the container with id f9703a2d7131bdcd69cfc34418b065d8780a169fd8ffe5d646fadffd408c4ace Jul 23 11:43:08.355009 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:08.354987 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73b2dck" event={"ID":"0748219c-2587-496d-a3c1-b218b6748b7c","Type":"ContainerStarted","Data":"f9703a2d7131bdcd69cfc34418b065d8780a169fd8ffe5d646fadffd408c4ace"} Jul 23 11:43:08.476088 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:08.476034 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xz224"] Jul 23 11:43:08.478726 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:08.478709 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xz224" Jul 23 11:43:08.493186 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:08.493154 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xz224"] Jul 23 11:43:08.503689 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:08.503668 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b4233887-c309-4a1f-9759-363bc6092fd9-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xz224\" (UID: \"b4233887-c309-4a1f-9759-363bc6092fd9\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xz224" Jul 23 11:43:08.503775 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:08.503724 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b4233887-c309-4a1f-9759-363bc6092fd9-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xz224\" (UID: \"b4233887-c309-4a1f-9759-363bc6092fd9\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xz224" Jul 23 11:43:08.503775 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:08.503753 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkdtd\" (UniqueName: \"kubernetes.io/projected/b4233887-c309-4a1f-9759-363bc6092fd9-kube-api-access-dkdtd\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xz224\" (UID: \"b4233887-c309-4a1f-9759-363bc6092fd9\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xz224" Jul 23 11:43:08.604186 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:08.604159 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b4233887-c309-4a1f-9759-363bc6092fd9-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xz224\" (UID: \"b4233887-c309-4a1f-9759-363bc6092fd9\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xz224" Jul 23 11:43:08.604308 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:08.604194 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkdtd\" (UniqueName: \"kubernetes.io/projected/b4233887-c309-4a1f-9759-363bc6092fd9-kube-api-access-dkdtd\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xz224\" (UID: \"b4233887-c309-4a1f-9759-363bc6092fd9\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xz224" Jul 23 11:43:08.604308 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:08.604242 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b4233887-c309-4a1f-9759-363bc6092fd9-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xz224\" (UID: \"b4233887-c309-4a1f-9759-363bc6092fd9\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xz224" Jul 23 11:43:08.604501 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:08.604483 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b4233887-c309-4a1f-9759-363bc6092fd9-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xz224\" (UID: \"b4233887-c309-4a1f-9759-363bc6092fd9\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xz224" Jul 23 11:43:08.604551 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:08.604507 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b4233887-c309-4a1f-9759-363bc6092fd9-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xz224\" (UID: \"b4233887-c309-4a1f-9759-363bc6092fd9\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xz224" Jul 23 11:43:08.612832 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:08.612809 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkdtd\" (UniqueName: \"kubernetes.io/projected/b4233887-c309-4a1f-9759-363bc6092fd9-kube-api-access-dkdtd\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xz224\" (UID: \"b4233887-c309-4a1f-9759-363bc6092fd9\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xz224" Jul 23 11:43:08.796002 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:08.795933 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xz224" Jul 23 11:43:08.919827 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:08.919804 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xz224"] Jul 23 11:43:08.921756 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:43:08.921736 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4233887_c309_4a1f_9759_363bc6092fd9.slice/crio-9840326f56968ff89bfc83737036b248df65160af6b31cd08dc0740fad4a097e WatchSource:0}: Error finding container 9840326f56968ff89bfc83737036b248df65160af6b31cd08dc0740fad4a097e: Status 404 returned error can't find the container with id 9840326f56968ff89bfc83737036b248df65160af6b31cd08dc0740fad4a097e Jul 23 11:43:09.080918 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:09.080887 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zhkzz"] Jul 23 11:43:09.083308 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:09.083294 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zhkzz" Jul 23 11:43:09.092463 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:09.092444 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zhkzz"] Jul 23 11:43:09.106668 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:09.106648 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2a8af5b7-35f7-4208-bf9f-f58d30954ae0-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zhkzz\" (UID: \"2a8af5b7-35f7-4208-bf9f-f58d30954ae0\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zhkzz" Jul 23 11:43:09.106767 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:09.106679 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2a8af5b7-35f7-4208-bf9f-f58d30954ae0-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zhkzz\" (UID: \"2a8af5b7-35f7-4208-bf9f-f58d30954ae0\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zhkzz" Jul 23 11:43:09.106884 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:09.106868 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szdpr\" (UniqueName: \"kubernetes.io/projected/2a8af5b7-35f7-4208-bf9f-f58d30954ae0-kube-api-access-szdpr\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zhkzz\" (UID: \"2a8af5b7-35f7-4208-bf9f-f58d30954ae0\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zhkzz" Jul 23 11:43:09.207788 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:09.207760 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szdpr\" (UniqueName: \"kubernetes.io/projected/2a8af5b7-35f7-4208-bf9f-f58d30954ae0-kube-api-access-szdpr\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zhkzz\" (UID: \"2a8af5b7-35f7-4208-bf9f-f58d30954ae0\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zhkzz" Jul 23 11:43:09.207961 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:09.207802 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2a8af5b7-35f7-4208-bf9f-f58d30954ae0-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zhkzz\" (UID: \"2a8af5b7-35f7-4208-bf9f-f58d30954ae0\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zhkzz" Jul 23 11:43:09.207961 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:09.207820 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2a8af5b7-35f7-4208-bf9f-f58d30954ae0-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zhkzz\" (UID: \"2a8af5b7-35f7-4208-bf9f-f58d30954ae0\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zhkzz" Jul 23 11:43:09.208142 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:09.208121 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2a8af5b7-35f7-4208-bf9f-f58d30954ae0-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zhkzz\" (UID: \"2a8af5b7-35f7-4208-bf9f-f58d30954ae0\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zhkzz" Jul 23 11:43:09.208190 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:09.208149 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2a8af5b7-35f7-4208-bf9f-f58d30954ae0-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zhkzz\" (UID: \"2a8af5b7-35f7-4208-bf9f-f58d30954ae0\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zhkzz" Jul 23 11:43:09.216767 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:09.216744 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-szdpr\" (UniqueName: \"kubernetes.io/projected/2a8af5b7-35f7-4208-bf9f-f58d30954ae0-kube-api-access-szdpr\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zhkzz\" (UID: \"2a8af5b7-35f7-4208-bf9f-f58d30954ae0\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zhkzz" Jul 23 11:43:09.360851 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:09.360781 2561 generic.go:356] "Generic (PLEG): container finished" podID="b4233887-c309-4a1f-9759-363bc6092fd9" containerID="0c4f365d412bc20d8e0529e9ba54dfb9c749acf6a3b6a30da3449a74a9551e23" exitCode=0 Jul 23 11:43:09.361005 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:09.360878 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xz224" event={"ID":"b4233887-c309-4a1f-9759-363bc6092fd9","Type":"ContainerDied","Data":"0c4f365d412bc20d8e0529e9ba54dfb9c749acf6a3b6a30da3449a74a9551e23"} Jul 23 11:43:09.361005 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:09.360956 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xz224" event={"ID":"b4233887-c309-4a1f-9759-363bc6092fd9","Type":"ContainerStarted","Data":"9840326f56968ff89bfc83737036b248df65160af6b31cd08dc0740fad4a097e"} Jul 23 11:43:09.362488 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:09.362469 2561 generic.go:356] "Generic (PLEG): container finished" podID="0748219c-2587-496d-a3c1-b218b6748b7c" containerID="db20444b0c04163013aecf5be810d8cc070450ea3cc18562f6fe06b5d6c63a51" exitCode=0 Jul 23 11:43:09.362567 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:09.362506 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73b2dck" event={"ID":"0748219c-2587-496d-a3c1-b218b6748b7c","Type":"ContainerDied","Data":"db20444b0c04163013aecf5be810d8cc070450ea3cc18562f6fe06b5d6c63a51"} Jul 23 11:43:09.392589 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:09.392566 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zhkzz" Jul 23 11:43:09.479179 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:09.479154 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nj7wg"] Jul 23 11:43:09.483414 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:09.483393 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nj7wg" Jul 23 11:43:09.493516 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:09.493495 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nj7wg"] Jul 23 11:43:09.510444 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:09.510421 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ffef83f1-4832-40f6-a8ca-8e62bc8db03b-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nj7wg\" (UID: \"ffef83f1-4832-40f6-a8ca-8e62bc8db03b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nj7wg" Jul 23 11:43:09.510569 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:09.510457 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ffef83f1-4832-40f6-a8ca-8e62bc8db03b-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nj7wg\" (UID: \"ffef83f1-4832-40f6-a8ca-8e62bc8db03b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nj7wg" Jul 23 11:43:09.510569 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:09.510526 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm7x7\" (UniqueName: \"kubernetes.io/projected/ffef83f1-4832-40f6-a8ca-8e62bc8db03b-kube-api-access-lm7x7\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nj7wg\" (UID: \"ffef83f1-4832-40f6-a8ca-8e62bc8db03b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nj7wg" Jul 23 11:43:09.531787 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:09.531768 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zhkzz"] Jul 23 11:43:09.534095 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:43:09.534071 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a8af5b7_35f7_4208_bf9f_f58d30954ae0.slice/crio-341bd9ba2c3b5a94e5f48b2e2409f511c4b5c55c7eb0a45765572fa836b87240 WatchSource:0}: Error finding container 341bd9ba2c3b5a94e5f48b2e2409f511c4b5c55c7eb0a45765572fa836b87240: Status 404 returned error can't find the container with id 341bd9ba2c3b5a94e5f48b2e2409f511c4b5c55c7eb0a45765572fa836b87240 Jul 23 11:43:09.611038 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:09.610992 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm7x7\" (UniqueName: \"kubernetes.io/projected/ffef83f1-4832-40f6-a8ca-8e62bc8db03b-kube-api-access-lm7x7\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nj7wg\" (UID: \"ffef83f1-4832-40f6-a8ca-8e62bc8db03b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nj7wg" Jul 23 11:43:09.611114 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:09.611067 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ffef83f1-4832-40f6-a8ca-8e62bc8db03b-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nj7wg\" (UID: \"ffef83f1-4832-40f6-a8ca-8e62bc8db03b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nj7wg" Jul 23 11:43:09.611162 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:09.611117 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ffef83f1-4832-40f6-a8ca-8e62bc8db03b-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nj7wg\" (UID: \"ffef83f1-4832-40f6-a8ca-8e62bc8db03b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nj7wg" Jul 23 11:43:09.611371 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:09.611354 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ffef83f1-4832-40f6-a8ca-8e62bc8db03b-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nj7wg\" (UID: \"ffef83f1-4832-40f6-a8ca-8e62bc8db03b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nj7wg" Jul 23 11:43:09.611404 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:09.611392 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ffef83f1-4832-40f6-a8ca-8e62bc8db03b-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nj7wg\" (UID: \"ffef83f1-4832-40f6-a8ca-8e62bc8db03b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nj7wg" Jul 23 11:43:09.619860 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:09.619837 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm7x7\" (UniqueName: \"kubernetes.io/projected/ffef83f1-4832-40f6-a8ca-8e62bc8db03b-kube-api-access-lm7x7\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nj7wg\" (UID: \"ffef83f1-4832-40f6-a8ca-8e62bc8db03b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nj7wg" Jul 23 11:43:09.794195 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:09.794169 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nj7wg" Jul 23 11:43:09.924094 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:09.924072 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nj7wg"] Jul 23 11:43:09.926073 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:43:09.926037 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffef83f1_4832_40f6_a8ca_8e62bc8db03b.slice/crio-098ea4e0fd9c8dea5b76372ee665777e604bde01bd0880f8db4dfdacc7236c02 WatchSource:0}: Error finding container 098ea4e0fd9c8dea5b76372ee665777e604bde01bd0880f8db4dfdacc7236c02: Status 404 returned error can't find the container with id 098ea4e0fd9c8dea5b76372ee665777e604bde01bd0880f8db4dfdacc7236c02 Jul 23 11:43:10.368659 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:10.368633 2561 generic.go:356] "Generic (PLEG): container finished" podID="ffef83f1-4832-40f6-a8ca-8e62bc8db03b" containerID="b117e2f61cd1d244ad48a598534ee1293bd46e6628507a7a8497dad911beb809" exitCode=0 Jul 23 11:43:10.368800 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:10.368726 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nj7wg" event={"ID":"ffef83f1-4832-40f6-a8ca-8e62bc8db03b","Type":"ContainerDied","Data":"b117e2f61cd1d244ad48a598534ee1293bd46e6628507a7a8497dad911beb809"} Jul 23 11:43:10.368800 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:10.368771 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nj7wg" event={"ID":"ffef83f1-4832-40f6-a8ca-8e62bc8db03b","Type":"ContainerStarted","Data":"098ea4e0fd9c8dea5b76372ee665777e604bde01bd0880f8db4dfdacc7236c02"} Jul 23 11:43:10.370592 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:10.370569 2561 generic.go:356] "Generic (PLEG): container finished" podID="0748219c-2587-496d-a3c1-b218b6748b7c" containerID="a01a012dfd0e6c4188f73b3ea9542793326e79682a34bcd157da0f97b78e0f51" exitCode=0 Jul 23 11:43:10.370714 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:10.370640 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73b2dck" event={"ID":"0748219c-2587-496d-a3c1-b218b6748b7c","Type":"ContainerDied","Data":"a01a012dfd0e6c4188f73b3ea9542793326e79682a34bcd157da0f97b78e0f51"} Jul 23 11:43:10.372268 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:10.372251 2561 generic.go:356] "Generic (PLEG): container finished" podID="2a8af5b7-35f7-4208-bf9f-f58d30954ae0" containerID="cbaa458a7fb2331b2017cbccbb089ca92bf8f91274fc7593cd4a009e0fbc4b87" exitCode=0 Jul 23 11:43:10.372356 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:10.372332 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zhkzz" event={"ID":"2a8af5b7-35f7-4208-bf9f-f58d30954ae0","Type":"ContainerDied","Data":"cbaa458a7fb2331b2017cbccbb089ca92bf8f91274fc7593cd4a009e0fbc4b87"} Jul 23 11:43:10.372401 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:10.372358 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zhkzz" event={"ID":"2a8af5b7-35f7-4208-bf9f-f58d30954ae0","Type":"ContainerStarted","Data":"341bd9ba2c3b5a94e5f48b2e2409f511c4b5c55c7eb0a45765572fa836b87240"} Jul 23 11:43:11.378838 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:11.378809 2561 generic.go:356] "Generic (PLEG): container finished" podID="b4233887-c309-4a1f-9759-363bc6092fd9" containerID="2aaafa3ad95724de42b454461f48e866cf847b67be0926e0bf7e342c98f9f07f" exitCode=0 Jul 23 11:43:11.379187 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:11.378883 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xz224" event={"ID":"b4233887-c309-4a1f-9759-363bc6092fd9","Type":"ContainerDied","Data":"2aaafa3ad95724de42b454461f48e866cf847b67be0926e0bf7e342c98f9f07f"} Jul 23 11:43:11.380803 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:11.380781 2561 generic.go:356] "Generic (PLEG): container finished" podID="ffef83f1-4832-40f6-a8ca-8e62bc8db03b" containerID="99c63283301bbbbfcc5e303b2de362cf6684798fe1bab002a2c80495c7b2f4fa" exitCode=0 Jul 23 11:43:11.380896 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:11.380809 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nj7wg" event={"ID":"ffef83f1-4832-40f6-a8ca-8e62bc8db03b","Type":"ContainerDied","Data":"99c63283301bbbbfcc5e303b2de362cf6684798fe1bab002a2c80495c7b2f4fa"} Jul 23 11:43:11.383752 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:11.383566 2561 generic.go:356] "Generic (PLEG): container finished" podID="0748219c-2587-496d-a3c1-b218b6748b7c" containerID="9844ab5992c2def8bd17175509239fc2aeaa6cbc47d8cd8812c8bb43d710c2cd" exitCode=0 Jul 23 11:43:11.383752 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:11.383746 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73b2dck" event={"ID":"0748219c-2587-496d-a3c1-b218b6748b7c","Type":"ContainerDied","Data":"9844ab5992c2def8bd17175509239fc2aeaa6cbc47d8cd8812c8bb43d710c2cd"} Jul 23 11:43:12.390397 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:12.390363 2561 generic.go:356] "Generic (PLEG): container finished" podID="2a8af5b7-35f7-4208-bf9f-f58d30954ae0" containerID="237992ad8de4e4bd3209ec23e6b761be19f7b8e7c56f5b62e96fdd905b6f7224" exitCode=0 Jul 23 11:43:12.390787 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:12.390444 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zhkzz" event={"ID":"2a8af5b7-35f7-4208-bf9f-f58d30954ae0","Type":"ContainerDied","Data":"237992ad8de4e4bd3209ec23e6b761be19f7b8e7c56f5b62e96fdd905b6f7224"} Jul 23 11:43:12.392613 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:12.392586 2561 generic.go:356] "Generic (PLEG): container finished" podID="b4233887-c309-4a1f-9759-363bc6092fd9" containerID="fe27175ef39741b871a983823798177d66f63a4b90875d204073e4d493703eed" exitCode=0 Jul 23 11:43:12.392696 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:12.392666 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xz224" event={"ID":"b4233887-c309-4a1f-9759-363bc6092fd9","Type":"ContainerDied","Data":"fe27175ef39741b871a983823798177d66f63a4b90875d204073e4d493703eed"} Jul 23 11:43:12.394650 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:12.394593 2561 generic.go:356] "Generic (PLEG): container finished" podID="ffef83f1-4832-40f6-a8ca-8e62bc8db03b" containerID="27e52d18ab2c1699b5070cae346930fc147f0fdb0c28c04b32b6cec60c07528e" exitCode=0 Jul 23 11:43:12.394714 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:12.394676 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nj7wg" event={"ID":"ffef83f1-4832-40f6-a8ca-8e62bc8db03b","Type":"ContainerDied","Data":"27e52d18ab2c1699b5070cae346930fc147f0fdb0c28c04b32b6cec60c07528e"} Jul 23 11:43:12.529041 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:12.529014 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73b2dck" Jul 23 11:43:12.631933 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:12.631879 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h4rlg\" (UniqueName: \"kubernetes.io/projected/0748219c-2587-496d-a3c1-b218b6748b7c-kube-api-access-h4rlg\") pod \"0748219c-2587-496d-a3c1-b218b6748b7c\" (UID: \"0748219c-2587-496d-a3c1-b218b6748b7c\") " Jul 23 11:43:12.632119 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:12.631978 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0748219c-2587-496d-a3c1-b218b6748b7c-util\") pod \"0748219c-2587-496d-a3c1-b218b6748b7c\" (UID: \"0748219c-2587-496d-a3c1-b218b6748b7c\") " Jul 23 11:43:12.632119 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:12.632008 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0748219c-2587-496d-a3c1-b218b6748b7c-bundle\") pod \"0748219c-2587-496d-a3c1-b218b6748b7c\" (UID: \"0748219c-2587-496d-a3c1-b218b6748b7c\") " Jul 23 11:43:12.632536 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:12.632514 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0748219c-2587-496d-a3c1-b218b6748b7c-bundle" (OuterVolumeSpecName: "bundle") pod "0748219c-2587-496d-a3c1-b218b6748b7c" (UID: "0748219c-2587-496d-a3c1-b218b6748b7c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 11:43:12.634227 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:12.634210 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0748219c-2587-496d-a3c1-b218b6748b7c-kube-api-access-h4rlg" (OuterVolumeSpecName: "kube-api-access-h4rlg") pod "0748219c-2587-496d-a3c1-b218b6748b7c" (UID: "0748219c-2587-496d-a3c1-b218b6748b7c"). InnerVolumeSpecName "kube-api-access-h4rlg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 11:43:12.637110 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:12.637080 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0748219c-2587-496d-a3c1-b218b6748b7c-util" (OuterVolumeSpecName: "util") pod "0748219c-2587-496d-a3c1-b218b6748b7c" (UID: "0748219c-2587-496d-a3c1-b218b6748b7c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 11:43:12.733184 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:12.733107 2561 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0748219c-2587-496d-a3c1-b218b6748b7c-util\") on node \"ip-10-0-134-236.ec2.internal\" DevicePath \"\"" Jul 23 11:43:12.733184 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:12.733133 2561 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0748219c-2587-496d-a3c1-b218b6748b7c-bundle\") on node \"ip-10-0-134-236.ec2.internal\" DevicePath \"\"" Jul 23 11:43:12.733184 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:12.733143 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4rlg\" (UniqueName: \"kubernetes.io/projected/0748219c-2587-496d-a3c1-b218b6748b7c-kube-api-access-h4rlg\") on node \"ip-10-0-134-236.ec2.internal\" DevicePath \"\"" Jul 23 11:43:13.400996 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:13.400962 2561 generic.go:356] "Generic (PLEG): container finished" podID="2a8af5b7-35f7-4208-bf9f-f58d30954ae0" containerID="f69807406d1c36c7d8f515f5a484dfe29b31313194abf3cc18046013d64211f3" exitCode=0 Jul 23 11:43:13.401421 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:13.401068 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zhkzz" event={"ID":"2a8af5b7-35f7-4208-bf9f-f58d30954ae0","Type":"ContainerDied","Data":"f69807406d1c36c7d8f515f5a484dfe29b31313194abf3cc18046013d64211f3"} Jul 23 11:43:13.405945 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:13.405897 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73b2dck" event={"ID":"0748219c-2587-496d-a3c1-b218b6748b7c","Type":"ContainerDied","Data":"f9703a2d7131bdcd69cfc34418b065d8780a169fd8ffe5d646fadffd408c4ace"} Jul 23 11:43:13.406067 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:13.405960 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73b2dck" Jul 23 11:43:13.406067 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:13.405963 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9703a2d7131bdcd69cfc34418b065d8780a169fd8ffe5d646fadffd408c4ace" Jul 23 11:43:13.534878 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:13.534856 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nj7wg" Jul 23 11:43:13.538343 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:13.538323 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ffef83f1-4832-40f6-a8ca-8e62bc8db03b-bundle\") pod \"ffef83f1-4832-40f6-a8ca-8e62bc8db03b\" (UID: \"ffef83f1-4832-40f6-a8ca-8e62bc8db03b\") " Jul 23 11:43:13.538455 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:13.538367 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ffef83f1-4832-40f6-a8ca-8e62bc8db03b-util\") pod \"ffef83f1-4832-40f6-a8ca-8e62bc8db03b\" (UID: \"ffef83f1-4832-40f6-a8ca-8e62bc8db03b\") " Jul 23 11:43:13.538455 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:13.538395 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lm7x7\" (UniqueName: \"kubernetes.io/projected/ffef83f1-4832-40f6-a8ca-8e62bc8db03b-kube-api-access-lm7x7\") pod \"ffef83f1-4832-40f6-a8ca-8e62bc8db03b\" (UID: \"ffef83f1-4832-40f6-a8ca-8e62bc8db03b\") " Jul 23 11:43:13.538775 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:13.538744 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffef83f1-4832-40f6-a8ca-8e62bc8db03b-bundle" (OuterVolumeSpecName: "bundle") pod "ffef83f1-4832-40f6-a8ca-8e62bc8db03b" (UID: "ffef83f1-4832-40f6-a8ca-8e62bc8db03b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 11:43:13.540624 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:13.540600 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffef83f1-4832-40f6-a8ca-8e62bc8db03b-kube-api-access-lm7x7" (OuterVolumeSpecName: "kube-api-access-lm7x7") pod "ffef83f1-4832-40f6-a8ca-8e62bc8db03b" (UID: "ffef83f1-4832-40f6-a8ca-8e62bc8db03b"). InnerVolumeSpecName "kube-api-access-lm7x7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 11:43:13.543717 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:13.543694 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffef83f1-4832-40f6-a8ca-8e62bc8db03b-util" (OuterVolumeSpecName: "util") pod "ffef83f1-4832-40f6-a8ca-8e62bc8db03b" (UID: "ffef83f1-4832-40f6-a8ca-8e62bc8db03b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 11:43:13.562015 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:13.561992 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xz224" Jul 23 11:43:13.639162 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:13.639132 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dkdtd\" (UniqueName: \"kubernetes.io/projected/b4233887-c309-4a1f-9759-363bc6092fd9-kube-api-access-dkdtd\") pod \"b4233887-c309-4a1f-9759-363bc6092fd9\" (UID: \"b4233887-c309-4a1f-9759-363bc6092fd9\") " Jul 23 11:43:13.639328 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:13.639186 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b4233887-c309-4a1f-9759-363bc6092fd9-bundle\") pod \"b4233887-c309-4a1f-9759-363bc6092fd9\" (UID: \"b4233887-c309-4a1f-9759-363bc6092fd9\") " Jul 23 11:43:13.639328 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:13.639229 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b4233887-c309-4a1f-9759-363bc6092fd9-util\") pod \"b4233887-c309-4a1f-9759-363bc6092fd9\" (UID: \"b4233887-c309-4a1f-9759-363bc6092fd9\") " Jul 23 11:43:13.639503 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:13.639480 2561 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ffef83f1-4832-40f6-a8ca-8e62bc8db03b-bundle\") on node \"ip-10-0-134-236.ec2.internal\" DevicePath \"\"" Jul 23 11:43:13.639503 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:13.639502 2561 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ffef83f1-4832-40f6-a8ca-8e62bc8db03b-util\") on node \"ip-10-0-134-236.ec2.internal\" DevicePath \"\"" Jul 23 11:43:13.639649 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:13.639516 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lm7x7\" (UniqueName: \"kubernetes.io/projected/ffef83f1-4832-40f6-a8ca-8e62bc8db03b-kube-api-access-lm7x7\") on node \"ip-10-0-134-236.ec2.internal\" DevicePath \"\"" Jul 23 11:43:13.639820 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:13.639788 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4233887-c309-4a1f-9759-363bc6092fd9-bundle" (OuterVolumeSpecName: "bundle") pod "b4233887-c309-4a1f-9759-363bc6092fd9" (UID: "b4233887-c309-4a1f-9759-363bc6092fd9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 11:43:13.641340 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:13.641317 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4233887-c309-4a1f-9759-363bc6092fd9-kube-api-access-dkdtd" (OuterVolumeSpecName: "kube-api-access-dkdtd") pod "b4233887-c309-4a1f-9759-363bc6092fd9" (UID: "b4233887-c309-4a1f-9759-363bc6092fd9"). InnerVolumeSpecName "kube-api-access-dkdtd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 11:43:13.644481 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:13.644463 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4233887-c309-4a1f-9759-363bc6092fd9-util" (OuterVolumeSpecName: "util") pod "b4233887-c309-4a1f-9759-363bc6092fd9" (UID: "b4233887-c309-4a1f-9759-363bc6092fd9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 11:43:13.740256 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:13.740177 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkdtd\" (UniqueName: \"kubernetes.io/projected/b4233887-c309-4a1f-9759-363bc6092fd9-kube-api-access-dkdtd\") on node \"ip-10-0-134-236.ec2.internal\" DevicePath \"\"" Jul 23 11:43:13.740256 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:13.740203 2561 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b4233887-c309-4a1f-9759-363bc6092fd9-bundle\") on node \"ip-10-0-134-236.ec2.internal\" DevicePath \"\"" Jul 23 11:43:13.740256 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:13.740212 2561 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b4233887-c309-4a1f-9759-363bc6092fd9-util\") on node \"ip-10-0-134-236.ec2.internal\" DevicePath \"\"" Jul 23 11:43:14.413298 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:14.413267 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xz224" Jul 23 11:43:14.413756 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:14.413259 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xz224" event={"ID":"b4233887-c309-4a1f-9759-363bc6092fd9","Type":"ContainerDied","Data":"9840326f56968ff89bfc83737036b248df65160af6b31cd08dc0740fad4a097e"} Jul 23 11:43:14.413756 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:14.413393 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9840326f56968ff89bfc83737036b248df65160af6b31cd08dc0740fad4a097e" Jul 23 11:43:14.415185 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:14.415169 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nj7wg" Jul 23 11:43:14.415297 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:14.415202 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nj7wg" event={"ID":"ffef83f1-4832-40f6-a8ca-8e62bc8db03b","Type":"ContainerDied","Data":"098ea4e0fd9c8dea5b76372ee665777e604bde01bd0880f8db4dfdacc7236c02"} Jul 23 11:43:14.415297 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:14.415238 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="098ea4e0fd9c8dea5b76372ee665777e604bde01bd0880f8db4dfdacc7236c02" Jul 23 11:43:14.544665 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:14.544638 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zhkzz" Jul 23 11:43:14.544891 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:14.544877 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2a8af5b7-35f7-4208-bf9f-f58d30954ae0-util\") pod \"2a8af5b7-35f7-4208-bf9f-f58d30954ae0\" (UID: \"2a8af5b7-35f7-4208-bf9f-f58d30954ae0\") " Jul 23 11:43:14.544974 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:14.544961 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-szdpr\" (UniqueName: \"kubernetes.io/projected/2a8af5b7-35f7-4208-bf9f-f58d30954ae0-kube-api-access-szdpr\") pod \"2a8af5b7-35f7-4208-bf9f-f58d30954ae0\" (UID: \"2a8af5b7-35f7-4208-bf9f-f58d30954ae0\") " Jul 23 11:43:14.545013 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:14.544992 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2a8af5b7-35f7-4208-bf9f-f58d30954ae0-bundle\") pod \"2a8af5b7-35f7-4208-bf9f-f58d30954ae0\" (UID: \"2a8af5b7-35f7-4208-bf9f-f58d30954ae0\") " Jul 23 11:43:14.545495 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:14.545470 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a8af5b7-35f7-4208-bf9f-f58d30954ae0-bundle" (OuterVolumeSpecName: "bundle") pod "2a8af5b7-35f7-4208-bf9f-f58d30954ae0" (UID: "2a8af5b7-35f7-4208-bf9f-f58d30954ae0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 11:43:14.547079 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:14.547058 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a8af5b7-35f7-4208-bf9f-f58d30954ae0-kube-api-access-szdpr" (OuterVolumeSpecName: "kube-api-access-szdpr") pod "2a8af5b7-35f7-4208-bf9f-f58d30954ae0" (UID: "2a8af5b7-35f7-4208-bf9f-f58d30954ae0"). InnerVolumeSpecName "kube-api-access-szdpr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 11:43:14.550244 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:14.550210 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a8af5b7-35f7-4208-bf9f-f58d30954ae0-util" (OuterVolumeSpecName: "util") pod "2a8af5b7-35f7-4208-bf9f-f58d30954ae0" (UID: "2a8af5b7-35f7-4208-bf9f-f58d30954ae0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 11:43:14.646366 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:14.646338 2561 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2a8af5b7-35f7-4208-bf9f-f58d30954ae0-util\") on node \"ip-10-0-134-236.ec2.internal\" DevicePath \"\"" Jul 23 11:43:14.646366 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:14.646364 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szdpr\" (UniqueName: \"kubernetes.io/projected/2a8af5b7-35f7-4208-bf9f-f58d30954ae0-kube-api-access-szdpr\") on node \"ip-10-0-134-236.ec2.internal\" DevicePath \"\"" Jul 23 11:43:14.646366 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:14.646374 2561 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2a8af5b7-35f7-4208-bf9f-f58d30954ae0-bundle\") on node \"ip-10-0-134-236.ec2.internal\" DevicePath \"\"" Jul 23 11:43:15.421835 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:15.421802 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zhkzz" Jul 23 11:43:15.422232 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:15.421833 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zhkzz" event={"ID":"2a8af5b7-35f7-4208-bf9f-f58d30954ae0","Type":"ContainerDied","Data":"341bd9ba2c3b5a94e5f48b2e2409f511c4b5c55c7eb0a45765572fa836b87240"} Jul 23 11:43:15.422232 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:15.421929 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="341bd9ba2c3b5a94e5f48b2e2409f511c4b5c55c7eb0a45765572fa836b87240" Jul 23 11:43:53.151583 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:53.151543 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-5fb4m"] Jul 23 11:43:53.152222 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:53.152003 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a8af5b7-35f7-4208-bf9f-f58d30954ae0" containerName="util" Jul 23 11:43:53.152222 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:53.152022 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a8af5b7-35f7-4208-bf9f-f58d30954ae0" containerName="util" Jul 23 11:43:53.152222 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:53.152036 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffef83f1-4832-40f6-a8ca-8e62bc8db03b" containerName="extract" Jul 23 11:43:53.152222 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:53.152044 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffef83f1-4832-40f6-a8ca-8e62bc8db03b" containerName="extract" Jul 23 11:43:53.152222 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:53.152058 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4233887-c309-4a1f-9759-363bc6092fd9" containerName="util" Jul 23 11:43:53.152222 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:53.152066 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4233887-c309-4a1f-9759-363bc6092fd9" containerName="util" Jul 23 11:43:53.152222 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:53.152074 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a8af5b7-35f7-4208-bf9f-f58d30954ae0" containerName="pull" Jul 23 11:43:53.152222 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:53.152082 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a8af5b7-35f7-4208-bf9f-f58d30954ae0" containerName="pull" Jul 23 11:43:53.152222 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:53.152093 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffef83f1-4832-40f6-a8ca-8e62bc8db03b" containerName="util" Jul 23 11:43:53.152222 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:53.152101 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffef83f1-4832-40f6-a8ca-8e62bc8db03b" containerName="util" Jul 23 11:43:53.152222 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:53.152114 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffef83f1-4832-40f6-a8ca-8e62bc8db03b" containerName="pull" Jul 23 11:43:53.152222 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:53.152122 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffef83f1-4832-40f6-a8ca-8e62bc8db03b" containerName="pull" Jul 23 11:43:53.152222 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:53.152133 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4233887-c309-4a1f-9759-363bc6092fd9" containerName="pull" Jul 23 11:43:53.152222 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:53.152140 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4233887-c309-4a1f-9759-363bc6092fd9" containerName="pull" Jul 23 11:43:53.152222 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:53.152153 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0748219c-2587-496d-a3c1-b218b6748b7c" containerName="util" Jul 23 11:43:53.152222 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:53.152162 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0748219c-2587-496d-a3c1-b218b6748b7c" containerName="util" Jul 23 11:43:53.152222 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:53.152174 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0748219c-2587-496d-a3c1-b218b6748b7c" containerName="extract" Jul 23 11:43:53.152222 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:53.152182 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0748219c-2587-496d-a3c1-b218b6748b7c" containerName="extract" Jul 23 11:43:53.152222 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:53.152193 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4233887-c309-4a1f-9759-363bc6092fd9" containerName="extract" Jul 23 11:43:53.152222 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:53.152200 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4233887-c309-4a1f-9759-363bc6092fd9" containerName="extract" Jul 23 11:43:53.152222 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:53.152209 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a8af5b7-35f7-4208-bf9f-f58d30954ae0" containerName="extract" Jul 23 11:43:53.152222 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:53.152216 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a8af5b7-35f7-4208-bf9f-f58d30954ae0" containerName="extract" Jul 23 11:43:53.152222 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:53.152227 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0748219c-2587-496d-a3c1-b218b6748b7c" containerName="pull" Jul 23 11:43:53.152222 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:53.152235 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0748219c-2587-496d-a3c1-b218b6748b7c" containerName="pull" Jul 23 11:43:53.153291 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:53.152321 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="0748219c-2587-496d-a3c1-b218b6748b7c" containerName="extract" Jul 23 11:43:53.153291 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:53.152333 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffef83f1-4832-40f6-a8ca-8e62bc8db03b" containerName="extract" Jul 23 11:43:53.153291 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:53.152342 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a8af5b7-35f7-4208-bf9f-f58d30954ae0" containerName="extract" Jul 23 11:43:53.153291 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:53.152352 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4233887-c309-4a1f-9759-363bc6092fd9" containerName="extract" Jul 23 11:43:53.156940 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:53.156901 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-5fb4m" Jul 23 11:43:53.159935 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:53.159900 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-qddrk\"" Jul 23 11:43:53.160783 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:53.160766 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 23 11:43:53.160870 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:53.160771 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 23 11:43:53.178176 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:53.178154 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-5fb4m"] Jul 23 11:43:53.245121 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:53.245094 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/20d76532-0f2d-407c-bc43-7bc72854a2aa-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-5fb4m\" (UID: \"20d76532-0f2d-407c-bc43-7bc72854a2aa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-5fb4m" Jul 23 11:43:53.245240 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:53.245136 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wdzj\" (UniqueName: \"kubernetes.io/projected/20d76532-0f2d-407c-bc43-7bc72854a2aa-kube-api-access-7wdzj\") pod \"kuadrant-operator-controller-manager-55467d9f7-5fb4m\" (UID: \"20d76532-0f2d-407c-bc43-7bc72854a2aa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-5fb4m" Jul 23 11:43:53.346581 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:53.346550 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/20d76532-0f2d-407c-bc43-7bc72854a2aa-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-5fb4m\" (UID: \"20d76532-0f2d-407c-bc43-7bc72854a2aa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-5fb4m" Jul 23 11:43:53.346710 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:53.346589 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wdzj\" (UniqueName: \"kubernetes.io/projected/20d76532-0f2d-407c-bc43-7bc72854a2aa-kube-api-access-7wdzj\") pod \"kuadrant-operator-controller-manager-55467d9f7-5fb4m\" (UID: \"20d76532-0f2d-407c-bc43-7bc72854a2aa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-5fb4m" Jul 23 11:43:53.346893 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:53.346877 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/20d76532-0f2d-407c-bc43-7bc72854a2aa-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-5fb4m\" (UID: \"20d76532-0f2d-407c-bc43-7bc72854a2aa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-5fb4m" Jul 23 11:43:53.361271 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:53.361253 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wdzj\" (UniqueName: \"kubernetes.io/projected/20d76532-0f2d-407c-bc43-7bc72854a2aa-kube-api-access-7wdzj\") pod \"kuadrant-operator-controller-manager-55467d9f7-5fb4m\" (UID: \"20d76532-0f2d-407c-bc43-7bc72854a2aa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-5fb4m" Jul 23 11:43:53.466630 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:53.466561 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-5fb4m" Jul 23 11:43:53.608902 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:53.608873 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-5fb4m"] Jul 23 11:43:53.616055 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:53.616033 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-5fb4m"] Jul 23 11:43:53.749702 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:43:53.749659 2561 log.go:32] "RunPodSandbox from runtime service failed" err=< Jul 23 11:43:53.749702 ip-10-0-134-236 kubenswrapper[2561]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-55467d9f7-5fb4m_kuadrant-system_20d76532-0f2d-407c-bc43-7bc72854a2aa_0(3081aea5d93e9356e2c70b98e4331b72bce74210a4b0184d8aa065a897afc6c4): error adding pod kuadrant-system_kuadrant-operator-controller-manager-55467d9f7-5fb4m to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"3081aea5d93e9356e2c70b98e4331b72bce74210a4b0184d8aa065a897afc6c4" Netns:"/var/run/netns/18ae79dd-43af-4fc9-af0d-636c8cf8e2e4" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=kuadrant-system;K8S_POD_NAME=kuadrant-operator-controller-manager-55467d9f7-5fb4m;K8S_POD_INFRA_CONTAINER_ID=3081aea5d93e9356e2c70b98e4331b72bce74210a4b0184d8aa065a897afc6c4;K8S_POD_UID=20d76532-0f2d-407c-bc43-7bc72854a2aa" Path:"" ERRORED: error configuring pod [kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-5fb4m] networking: Multus: [kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-5fb4m/20d76532-0f2d-407c-bc43-7bc72854a2aa]: error setting the networks status: SetPodNetworkStatusAnnotation: failed to update the pod kuadrant-operator-controller-manager-55467d9f7-5fb4m in out of cluster comm: SetNetworkStatus: failed to update the pod kuadrant-operator-controller-manager-55467d9f7-5fb4m in out of cluster comm: status update failed for pod /: pods "kuadrant-operator-controller-manager-55467d9f7-5fb4m" not found Jul 23 11:43:53.749702 ip-10-0-134-236 kubenswrapper[2561]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Jul 23 11:43:53.749702 ip-10-0-134-236 kubenswrapper[2561]: > Jul 23 11:43:53.749956 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:43:53.749739 2561 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=< Jul 23 11:43:53.749956 ip-10-0-134-236 kubenswrapper[2561]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-55467d9f7-5fb4m_kuadrant-system_20d76532-0f2d-407c-bc43-7bc72854a2aa_0(3081aea5d93e9356e2c70b98e4331b72bce74210a4b0184d8aa065a897afc6c4): error adding pod kuadrant-system_kuadrant-operator-controller-manager-55467d9f7-5fb4m to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"3081aea5d93e9356e2c70b98e4331b72bce74210a4b0184d8aa065a897afc6c4" Netns:"/var/run/netns/18ae79dd-43af-4fc9-af0d-636c8cf8e2e4" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=kuadrant-system;K8S_POD_NAME=kuadrant-operator-controller-manager-55467d9f7-5fb4m;K8S_POD_INFRA_CONTAINER_ID=3081aea5d93e9356e2c70b98e4331b72bce74210a4b0184d8aa065a897afc6c4;K8S_POD_UID=20d76532-0f2d-407c-bc43-7bc72854a2aa" Path:"" ERRORED: error configuring pod [kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-5fb4m] networking: Multus: [kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-5fb4m/20d76532-0f2d-407c-bc43-7bc72854a2aa]: error setting the networks status: SetPodNetworkStatusAnnotation: failed to update the pod kuadrant-operator-controller-manager-55467d9f7-5fb4m in out of cluster comm: SetNetworkStatus: failed to update the pod kuadrant-operator-controller-manager-55467d9f7-5fb4m in out of cluster comm: status update failed for pod /: pods "kuadrant-operator-controller-manager-55467d9f7-5fb4m" not found Jul 23 11:43:53.749956 ip-10-0-134-236 kubenswrapper[2561]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Jul 23 11:43:53.749956 ip-10-0-134-236 kubenswrapper[2561]: > pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-5fb4m" Jul 23 11:43:54.597853 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:54.597827 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-5fb4m" Jul 23 11:43:54.600771 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:43:54.600738 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-5fb4m\" is forbidden: User \"system:node:ip-10-0-134-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-236.ec2.internal' and this object" podUID="20d76532-0f2d-407c-bc43-7bc72854a2aa" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-5fb4m" Jul 23 11:43:54.605575 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:54.605555 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-5fb4m" Jul 23 11:43:54.608002 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:43:54.607981 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-5fb4m\" is forbidden: User \"system:node:ip-10-0-134-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-236.ec2.internal' and this object" podUID="20d76532-0f2d-407c-bc43-7bc72854a2aa" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-5fb4m" Jul 23 11:43:54.759719 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:54.759684 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/20d76532-0f2d-407c-bc43-7bc72854a2aa-extensions-socket-volume\") pod \"20d76532-0f2d-407c-bc43-7bc72854a2aa\" (UID: \"20d76532-0f2d-407c-bc43-7bc72854a2aa\") " Jul 23 11:43:54.759719 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:54.759724 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7wdzj\" (UniqueName: \"kubernetes.io/projected/20d76532-0f2d-407c-bc43-7bc72854a2aa-kube-api-access-7wdzj\") pod \"20d76532-0f2d-407c-bc43-7bc72854a2aa\" (UID: \"20d76532-0f2d-407c-bc43-7bc72854a2aa\") " Jul 23 11:43:54.760013 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:54.759979 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20d76532-0f2d-407c-bc43-7bc72854a2aa-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "20d76532-0f2d-407c-bc43-7bc72854a2aa" (UID: "20d76532-0f2d-407c-bc43-7bc72854a2aa"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 11:43:54.761953 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:54.761932 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20d76532-0f2d-407c-bc43-7bc72854a2aa-kube-api-access-7wdzj" (OuterVolumeSpecName: "kube-api-access-7wdzj") pod "20d76532-0f2d-407c-bc43-7bc72854a2aa" (UID: "20d76532-0f2d-407c-bc43-7bc72854a2aa"). InnerVolumeSpecName "kube-api-access-7wdzj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 11:43:54.860287 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:54.860212 2561 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/20d76532-0f2d-407c-bc43-7bc72854a2aa-extensions-socket-volume\") on node \"ip-10-0-134-236.ec2.internal\" DevicePath \"\"" Jul 23 11:43:54.860287 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:54.860235 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wdzj\" (UniqueName: \"kubernetes.io/projected/20d76532-0f2d-407c-bc43-7bc72854a2aa-kube-api-access-7wdzj\") on node \"ip-10-0-134-236.ec2.internal\" DevicePath \"\"" Jul 23 11:43:55.455281 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:55.455248 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20d76532-0f2d-407c-bc43-7bc72854a2aa" path="/var/lib/kubelet/pods/20d76532-0f2d-407c-bc43-7bc72854a2aa/volumes" Jul 23 11:43:55.603421 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:43:55.603393 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-5fb4m" Jul 23 11:43:55.609496 ip-10-0-134-236 kubenswrapper[2561]: E0723 11:43:55.609463 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-5fb4m\" is forbidden: User \"system:node:ip-10-0-134-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-236.ec2.internal' and this object" podUID="20d76532-0f2d-407c-bc43-7bc72854a2aa" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-5fb4m" Jul 23 11:44:42.679991 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:44:42.679881 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 11:44:42.683313 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:44:42.683290 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-t7jsv" Jul 23 11:44:42.686067 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:44:42.686042 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 23 11:44:42.686184 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:44:42.686042 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 23 11:44:42.686184 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:44:42.686045 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 23 11:44:42.686184 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:44:42.686045 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-gqsgb\"" Jul 23 11:44:42.691723 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:44:42.691704 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 11:44:42.722425 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:44:42.722390 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 11:44:42.756150 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:44:42.756119 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9sqh\" (UniqueName: \"kubernetes.io/projected/5fbe789f-5522-405d-a998-b7f4181d255a-kube-api-access-f9sqh\") pod \"limitador-limitador-54d6d988d5-t7jsv\" (UID: \"5fbe789f-5522-405d-a998-b7f4181d255a\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-t7jsv" Jul 23 11:44:42.756283 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:44:42.756156 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5fbe789f-5522-405d-a998-b7f4181d255a-config-file\") pod \"limitador-limitador-54d6d988d5-t7jsv\" (UID: \"5fbe789f-5522-405d-a998-b7f4181d255a\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-t7jsv" Jul 23 11:44:42.857138 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:44:42.857095 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9sqh\" (UniqueName: \"kubernetes.io/projected/5fbe789f-5522-405d-a998-b7f4181d255a-kube-api-access-f9sqh\") pod \"limitador-limitador-54d6d988d5-t7jsv\" (UID: \"5fbe789f-5522-405d-a998-b7f4181d255a\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-t7jsv" Jul 23 11:44:42.857138 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:44:42.857137 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5fbe789f-5522-405d-a998-b7f4181d255a-config-file\") pod \"limitador-limitador-54d6d988d5-t7jsv\" (UID: \"5fbe789f-5522-405d-a998-b7f4181d255a\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-t7jsv" Jul 23 11:44:42.857753 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:44:42.857736 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5fbe789f-5522-405d-a998-b7f4181d255a-config-file\") pod \"limitador-limitador-54d6d988d5-t7jsv\" (UID: \"5fbe789f-5522-405d-a998-b7f4181d255a\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-t7jsv" Jul 23 11:44:42.865633 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:44:42.865601 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9sqh\" (UniqueName: \"kubernetes.io/projected/5fbe789f-5522-405d-a998-b7f4181d255a-kube-api-access-f9sqh\") pod \"limitador-limitador-54d6d988d5-t7jsv\" (UID: \"5fbe789f-5522-405d-a998-b7f4181d255a\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-t7jsv" Jul 23 11:44:42.994346 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:44:42.994249 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-t7jsv" Jul 23 11:44:43.135829 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:44:43.135806 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 11:44:43.138396 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:44:43.138352 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fbe789f_5522_405d_a998_b7f4181d255a.slice/crio-9e3ecb1c6881dde0b3d953a80d910ac5ded9e12d371a0ce8044cbf9ed321aac7 WatchSource:0}: Error finding container 9e3ecb1c6881dde0b3d953a80d910ac5ded9e12d371a0ce8044cbf9ed321aac7: Status 404 returned error can't find the container with id 9e3ecb1c6881dde0b3d953a80d910ac5ded9e12d371a0ce8044cbf9ed321aac7 Jul 23 11:44:43.832493 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:44:43.832460 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-t7jsv" event={"ID":"5fbe789f-5522-405d-a998-b7f4181d255a","Type":"ContainerStarted","Data":"9e3ecb1c6881dde0b3d953a80d910ac5ded9e12d371a0ce8044cbf9ed321aac7"} Jul 23 11:44:46.850777 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:44:46.850742 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-t7jsv" event={"ID":"5fbe789f-5522-405d-a998-b7f4181d255a","Type":"ContainerStarted","Data":"4ec8de1a22c917033b37e6c44d1a2accaba19af12cca80b74e65746ba9178299"} Jul 23 11:44:46.851194 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:44:46.850884 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-t7jsv" Jul 23 11:44:46.869736 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:44:46.869689 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-t7jsv" podStartSLOduration=2.056843657 podStartE2EDuration="4.869673908s" totalImagesPullingTime="2.812830251s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:44:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:44:46.868248959 +0000 UTC m=+553.939939002" watchObservedRunningTime="2026-07-23 11:44:46.869673908 +0000 UTC m=+553.941363897" Jul 23 11:44:57.857153 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:44:57.857123 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-t7jsv" Jul 23 11:45:33.349543 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:45:33.349515 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-7slpj_65247e1f-fc10-4b40-af67-5e646f352263/console-operator/1.log" Jul 23 11:45:33.350639 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:45:33.350618 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-7slpj_65247e1f-fc10-4b40-af67-5e646f352263/console-operator/1.log" Jul 23 11:45:57.534367 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:45:57.534284 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 11:46:17.584855 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:46:17.584828 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 11:46:33.555270 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:46:33.555239 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 11:46:40.015545 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:46:40.015514 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 11:46:47.846985 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:46:47.846953 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr9r68"] Jul 23 11:46:47.849818 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:46:47.849801 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr9r68" Jul 23 11:46:47.853231 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:46:47.853210 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-btpzj\"" Jul 23 11:46:47.853231 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:46:47.853221 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 23 11:46:47.853383 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:46:47.853258 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 23 11:46:47.853383 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:46:47.853228 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 23 11:46:47.860334 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:46:47.860314 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr9r68"] Jul 23 11:46:47.911047 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:46:47.911019 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89g2x\" (UniqueName: \"kubernetes.io/projected/6aecbb33-e47a-41f3-b284-9f323c93ff41-kube-api-access-89g2x\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr9r68\" (UID: \"6aecbb33-e47a-41f3-b284-9f323c93ff41\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr9r68" Jul 23 11:46:47.911142 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:46:47.911061 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6aecbb33-e47a-41f3-b284-9f323c93ff41-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr9r68\" (UID: \"6aecbb33-e47a-41f3-b284-9f323c93ff41\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr9r68" Jul 23 11:46:47.911142 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:46:47.911081 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6aecbb33-e47a-41f3-b284-9f323c93ff41-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr9r68\" (UID: \"6aecbb33-e47a-41f3-b284-9f323c93ff41\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr9r68" Jul 23 11:46:47.911142 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:46:47.911131 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6aecbb33-e47a-41f3-b284-9f323c93ff41-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr9r68\" (UID: \"6aecbb33-e47a-41f3-b284-9f323c93ff41\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr9r68" Jul 23 11:46:47.911239 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:46:47.911159 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6aecbb33-e47a-41f3-b284-9f323c93ff41-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr9r68\" (UID: \"6aecbb33-e47a-41f3-b284-9f323c93ff41\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr9r68" Jul 23 11:46:47.911239 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:46:47.911177 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6aecbb33-e47a-41f3-b284-9f323c93ff41-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr9r68\" (UID: \"6aecbb33-e47a-41f3-b284-9f323c93ff41\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr9r68" Jul 23 11:46:48.012421 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:46:48.012394 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89g2x\" (UniqueName: \"kubernetes.io/projected/6aecbb33-e47a-41f3-b284-9f323c93ff41-kube-api-access-89g2x\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr9r68\" (UID: \"6aecbb33-e47a-41f3-b284-9f323c93ff41\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr9r68" Jul 23 11:46:48.012526 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:46:48.012444 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6aecbb33-e47a-41f3-b284-9f323c93ff41-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr9r68\" (UID: \"6aecbb33-e47a-41f3-b284-9f323c93ff41\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr9r68" Jul 23 11:46:48.012526 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:46:48.012478 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6aecbb33-e47a-41f3-b284-9f323c93ff41-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr9r68\" (UID: \"6aecbb33-e47a-41f3-b284-9f323c93ff41\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr9r68" Jul 23 11:46:48.012526 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:46:48.012515 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6aecbb33-e47a-41f3-b284-9f323c93ff41-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr9r68\" (UID: \"6aecbb33-e47a-41f3-b284-9f323c93ff41\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr9r68" Jul 23 11:46:48.012681 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:46:48.012559 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6aecbb33-e47a-41f3-b284-9f323c93ff41-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr9r68\" (UID: \"6aecbb33-e47a-41f3-b284-9f323c93ff41\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr9r68" Jul 23 11:46:48.012681 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:46:48.012589 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6aecbb33-e47a-41f3-b284-9f323c93ff41-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr9r68\" (UID: \"6aecbb33-e47a-41f3-b284-9f323c93ff41\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr9r68" Jul 23 11:46:48.013021 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:46:48.012995 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6aecbb33-e47a-41f3-b284-9f323c93ff41-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr9r68\" (UID: \"6aecbb33-e47a-41f3-b284-9f323c93ff41\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr9r68" Jul 23 11:46:48.013119 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:46:48.013049 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6aecbb33-e47a-41f3-b284-9f323c93ff41-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr9r68\" (UID: \"6aecbb33-e47a-41f3-b284-9f323c93ff41\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr9r68" Jul 23 11:46:48.013185 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:46:48.013162 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6aecbb33-e47a-41f3-b284-9f323c93ff41-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr9r68\" (UID: \"6aecbb33-e47a-41f3-b284-9f323c93ff41\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr9r68" Jul 23 11:46:48.015564 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:46:48.015533 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6aecbb33-e47a-41f3-b284-9f323c93ff41-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr9r68\" (UID: \"6aecbb33-e47a-41f3-b284-9f323c93ff41\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr9r68" Jul 23 11:46:48.015665 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:46:48.015570 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6aecbb33-e47a-41f3-b284-9f323c93ff41-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr9r68\" (UID: \"6aecbb33-e47a-41f3-b284-9f323c93ff41\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr9r68" Jul 23 11:46:48.020457 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:46:48.020436 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89g2x\" (UniqueName: \"kubernetes.io/projected/6aecbb33-e47a-41f3-b284-9f323c93ff41-kube-api-access-89g2x\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr9r68\" (UID: \"6aecbb33-e47a-41f3-b284-9f323c93ff41\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr9r68" Jul 23 11:46:48.159735 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:46:48.159665 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr9r68" Jul 23 11:46:48.291950 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:46:48.291777 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr9r68"] Jul 23 11:46:48.294573 ip-10-0-134-236 kubenswrapper[2561]: W0723 11:46:48.294551 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aecbb33_e47a_41f3_b284_9f323c93ff41.slice/crio-320c21472e5d19d46a55b85dae92df32870ca296d73cd0cb9917f3cad954d83f WatchSource:0}: Error finding container 320c21472e5d19d46a55b85dae92df32870ca296d73cd0cb9917f3cad954d83f: Status 404 returned error can't find the container with id 320c21472e5d19d46a55b85dae92df32870ca296d73cd0cb9917f3cad954d83f Jul 23 11:46:48.295998 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:46:48.295976 2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 11:46:48.430786 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:46:48.430702 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr9r68" event={"ID":"6aecbb33-e47a-41f3-b284-9f323c93ff41","Type":"ContainerStarted","Data":"320c21472e5d19d46a55b85dae92df32870ca296d73cd0cb9917f3cad954d83f"} Jul 23 11:46:49.340489 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:46:49.340456 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 11:46:50.442285 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:46:50.442250 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr9r68" event={"ID":"6aecbb33-e47a-41f3-b284-9f323c93ff41","Type":"ContainerStarted","Data":"59ee0ef7ff2f6875462fecd7d69d4380a7f221fc458d351e78f25ca5e9b5d9f1"} Jul 23 11:46:50.463326 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:46:50.463282 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr9r68" podStartSLOduration=2.343730139 podStartE2EDuration="3.463267667s" totalImagesPullingTime="1.119537528s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:46:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:46:50.461131233 +0000 UTC m=+677.532821234" watchObservedRunningTime="2026-07-23 11:46:50.463267667 +0000 UTC m=+677.534957656" Jul 23 11:46:58.160750 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:46:58.160712 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr9r68" Jul 23 11:46:58.161247 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:46:58.160777 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr9r68" Jul 23 11:46:58.172592 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:46:58.172574 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr9r68" Jul 23 11:46:58.490890 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:46:58.490825 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr9r68" Jul 23 11:47:03.750455 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:47:03.750423 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 11:47:17.840075 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:47:17.840039 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 11:48:49.951497 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:48:49.951464 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 11:49:03.440644 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:49:03.440560 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 11:49:12.142013 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:49:12.141941 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 11:50:33.382108 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:50:33.382036 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-7slpj_65247e1f-fc10-4b40-af67-5e646f352263/console-operator/1.log" Jul 23 11:50:33.383871 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:50:33.383850 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-7slpj_65247e1f-fc10-4b40-af67-5e646f352263/console-operator/1.log" Jul 23 11:50:49.950449 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:50:49.950419 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 11:51:01.145061 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:51:01.145030 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 11:51:36.554622 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:51:36.554590 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 11:51:45.346100 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:51:45.346065 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 11:51:56.148573 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:51:56.148504 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 11:52:04.336179 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:52:04.336143 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 11:52:15.345854 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:52:15.345818 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 11:52:24.537583 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:52:24.537551 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 11:52:36.041697 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:52:36.041665 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 11:52:45.442179 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:52:45.442143 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 11:53:07.956131 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:53:07.956103 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 11:53:17.038051 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:53:17.038018 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 11:53:40.545454 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:53:40.545362 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 11:53:52.332514 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:53:52.332473 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 11:54:09.250264 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:54:09.250230 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 11:54:18.953892 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:54:18.953861 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 11:54:29.386483 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:54:29.386447 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 11:54:38.449924 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:54:38.449873 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 11:54:48.344574 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:54:48.344538 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 11:55:33.415113 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:55:33.415082 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-7slpj_65247e1f-fc10-4b40-af67-5e646f352263/console-operator/1.log" Jul 23 11:55:33.417944 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:55:33.417920 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-7slpj_65247e1f-fc10-4b40-af67-5e646f352263/console-operator/1.log" Jul 23 11:56:34.794453 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:56:34.794419 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 11:56:45.425638 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:56:45.425603 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 11:57:01.991802 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:57:01.991768 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 11:57:12.706625 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:57:12.706588 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 11:57:24.056161 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:57:24.056124 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 11:57:33.663967 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:57:33.663938 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 11:57:42.837446 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:57:42.837410 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 11:57:54.057930 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:57:54.057834 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 11:58:04.348884 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:58:04.348848 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 11:58:13.940355 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:58:13.940316 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 11:58:24.644876 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:58:24.644842 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 11:58:49.140795 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:58:49.140764 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 11:58:56.530430 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:58:56.530398 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 11:59:16.149125 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:59:16.149096 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 11:59:33.943976 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:59:33.943882 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 11:59:54.246744 ip-10-0-134-236 kubenswrapper[2561]: I0723 11:59:54.246713 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 12:00:00.136510 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:00:00.136474 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29746800-wtftr"] Jul 23 12:00:00.139961 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:00:00.139939 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746800-wtftr" Jul 23 12:00:00.142446 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:00:00.142429 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q8m5p\"" Jul 23 12:00:00.148347 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:00:00.148324 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29746800-wtftr"] Jul 23 12:00:00.150391 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:00:00.150373 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746800-wtftr" Jul 23 12:00:00.280496 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:00:00.280475 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29746800-wtftr"] Jul 23 12:00:00.282012 ip-10-0-134-236 kubenswrapper[2561]: W0723 12:00:00.281969 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35e2b33c_01af_47d6_81b1_2a68d9d244e0.slice/crio-53bff376c022abb97e0c8ecb9852eca21ec44d97405554cab235aefacf377e5b WatchSource:0}: Error finding container 53bff376c022abb97e0c8ecb9852eca21ec44d97405554cab235aefacf377e5b: Status 404 returned error can't find the container with id 53bff376c022abb97e0c8ecb9852eca21ec44d97405554cab235aefacf377e5b Jul 23 12:00:00.283762 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:00:00.283739 2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 12:00:01.087695 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:00:01.087658 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746800-wtftr" event={"ID":"35e2b33c-01af-47d6-81b1-2a68d9d244e0","Type":"ContainerStarted","Data":"53bff376c022abb97e0c8ecb9852eca21ec44d97405554cab235aefacf377e5b"} Jul 23 12:00:03.098890 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:00:03.098854 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746800-wtftr" event={"ID":"35e2b33c-01af-47d6-81b1-2a68d9d244e0","Type":"ContainerStarted","Data":"eaf0f8fb08bc5c7d03f81fca94247c253337b48d23bf353d4523bfe59a4dab2a"} Jul 23 12:00:03.122050 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:00:03.122005 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746800-wtftr" podStartSLOduration=1.178839402 podStartE2EDuration="3.121988699s" totalImagesPullingTime="1.943149297s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 12:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 12:00:03.121446143 +0000 UTC m=+1470.193136133" watchObservedRunningTime="2026-07-23 12:00:03.121988699 +0000 UTC m=+1470.193678688" Jul 23 12:00:04.106108 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:00:04.106074 2561 generic.go:356] "Generic (PLEG): container finished" podID="35e2b33c-01af-47d6-81b1-2a68d9d244e0" containerID="eaf0f8fb08bc5c7d03f81fca94247c253337b48d23bf353d4523bfe59a4dab2a" exitCode=0 Jul 23 12:00:04.106536 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:00:04.106126 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746800-wtftr" event={"ID":"35e2b33c-01af-47d6-81b1-2a68d9d244e0","Type":"ContainerDied","Data":"eaf0f8fb08bc5c7d03f81fca94247c253337b48d23bf353d4523bfe59a4dab2a"} Jul 23 12:00:05.235577 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:00:05.235555 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746800-wtftr" Jul 23 12:00:06.116918 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:00:06.116873 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746800-wtftr" event={"ID":"35e2b33c-01af-47d6-81b1-2a68d9d244e0","Type":"ContainerDied","Data":"53bff376c022abb97e0c8ecb9852eca21ec44d97405554cab235aefacf377e5b"} Jul 23 12:00:06.117084 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:00:06.116925 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746800-wtftr" Jul 23 12:00:06.117084 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:00:06.116937 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53bff376c022abb97e0c8ecb9852eca21ec44d97405554cab235aefacf377e5b" Jul 23 12:00:08.347508 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:00:08.347477 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 12:00:15.040278 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:00:15.037200 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 12:00:32.359988 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:00:32.359954 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 12:00:33.447499 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:00:33.447473 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-7slpj_65247e1f-fc10-4b40-af67-5e646f352263/console-operator/1.log" Jul 23 12:00:33.453361 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:00:33.453336 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-7slpj_65247e1f-fc10-4b40-af67-5e646f352263/console-operator/1.log" Jul 23 12:00:43.350253 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:00:43.350219 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 12:00:53.068513 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:00:53.068471 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 12:01:00.808857 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:01:00.808771 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 12:01:11.345417 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:01:11.345382 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 12:01:21.072193 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:01:21.072156 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 12:01:34.294292 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:01:34.294257 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 12:01:44.146634 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:01:44.146591 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 12:01:54.346209 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:01:54.346166 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 12:02:04.072158 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:02:04.072114 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 12:02:17.341341 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:02:17.341299 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 12:02:26.450034 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:02:26.449959 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 12:02:39.050514 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:02:39.050480 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 12:02:51.055673 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:02:51.055639 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 12:03:01.999241 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:03:01.999207 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 12:03:08.959992 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:03:08.959958 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 12:03:23.261296 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:03:23.261265 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 12:03:36.282684 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:03:36.282651 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 12:03:49.867029 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:03:49.866995 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 12:04:11.981557 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:04:11.981478 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9fab72d8-q5mlj"] Jul 23 12:04:11.982044 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:04:11.981793 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35e2b33c-01af-47d6-81b1-2a68d9d244e0" containerName="cleanup" Jul 23 12:04:11.982044 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:04:11.981804 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="35e2b33c-01af-47d6-81b1-2a68d9d244e0" containerName="cleanup" Jul 23 12:04:11.982044 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:04:11.981864 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="35e2b33c-01af-47d6-81b1-2a68d9d244e0" containerName="cleanup" Jul 23 12:04:11.984678 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:04:11.984661 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9fab72d8-q5mlj" Jul 23 12:04:11.987903 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:04:11.987878 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 23 12:04:11.988094 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:04:11.988060 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q8m5p\"" Jul 23 12:04:11.994688 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:04:11.994657 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9fab72d8-q5mlj"] Jul 23 12:04:12.064638 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:04:12.064599 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3bdca09-5396-4e9c-9263-6eb9fafb2916-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9fab72d8-q5mlj\" (UID: \"d3bdca09-5396-4e9c-9263-6eb9fafb2916\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9fab72d8-q5mlj" Jul 23 12:04:12.165988 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:04:12.165947 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3bdca09-5396-4e9c-9263-6eb9fafb2916-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9fab72d8-q5mlj\" (UID: \"d3bdca09-5396-4e9c-9263-6eb9fafb2916\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9fab72d8-q5mlj" Jul 23 12:04:12.166602 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:04:12.166582 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3bdca09-5396-4e9c-9263-6eb9fafb2916-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9fab72d8-q5mlj\" (UID: \"d3bdca09-5396-4e9c-9263-6eb9fafb2916\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9fab72d8-q5mlj" Jul 23 12:04:12.295352 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:04:12.295270 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9fab72d8-q5mlj" Jul 23 12:04:12.431489 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:04:12.431462 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9fab72d8-q5mlj"] Jul 23 12:04:12.433123 ip-10-0-134-236 kubenswrapper[2561]: W0723 12:04:12.433096 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3bdca09_5396_4e9c_9263_6eb9fafb2916.slice/crio-deed100cc54ace009a753c72b55733d57a6cb4b9b5763a0c67b764914d9fa77d WatchSource:0}: Error finding container deed100cc54ace009a753c72b55733d57a6cb4b9b5763a0c67b764914d9fa77d: Status 404 returned error can't find the container with id deed100cc54ace009a753c72b55733d57a6cb4b9b5763a0c67b764914d9fa77d Jul 23 12:04:13.326843 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:04:13.326807 2561 generic.go:356] "Generic (PLEG): container finished" podID="d3bdca09-5396-4e9c-9263-6eb9fafb2916" containerID="e295792c255500529e9335c61594bc490a7d6403d633b9daca64f3845082990a" exitCode=0 Jul 23 12:04:13.327275 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:04:13.326896 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9fab72d8-q5mlj" event={"ID":"d3bdca09-5396-4e9c-9263-6eb9fafb2916","Type":"ContainerDied","Data":"e295792c255500529e9335c61594bc490a7d6403d633b9daca64f3845082990a"} Jul 23 12:04:13.327275 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:04:13.326962 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9fab72d8-q5mlj" event={"ID":"d3bdca09-5396-4e9c-9263-6eb9fafb2916","Type":"ContainerStarted","Data":"deed100cc54ace009a753c72b55733d57a6cb4b9b5763a0c67b764914d9fa77d"} Jul 23 12:04:14.465876 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:04:14.465854 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9fab72d8-q5mlj" Jul 23 12:04:14.587232 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:04:14.587137 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3bdca09-5396-4e9c-9263-6eb9fafb2916-maas-api-service-ca\") pod \"d3bdca09-5396-4e9c-9263-6eb9fafb2916\" (UID: \"d3bdca09-5396-4e9c-9263-6eb9fafb2916\") " Jul 23 12:04:14.587519 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:04:14.587478 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3bdca09-5396-4e9c-9263-6eb9fafb2916-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d3bdca09-5396-4e9c-9263-6eb9fafb2916" (UID: "d3bdca09-5396-4e9c-9263-6eb9fafb2916"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 12:04:14.688388 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:04:14.688353 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3bdca09-5396-4e9c-9263-6eb9fafb2916-maas-api-service-ca\") on node \"ip-10-0-134-236.ec2.internal\" DevicePath \"\"" Jul 23 12:04:15.337820 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:04:15.337790 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9fab72d8-q5mlj" Jul 23 12:04:15.337820 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:04:15.337812 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9fab72d8-q5mlj" event={"ID":"d3bdca09-5396-4e9c-9263-6eb9fafb2916","Type":"ContainerDied","Data":"deed100cc54ace009a753c72b55733d57a6cb4b9b5763a0c67b764914d9fa77d"} Jul 23 12:04:15.338054 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:04:15.337858 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="deed100cc54ace009a753c72b55733d57a6cb4b9b5763a0c67b764914d9fa77d" Jul 23 12:04:25.654873 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:04:25.654836 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 12:04:27.079178 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:04:27.079147 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9fab72d8-q5mlj"] Jul 23 12:04:27.082832 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:04:27.082810 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9fab72d8-q5mlj"] Jul 23 12:04:27.460050 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:04:27.459969 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3bdca09-5396-4e9c-9263-6eb9fafb2916" path="/var/lib/kubelet/pods/d3bdca09-5396-4e9c-9263-6eb9fafb2916/volumes" Jul 23 12:04:38.052939 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:04:38.052889 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 12:04:55.739532 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:04:55.739500 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9e27edaf-mfbnh"] Jul 23 12:04:55.739949 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:04:55.739835 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3bdca09-5396-4e9c-9263-6eb9fafb2916" containerName="revoke-keys" Jul 23 12:04:55.739949 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:04:55.739846 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3bdca09-5396-4e9c-9263-6eb9fafb2916" containerName="revoke-keys" Jul 23 12:04:55.739949 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:04:55.739897 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3bdca09-5396-4e9c-9263-6eb9fafb2916" containerName="revoke-keys" Jul 23 12:04:55.742547 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:04:55.742530 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9e27edaf-mfbnh" Jul 23 12:04:55.745491 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:04:55.745466 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 23 12:04:55.745666 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:04:55.745653 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q8m5p\"" Jul 23 12:04:55.752478 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:04:55.752453 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9e27edaf-mfbnh"] Jul 23 12:04:55.842571 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:04:55.842537 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9f58d5e-3933-4d15-861e-42546183ff66-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-9e27edaf-mfbnh\" (UID: \"b9f58d5e-3933-4d15-861e-42546183ff66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9e27edaf-mfbnh" Jul 23 12:04:55.943258 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:04:55.943224 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9f58d5e-3933-4d15-861e-42546183ff66-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-9e27edaf-mfbnh\" (UID: \"b9f58d5e-3933-4d15-861e-42546183ff66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9e27edaf-mfbnh" Jul 23 12:04:55.943873 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:04:55.943855 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9f58d5e-3933-4d15-861e-42546183ff66-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-9e27edaf-mfbnh\" (UID: \"b9f58d5e-3933-4d15-861e-42546183ff66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9e27edaf-mfbnh" Jul 23 12:04:56.055509 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:04:56.055430 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9e27edaf-mfbnh" Jul 23 12:04:56.186025 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:04:56.185998 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9e27edaf-mfbnh"] Jul 23 12:04:56.188285 ip-10-0-134-236 kubenswrapper[2561]: W0723 12:04:56.188256 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9f58d5e_3933_4d15_861e_42546183ff66.slice/crio-33bfb50afa7c523249b985ba5adc41009742a7e37a265af1561b92ff7f1706e5 WatchSource:0}: Error finding container 33bfb50afa7c523249b985ba5adc41009742a7e37a265af1561b92ff7f1706e5: Status 404 returned error can't find the container with id 33bfb50afa7c523249b985ba5adc41009742a7e37a265af1561b92ff7f1706e5 Jul 23 12:04:56.540194 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:04:56.540167 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9e27edaf-mfbnh" event={"ID":"b9f58d5e-3933-4d15-861e-42546183ff66","Type":"ContainerStarted","Data":"249fd5081e20a7345b9469110da0f9ac45673c64c43a2b2adfa4f13fca9b5cde"} Jul 23 12:04:56.540383 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:04:56.540212 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9e27edaf-mfbnh" event={"ID":"b9f58d5e-3933-4d15-861e-42546183ff66","Type":"ContainerStarted","Data":"33bfb50afa7c523249b985ba5adc41009742a7e37a265af1561b92ff7f1706e5"} Jul 23 12:04:56.567217 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:04:56.567175 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9e27edaf-mfbnh" podStartSLOduration=1.56715982 podStartE2EDuration="1.56715982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 12:04:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 12:04:56.56645861 +0000 UTC m=+1763.638148602" watchObservedRunningTime="2026-07-23 12:04:56.56715982 +0000 UTC m=+1763.638849858" Jul 23 12:04:57.547171 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:04:57.547137 2561 generic.go:356] "Generic (PLEG): container finished" podID="b9f58d5e-3933-4d15-861e-42546183ff66" containerID="249fd5081e20a7345b9469110da0f9ac45673c64c43a2b2adfa4f13fca9b5cde" exitCode=0 Jul 23 12:04:57.547557 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:04:57.547186 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9e27edaf-mfbnh" event={"ID":"b9f58d5e-3933-4d15-861e-42546183ff66","Type":"ContainerDied","Data":"249fd5081e20a7345b9469110da0f9ac45673c64c43a2b2adfa4f13fca9b5cde"} Jul 23 12:04:58.683087 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:04:58.683063 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9e27edaf-mfbnh" Jul 23 12:04:58.768846 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:04:58.768812 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9f58d5e-3933-4d15-861e-42546183ff66-maas-api-service-ca\") pod \"b9f58d5e-3933-4d15-861e-42546183ff66\" (UID: \"b9f58d5e-3933-4d15-861e-42546183ff66\") " Jul 23 12:04:58.769211 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:04:58.769184 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9f58d5e-3933-4d15-861e-42546183ff66-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b9f58d5e-3933-4d15-861e-42546183ff66" (UID: "b9f58d5e-3933-4d15-861e-42546183ff66"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 12:04:58.870023 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:04:58.869996 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9f58d5e-3933-4d15-861e-42546183ff66-maas-api-service-ca\") on node \"ip-10-0-134-236.ec2.internal\" DevicePath \"\"" Jul 23 12:04:59.559369 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:04:59.559343 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9e27edaf-mfbnh" Jul 23 12:04:59.559540 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:04:59.559342 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9e27edaf-mfbnh" event={"ID":"b9f58d5e-3933-4d15-861e-42546183ff66","Type":"ContainerDied","Data":"33bfb50afa7c523249b985ba5adc41009742a7e37a265af1561b92ff7f1706e5"} Jul 23 12:04:59.559540 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:04:59.559466 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33bfb50afa7c523249b985ba5adc41009742a7e37a265af1561b92ff7f1706e5" Jul 23 12:05:09.250993 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:05:09.250955 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 12:05:10.833006 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:05:10.832967 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9e27edaf-mfbnh"] Jul 23 12:05:10.842072 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:05:10.842042 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9e27edaf-mfbnh"] Jul 23 12:05:11.454438 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:05:11.454405 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9f58d5e-3933-4d15-861e-42546183ff66" path="/var/lib/kubelet/pods/b9f58d5e-3933-4d15-861e-42546183ff66/volumes" Jul 23 12:05:20.556930 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:05:20.556885 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 12:05:33.480593 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:05:33.480564 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-7slpj_65247e1f-fc10-4b40-af67-5e646f352263/console-operator/1.log" Jul 23 12:05:33.486953 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:05:33.486925 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-7slpj_65247e1f-fc10-4b40-af67-5e646f352263/console-operator/1.log" Jul 23 12:05:40.739257 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:05:40.739221 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1664359-6b6wr"] Jul 23 12:05:40.739644 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:05:40.739549 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9f58d5e-3933-4d15-861e-42546183ff66" containerName="revoke-keys" Jul 23 12:05:40.739644 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:05:40.739560 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9f58d5e-3933-4d15-861e-42546183ff66" containerName="revoke-keys" Jul 23 12:05:40.739644 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:05:40.739622 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9f58d5e-3933-4d15-861e-42546183ff66" containerName="revoke-keys" Jul 23 12:05:40.742274 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:05:40.742256 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1664359-6b6wr" Jul 23 12:05:40.744954 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:05:40.744900 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q8m5p\"" Jul 23 12:05:40.745276 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:05:40.745257 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 23 12:05:40.753868 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:05:40.753842 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1664359-6b6wr"] Jul 23 12:05:40.835319 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:05:40.835268 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/948aef2d-2bc5-466f-b4a8-9a0b3af5122b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-c1664359-6b6wr\" (UID: \"948aef2d-2bc5-466f-b4a8-9a0b3af5122b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1664359-6b6wr" Jul 23 12:05:40.936383 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:05:40.936341 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/948aef2d-2bc5-466f-b4a8-9a0b3af5122b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-c1664359-6b6wr\" (UID: \"948aef2d-2bc5-466f-b4a8-9a0b3af5122b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1664359-6b6wr" Jul 23 12:05:40.936987 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:05:40.936966 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/948aef2d-2bc5-466f-b4a8-9a0b3af5122b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-c1664359-6b6wr\" (UID: \"948aef2d-2bc5-466f-b4a8-9a0b3af5122b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1664359-6b6wr" Jul 23 12:05:41.054171 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:05:41.054067 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1664359-6b6wr" Jul 23 12:05:41.197727 ip-10-0-134-236 kubenswrapper[2561]: W0723 12:05:41.197693 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod948aef2d_2bc5_466f_b4a8_9a0b3af5122b.slice/crio-68352a454ad31f5400cf621620130cd716d9692ed8899f443d86159dc398ae99 WatchSource:0}: Error finding container 68352a454ad31f5400cf621620130cd716d9692ed8899f443d86159dc398ae99: Status 404 returned error can't find the container with id 68352a454ad31f5400cf621620130cd716d9692ed8899f443d86159dc398ae99 Jul 23 12:05:41.197854 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:05:41.197752 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1664359-6b6wr"] Jul 23 12:05:41.771305 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:05:41.771215 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1664359-6b6wr" event={"ID":"948aef2d-2bc5-466f-b4a8-9a0b3af5122b","Type":"ContainerStarted","Data":"54d5f1ceb458927f99884153d440ede7cb87c35bf551184eec6f1eeb1f5a3107"} Jul 23 12:05:41.771305 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:05:41.771273 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1664359-6b6wr" event={"ID":"948aef2d-2bc5-466f-b4a8-9a0b3af5122b","Type":"ContainerStarted","Data":"68352a454ad31f5400cf621620130cd716d9692ed8899f443d86159dc398ae99"} Jul 23 12:05:41.791143 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:05:41.791072 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1664359-6b6wr" podStartSLOduration=1.79105564 podStartE2EDuration="1.79105564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 12:05:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 12:05:41.789875524 +0000 UTC m=+1808.861565531" watchObservedRunningTime="2026-07-23 12:05:41.79105564 +0000 UTC m=+1808.862745630" Jul 23 12:05:42.777888 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:05:42.777847 2561 generic.go:356] "Generic (PLEG): container finished" podID="948aef2d-2bc5-466f-b4a8-9a0b3af5122b" containerID="54d5f1ceb458927f99884153d440ede7cb87c35bf551184eec6f1eeb1f5a3107" exitCode=0 Jul 23 12:05:42.778302 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:05:42.777944 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1664359-6b6wr" event={"ID":"948aef2d-2bc5-466f-b4a8-9a0b3af5122b","Type":"ContainerDied","Data":"54d5f1ceb458927f99884153d440ede7cb87c35bf551184eec6f1eeb1f5a3107"} Jul 23 12:05:43.926441 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:05:43.926410 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1664359-6b6wr" Jul 23 12:05:43.956213 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:05:43.956179 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/948aef2d-2bc5-466f-b4a8-9a0b3af5122b-maas-api-service-ca\") pod \"948aef2d-2bc5-466f-b4a8-9a0b3af5122b\" (UID: \"948aef2d-2bc5-466f-b4a8-9a0b3af5122b\") " Jul 23 12:05:43.956542 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:05:43.956518 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/948aef2d-2bc5-466f-b4a8-9a0b3af5122b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "948aef2d-2bc5-466f-b4a8-9a0b3af5122b" (UID: "948aef2d-2bc5-466f-b4a8-9a0b3af5122b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 12:05:44.056889 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:05:44.056785 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/948aef2d-2bc5-466f-b4a8-9a0b3af5122b-maas-api-service-ca\") on node \"ip-10-0-134-236.ec2.internal\" DevicePath \"\"" Jul 23 12:05:44.791280 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:05:44.791247 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1664359-6b6wr" Jul 23 12:05:44.791619 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:05:44.791278 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1664359-6b6wr" event={"ID":"948aef2d-2bc5-466f-b4a8-9a0b3af5122b","Type":"ContainerDied","Data":"68352a454ad31f5400cf621620130cd716d9692ed8899f443d86159dc398ae99"} Jul 23 12:05:44.791619 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:05:44.791327 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="68352a454ad31f5400cf621620130cd716d9692ed8899f443d86159dc398ae99" Jul 23 12:05:54.151848 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:05:54.151810 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 12:05:55.901133 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:05:55.901103 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1664359-6b6wr"] Jul 23 12:05:55.905188 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:05:55.905163 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1664359-6b6wr"] Jul 23 12:05:57.454990 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:05:57.454948 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="948aef2d-2bc5-466f-b4a8-9a0b3af5122b" path="/var/lib/kubelet/pods/948aef2d-2bc5-466f-b4a8-9a0b3af5122b/volumes" Jul 23 12:06:07.059285 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:06:07.059241 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 12:06:24.945026 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:06:24.944992 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2bbf915-fft48"] Jul 23 12:06:24.945438 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:06:24.945344 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="948aef2d-2bc5-466f-b4a8-9a0b3af5122b" containerName="revoke-keys" Jul 23 12:06:24.945438 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:06:24.945357 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="948aef2d-2bc5-466f-b4a8-9a0b3af5122b" containerName="revoke-keys" Jul 23 12:06:24.945438 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:06:24.945434 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="948aef2d-2bc5-466f-b4a8-9a0b3af5122b" containerName="revoke-keys" Jul 23 12:06:24.948403 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:06:24.948385 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2bbf915-fft48" Jul 23 12:06:24.951990 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:06:24.951966 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q8m5p\"" Jul 23 12:06:24.952110 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:06:24.951966 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 23 12:06:24.958474 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:06:24.958437 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2bbf915-fft48"] Jul 23 12:06:25.117694 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:06:25.117659 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04687c1b-6a56-4c53-8399-397abf9985f4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-c2bbf915-fft48\" (UID: \"04687c1b-6a56-4c53-8399-397abf9985f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2bbf915-fft48" Jul 23 12:06:25.219201 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:06:25.219102 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04687c1b-6a56-4c53-8399-397abf9985f4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-c2bbf915-fft48\" (UID: \"04687c1b-6a56-4c53-8399-397abf9985f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2bbf915-fft48" Jul 23 12:06:25.219821 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:06:25.219784 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04687c1b-6a56-4c53-8399-397abf9985f4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-c2bbf915-fft48\" (UID: \"04687c1b-6a56-4c53-8399-397abf9985f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2bbf915-fft48" Jul 23 12:06:25.259354 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:06:25.259310 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2bbf915-fft48" Jul 23 12:06:25.401592 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:06:25.401564 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2bbf915-fft48"] Jul 23 12:06:25.403458 ip-10-0-134-236 kubenswrapper[2561]: W0723 12:06:25.403430 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04687c1b_6a56_4c53_8399_397abf9985f4.slice/crio-91bd715e934b004e6d44a16e068d38c8647c89c006d63eba112fa84c6f4137f0 WatchSource:0}: Error finding container 91bd715e934b004e6d44a16e068d38c8647c89c006d63eba112fa84c6f4137f0: Status 404 returned error can't find the container with id 91bd715e934b004e6d44a16e068d38c8647c89c006d63eba112fa84c6f4137f0 Jul 23 12:06:25.997790 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:06:25.997701 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2bbf915-fft48" event={"ID":"04687c1b-6a56-4c53-8399-397abf9985f4","Type":"ContainerStarted","Data":"5647360ab0b73ef66635e4fb60f10ca2fece4aed9eaf6de71d6a70659b792441"} Jul 23 12:06:25.997790 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:06:25.997750 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2bbf915-fft48" event={"ID":"04687c1b-6a56-4c53-8399-397abf9985f4","Type":"ContainerStarted","Data":"91bd715e934b004e6d44a16e068d38c8647c89c006d63eba112fa84c6f4137f0"} Jul 23 12:06:26.014895 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:06:26.014835 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2bbf915-fft48" podStartSLOduration=2.0148129 podStartE2EDuration="2.0148129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 12:06:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 12:06:26.014399844 +0000 UTC m=+1853.086089835" watchObservedRunningTime="2026-07-23 12:06:26.0148129 +0000 UTC m=+1853.086502892" Jul 23 12:06:27.004514 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:06:27.004477 2561 generic.go:356] "Generic (PLEG): container finished" podID="04687c1b-6a56-4c53-8399-397abf9985f4" containerID="5647360ab0b73ef66635e4fb60f10ca2fece4aed9eaf6de71d6a70659b792441" exitCode=0 Jul 23 12:06:27.004893 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:06:27.004520 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2bbf915-fft48" event={"ID":"04687c1b-6a56-4c53-8399-397abf9985f4","Type":"ContainerDied","Data":"5647360ab0b73ef66635e4fb60f10ca2fece4aed9eaf6de71d6a70659b792441"} Jul 23 12:06:28.142325 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:06:28.142300 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2bbf915-fft48" Jul 23 12:06:28.247882 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:06:28.247842 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04687c1b-6a56-4c53-8399-397abf9985f4-maas-api-service-ca\") pod \"04687c1b-6a56-4c53-8399-397abf9985f4\" (UID: \"04687c1b-6a56-4c53-8399-397abf9985f4\") " Jul 23 12:06:28.248249 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:06:28.248224 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04687c1b-6a56-4c53-8399-397abf9985f4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "04687c1b-6a56-4c53-8399-397abf9985f4" (UID: "04687c1b-6a56-4c53-8399-397abf9985f4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 12:06:28.349058 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:06:28.349016 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04687c1b-6a56-4c53-8399-397abf9985f4-maas-api-service-ca\") on node \"ip-10-0-134-236.ec2.internal\" DevicePath \"\"" Jul 23 12:06:29.016509 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:06:29.016475 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2bbf915-fft48" event={"ID":"04687c1b-6a56-4c53-8399-397abf9985f4","Type":"ContainerDied","Data":"91bd715e934b004e6d44a16e068d38c8647c89c006d63eba112fa84c6f4137f0"} Jul 23 12:06:29.016509 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:06:29.016507 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2bbf915-fft48" Jul 23 12:06:29.016728 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:06:29.016523 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="91bd715e934b004e6d44a16e068d38c8647c89c006d63eba112fa84c6f4137f0" Jul 23 12:06:38.453885 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:06:38.453843 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 12:06:40.083653 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:06:40.083619 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2bbf915-fft48"] Jul 23 12:06:40.094293 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:06:40.094264 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c2bbf915-fft48"] Jul 23 12:06:41.455142 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:06:41.455107 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04687c1b-6a56-4c53-8399-397abf9985f4" path="/var/lib/kubelet/pods/04687c1b-6a56-4c53-8399-397abf9985f4/volumes" Jul 23 12:08:09.151891 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:08:09.151858 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 12:08:19.457464 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:08:19.457430 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 12:08:32.086334 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:08:32.086258 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 12:08:39.277240 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:08:39.277195 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-110e698d-pmqsg"] Jul 23 12:08:39.277755 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:08:39.277667 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04687c1b-6a56-4c53-8399-397abf9985f4" containerName="revoke-keys" Jul 23 12:08:39.277755 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:08:39.277688 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="04687c1b-6a56-4c53-8399-397abf9985f4" containerName="revoke-keys" Jul 23 12:08:39.277872 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:08:39.277797 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="04687c1b-6a56-4c53-8399-397abf9985f4" containerName="revoke-keys" Jul 23 12:08:39.280821 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:08:39.280797 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-110e698d-pmqsg" Jul 23 12:08:39.283520 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:08:39.283494 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 23 12:08:39.283633 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:08:39.283555 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q8m5p\"" Jul 23 12:08:39.290390 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:08:39.290368 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-110e698d-pmqsg"] Jul 23 12:08:39.321585 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:08:39.321560 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd9b1aa0-96c1-4742-92e5-6f9a6bd0c18a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-110e698d-pmqsg\" (UID: \"cd9b1aa0-96c1-4742-92e5-6f9a6bd0c18a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-110e698d-pmqsg" Jul 23 12:08:39.422451 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:08:39.422425 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd9b1aa0-96c1-4742-92e5-6f9a6bd0c18a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-110e698d-pmqsg\" (UID: \"cd9b1aa0-96c1-4742-92e5-6f9a6bd0c18a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-110e698d-pmqsg" Jul 23 12:08:39.423012 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:08:39.422993 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd9b1aa0-96c1-4742-92e5-6f9a6bd0c18a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-110e698d-pmqsg\" (UID: \"cd9b1aa0-96c1-4742-92e5-6f9a6bd0c18a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-110e698d-pmqsg" Jul 23 12:08:39.592988 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:08:39.592967 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-110e698d-pmqsg" Jul 23 12:08:39.962118 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:08:39.962049 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-110e698d-pmqsg"] Jul 23 12:08:39.965562 ip-10-0-134-236 kubenswrapper[2561]: W0723 12:08:39.965531 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd9b1aa0_96c1_4742_92e5_6f9a6bd0c18a.slice/crio-bdd760396c30ad256eaaa590eb475d2ca42e5b184be7de2416d4ec6e97357fb8 WatchSource:0}: Error finding container bdd760396c30ad256eaaa590eb475d2ca42e5b184be7de2416d4ec6e97357fb8: Status 404 returned error can't find the container with id bdd760396c30ad256eaaa590eb475d2ca42e5b184be7de2416d4ec6e97357fb8 Jul 23 12:08:40.685582 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:08:40.685545 2561 generic.go:356] "Generic (PLEG): container finished" podID="cd9b1aa0-96c1-4742-92e5-6f9a6bd0c18a" containerID="9df8b216b8550ce15a18ffc6aa08adc09d327a2de232972e67f2d7fc9104d3fb" exitCode=0 Jul 23 12:08:40.685986 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:08:40.685606 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-110e698d-pmqsg" event={"ID":"cd9b1aa0-96c1-4742-92e5-6f9a6bd0c18a","Type":"ContainerDied","Data":"9df8b216b8550ce15a18ffc6aa08adc09d327a2de232972e67f2d7fc9104d3fb"} Jul 23 12:08:40.685986 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:08:40.685649 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-110e698d-pmqsg" event={"ID":"cd9b1aa0-96c1-4742-92e5-6f9a6bd0c18a","Type":"ContainerStarted","Data":"bdd760396c30ad256eaaa590eb475d2ca42e5b184be7de2416d4ec6e97357fb8"} Jul 23 12:08:41.819573 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:08:41.819548 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-110e698d-pmqsg" Jul 23 12:08:41.843460 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:08:41.843434 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd9b1aa0-96c1-4742-92e5-6f9a6bd0c18a-maas-api-service-ca\") pod \"cd9b1aa0-96c1-4742-92e5-6f9a6bd0c18a\" (UID: \"cd9b1aa0-96c1-4742-92e5-6f9a6bd0c18a\") " Jul 23 12:08:41.843785 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:08:41.843752 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd9b1aa0-96c1-4742-92e5-6f9a6bd0c18a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cd9b1aa0-96c1-4742-92e5-6f9a6bd0c18a" (UID: "cd9b1aa0-96c1-4742-92e5-6f9a6bd0c18a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 12:08:41.944972 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:08:41.944888 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd9b1aa0-96c1-4742-92e5-6f9a6bd0c18a-maas-api-service-ca\") on node \"ip-10-0-134-236.ec2.internal\" DevicePath \"\"" Jul 23 12:08:42.696847 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:08:42.696817 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-110e698d-pmqsg" Jul 23 12:08:42.696847 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:08:42.696822 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-110e698d-pmqsg" event={"ID":"cd9b1aa0-96c1-4742-92e5-6f9a6bd0c18a","Type":"ContainerDied","Data":"bdd760396c30ad256eaaa590eb475d2ca42e5b184be7de2416d4ec6e97357fb8"} Jul 23 12:08:42.697097 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:08:42.696868 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bdd760396c30ad256eaaa590eb475d2ca42e5b184be7de2416d4ec6e97357fb8" Jul 23 12:08:52.486779 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:08:52.486732 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 12:08:54.406273 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:08:54.406241 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-110e698d-pmqsg"] Jul 23 12:08:54.421618 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:08:54.417582 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-110e698d-pmqsg"] Jul 23 12:08:55.454373 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:08:55.454336 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd9b1aa0-96c1-4742-92e5-6f9a6bd0c18a" path="/var/lib/kubelet/pods/cd9b1aa0-96c1-4742-92e5-6f9a6bd0c18a/volumes" Jul 23 12:09:30.348568 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:09:30.348527 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 12:10:05.583015 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:10:05.582927 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-66b55d3d/e2e-lifecycle-model-66b55d3d-kserve-86bb59bf5f-smqhv"] Jul 23 12:10:05.583479 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:10:05.583242 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd9b1aa0-96c1-4742-92e5-6f9a6bd0c18a" containerName="revoke-keys" Jul 23 12:10:05.583479 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:10:05.583253 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd9b1aa0-96c1-4742-92e5-6f9a6bd0c18a" containerName="revoke-keys" Jul 23 12:10:05.583479 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:10:05.583314 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd9b1aa0-96c1-4742-92e5-6f9a6bd0c18a" containerName="revoke-keys" Jul 23 12:10:05.586208 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:10:05.586192 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-66b55d3d/e2e-lifecycle-model-66b55d3d-kserve-86bb59bf5f-smqhv" Jul 23 12:10:05.593528 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:10:05.593505 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-66b55d3d\"/\"kube-root-ca.crt\"" Jul 23 12:10:05.593619 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:10:05.593505 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-66b55d3d\"/\"openshift-service-ca.crt\"" Jul 23 12:10:05.593619 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:10:05.593509 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-66b55d3d\"/\"default-dockercfg-jstcx\"" Jul 23 12:10:05.593689 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:10:05.593518 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-66b55d3d\"/\"e2e-lifecycle-model-66b55d3d-kserve-self-signed-certs\"" Jul 23 12:10:05.611848 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:10:05.611815 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-66b55d3d/e2e-lifecycle-model-66b55d3d-kserve-86bb59bf5f-smqhv"] Jul 23 12:10:05.750717 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:10:05.750678 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85fcb1cf-fe93-4233-b5b9-22e3628e3021-tmp-dir\") pod \"e2e-lifecycle-model-66b55d3d-kserve-86bb59bf5f-smqhv\" (UID: \"85fcb1cf-fe93-4233-b5b9-22e3628e3021\") " pod="ai-tenant-e2e-mt-66b55d3d/e2e-lifecycle-model-66b55d3d-kserve-86bb59bf5f-smqhv" Jul 23 12:10:05.750928 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:10:05.750736 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85fcb1cf-fe93-4233-b5b9-22e3628e3021-model-cache\") pod \"e2e-lifecycle-model-66b55d3d-kserve-86bb59bf5f-smqhv\" (UID: \"85fcb1cf-fe93-4233-b5b9-22e3628e3021\") " pod="ai-tenant-e2e-mt-66b55d3d/e2e-lifecycle-model-66b55d3d-kserve-86bb59bf5f-smqhv" Jul 23 12:10:05.750928 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:10:05.750820 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sphzr\" (UniqueName: \"kubernetes.io/projected/85fcb1cf-fe93-4233-b5b9-22e3628e3021-kube-api-access-sphzr\") pod \"e2e-lifecycle-model-66b55d3d-kserve-86bb59bf5f-smqhv\" (UID: \"85fcb1cf-fe93-4233-b5b9-22e3628e3021\") " pod="ai-tenant-e2e-mt-66b55d3d/e2e-lifecycle-model-66b55d3d-kserve-86bb59bf5f-smqhv" Jul 23 12:10:05.750928 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:10:05.750881 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85fcb1cf-fe93-4233-b5b9-22e3628e3021-tls-certs\") pod \"e2e-lifecycle-model-66b55d3d-kserve-86bb59bf5f-smqhv\" (UID: \"85fcb1cf-fe93-4233-b5b9-22e3628e3021\") " pod="ai-tenant-e2e-mt-66b55d3d/e2e-lifecycle-model-66b55d3d-kserve-86bb59bf5f-smqhv" Jul 23 12:10:05.751085 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:10:05.750937 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85fcb1cf-fe93-4233-b5b9-22e3628e3021-dshm\") pod \"e2e-lifecycle-model-66b55d3d-kserve-86bb59bf5f-smqhv\" (UID: \"85fcb1cf-fe93-4233-b5b9-22e3628e3021\") " pod="ai-tenant-e2e-mt-66b55d3d/e2e-lifecycle-model-66b55d3d-kserve-86bb59bf5f-smqhv" Jul 23 12:10:05.751085 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:10:05.750984 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85fcb1cf-fe93-4233-b5b9-22e3628e3021-home\") pod \"e2e-lifecycle-model-66b55d3d-kserve-86bb59bf5f-smqhv\" (UID: \"85fcb1cf-fe93-4233-b5b9-22e3628e3021\") " pod="ai-tenant-e2e-mt-66b55d3d/e2e-lifecycle-model-66b55d3d-kserve-86bb59bf5f-smqhv" Jul 23 12:10:05.852082 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:10:05.851985 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85fcb1cf-fe93-4233-b5b9-22e3628e3021-tmp-dir\") pod \"e2e-lifecycle-model-66b55d3d-kserve-86bb59bf5f-smqhv\" (UID: \"85fcb1cf-fe93-4233-b5b9-22e3628e3021\") " pod="ai-tenant-e2e-mt-66b55d3d/e2e-lifecycle-model-66b55d3d-kserve-86bb59bf5f-smqhv" Jul 23 12:10:05.852082 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:10:05.852040 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85fcb1cf-fe93-4233-b5b9-22e3628e3021-model-cache\") pod \"e2e-lifecycle-model-66b55d3d-kserve-86bb59bf5f-smqhv\" (UID: \"85fcb1cf-fe93-4233-b5b9-22e3628e3021\") " pod="ai-tenant-e2e-mt-66b55d3d/e2e-lifecycle-model-66b55d3d-kserve-86bb59bf5f-smqhv" Jul 23 12:10:05.852082 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:10:05.852060 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sphzr\" (UniqueName: \"kubernetes.io/projected/85fcb1cf-fe93-4233-b5b9-22e3628e3021-kube-api-access-sphzr\") pod \"e2e-lifecycle-model-66b55d3d-kserve-86bb59bf5f-smqhv\" (UID: \"85fcb1cf-fe93-4233-b5b9-22e3628e3021\") " pod="ai-tenant-e2e-mt-66b55d3d/e2e-lifecycle-model-66b55d3d-kserve-86bb59bf5f-smqhv" Jul 23 12:10:05.852324 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:10:05.852089 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85fcb1cf-fe93-4233-b5b9-22e3628e3021-tls-certs\") pod \"e2e-lifecycle-model-66b55d3d-kserve-86bb59bf5f-smqhv\" (UID: \"85fcb1cf-fe93-4233-b5b9-22e3628e3021\") " pod="ai-tenant-e2e-mt-66b55d3d/e2e-lifecycle-model-66b55d3d-kserve-86bb59bf5f-smqhv" Jul 23 12:10:05.852324 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:10:05.852133 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85fcb1cf-fe93-4233-b5b9-22e3628e3021-dshm\") pod \"e2e-lifecycle-model-66b55d3d-kserve-86bb59bf5f-smqhv\" (UID: \"85fcb1cf-fe93-4233-b5b9-22e3628e3021\") " pod="ai-tenant-e2e-mt-66b55d3d/e2e-lifecycle-model-66b55d3d-kserve-86bb59bf5f-smqhv" Jul 23 12:10:05.852324 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:10:05.852164 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85fcb1cf-fe93-4233-b5b9-22e3628e3021-home\") pod \"e2e-lifecycle-model-66b55d3d-kserve-86bb59bf5f-smqhv\" (UID: \"85fcb1cf-fe93-4233-b5b9-22e3628e3021\") " pod="ai-tenant-e2e-mt-66b55d3d/e2e-lifecycle-model-66b55d3d-kserve-86bb59bf5f-smqhv" Jul 23 12:10:05.852488 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:10:05.852464 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85fcb1cf-fe93-4233-b5b9-22e3628e3021-model-cache\") pod \"e2e-lifecycle-model-66b55d3d-kserve-86bb59bf5f-smqhv\" (UID: \"85fcb1cf-fe93-4233-b5b9-22e3628e3021\") " pod="ai-tenant-e2e-mt-66b55d3d/e2e-lifecycle-model-66b55d3d-kserve-86bb59bf5f-smqhv" Jul 23 12:10:05.852529 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:10:05.852490 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85fcb1cf-fe93-4233-b5b9-22e3628e3021-tmp-dir\") pod \"e2e-lifecycle-model-66b55d3d-kserve-86bb59bf5f-smqhv\" (UID: \"85fcb1cf-fe93-4233-b5b9-22e3628e3021\") " pod="ai-tenant-e2e-mt-66b55d3d/e2e-lifecycle-model-66b55d3d-kserve-86bb59bf5f-smqhv" Jul 23 12:10:05.852564 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:10:05.852537 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85fcb1cf-fe93-4233-b5b9-22e3628e3021-home\") pod \"e2e-lifecycle-model-66b55d3d-kserve-86bb59bf5f-smqhv\" (UID: \"85fcb1cf-fe93-4233-b5b9-22e3628e3021\") " pod="ai-tenant-e2e-mt-66b55d3d/e2e-lifecycle-model-66b55d3d-kserve-86bb59bf5f-smqhv" Jul 23 12:10:05.854510 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:10:05.854493 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85fcb1cf-fe93-4233-b5b9-22e3628e3021-dshm\") pod \"e2e-lifecycle-model-66b55d3d-kserve-86bb59bf5f-smqhv\" (UID: \"85fcb1cf-fe93-4233-b5b9-22e3628e3021\") " pod="ai-tenant-e2e-mt-66b55d3d/e2e-lifecycle-model-66b55d3d-kserve-86bb59bf5f-smqhv" Jul 23 12:10:05.854880 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:10:05.854860 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85fcb1cf-fe93-4233-b5b9-22e3628e3021-tls-certs\") pod \"e2e-lifecycle-model-66b55d3d-kserve-86bb59bf5f-smqhv\" (UID: \"85fcb1cf-fe93-4233-b5b9-22e3628e3021\") " pod="ai-tenant-e2e-mt-66b55d3d/e2e-lifecycle-model-66b55d3d-kserve-86bb59bf5f-smqhv" Jul 23 12:10:05.861837 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:10:05.861811 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sphzr\" (UniqueName: \"kubernetes.io/projected/85fcb1cf-fe93-4233-b5b9-22e3628e3021-kube-api-access-sphzr\") pod \"e2e-lifecycle-model-66b55d3d-kserve-86bb59bf5f-smqhv\" (UID: \"85fcb1cf-fe93-4233-b5b9-22e3628e3021\") " pod="ai-tenant-e2e-mt-66b55d3d/e2e-lifecycle-model-66b55d3d-kserve-86bb59bf5f-smqhv" Jul 23 12:10:05.895813 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:10:05.895779 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-66b55d3d/e2e-lifecycle-model-66b55d3d-kserve-86bb59bf5f-smqhv" Jul 23 12:10:06.038448 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:10:06.038426 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-66b55d3d/e2e-lifecycle-model-66b55d3d-kserve-86bb59bf5f-smqhv"] Jul 23 12:10:06.040214 ip-10-0-134-236 kubenswrapper[2561]: W0723 12:10:06.040187 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod85fcb1cf_fe93_4233_b5b9_22e3628e3021.slice/crio-004c558095113c179b4a649c0d27e833ef560b281eab781ecd3236faaee9e1ad WatchSource:0}: Error finding container 004c558095113c179b4a649c0d27e833ef560b281eab781ecd3236faaee9e1ad: Status 404 returned error can't find the container with id 004c558095113c179b4a649c0d27e833ef560b281eab781ecd3236faaee9e1ad Jul 23 12:10:06.132941 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:10:06.132835 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-66b55d3d/e2e-lifecycle-model-66b55d3d-kserve-86bb59bf5f-smqhv" event={"ID":"85fcb1cf-fe93-4233-b5b9-22e3628e3021","Type":"ContainerStarted","Data":"7cbc12cdeaef4a1950c556315707b77d4677bc4c71d915ba6574ee49caead5b9"} Jul 23 12:10:06.132941 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:10:06.132882 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-66b55d3d/e2e-lifecycle-model-66b55d3d-kserve-86bb59bf5f-smqhv" event={"ID":"85fcb1cf-fe93-4233-b5b9-22e3628e3021","Type":"ContainerStarted","Data":"004c558095113c179b4a649c0d27e833ef560b281eab781ecd3236faaee9e1ad"} Jul 23 12:10:08.939849 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:10:08.939790 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-66b55d3d/e2e-lifecycle-model-66b55d3d-kserve-86bb59bf5f-smqhv" podStartSLOduration=3.939772628 podStartE2EDuration="3.939772628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 12:10:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 12:10:06.16345476 +0000 UTC m=+2073.235144751" watchObservedRunningTime="2026-07-23 12:10:08.939772628 +0000 UTC m=+2076.011462626" Jul 23 12:10:08.940282 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:10:08.940139 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 12:10:15.896819 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:10:15.896787 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-66b55d3d/e2e-lifecycle-model-66b55d3d-kserve-86bb59bf5f-smqhv" Jul 23 12:10:15.897238 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:10:15.896836 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-66b55d3d/e2e-lifecycle-model-66b55d3d-kserve-86bb59bf5f-smqhv" Jul 23 12:10:15.909578 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:10:15.909553 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-66b55d3d/e2e-lifecycle-model-66b55d3d-kserve-86bb59bf5f-smqhv" Jul 23 12:10:16.195807 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:10:16.195721 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-66b55d3d/e2e-lifecycle-model-66b55d3d-kserve-86bb59bf5f-smqhv" Jul 23 12:10:33.518476 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:10:33.518449 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-7slpj_65247e1f-fc10-4b40-af67-5e646f352263/console-operator/1.log" Jul 23 12:10:33.521679 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:10:33.521654 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-7slpj_65247e1f-fc10-4b40-af67-5e646f352263/console-operator/1.log" Jul 23 12:10:33.540882 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:10:33.540847 2561 scope.go:117] "RemoveContainer" containerID="e295792c255500529e9335c61594bc490a7d6403d633b9daca64f3845082990a" Jul 23 12:10:33.548877 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:10:33.548843 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 12:10:39.873805 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:10:39.873768 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-66b55d3d-rxzww"] Jul 23 12:10:39.876883 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:10:39.876867 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-66b55d3d-rxzww" Jul 23 12:10:39.879678 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:10:39.879659 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 23 12:10:39.879789 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:10:39.879764 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q8m5p\"" Jul 23 12:10:39.895517 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:10:39.895487 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-66b55d3d-rxzww"] Jul 23 12:10:39.945222 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:10:39.945183 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af161750-370c-41dc-b88a-cfc1d6db4252-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-66b55d3d-rxzww\" (UID: \"af161750-370c-41dc-b88a-cfc1d6db4252\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-66b55d3d-rxzww" Jul 23 12:10:40.046513 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:10:40.046452 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af161750-370c-41dc-b88a-cfc1d6db4252-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-66b55d3d-rxzww\" (UID: \"af161750-370c-41dc-b88a-cfc1d6db4252\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-66b55d3d-rxzww" Jul 23 12:10:40.047198 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:10:40.047176 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af161750-370c-41dc-b88a-cfc1d6db4252-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-66b55d3d-rxzww\" (UID: \"af161750-370c-41dc-b88a-cfc1d6db4252\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-66b55d3d-rxzww" Jul 23 12:10:40.188975 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:10:40.188860 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-66b55d3d-rxzww" Jul 23 12:10:40.323184 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:10:40.323151 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-66b55d3d-rxzww"] Jul 23 12:10:40.325719 ip-10-0-134-236 kubenswrapper[2561]: W0723 12:10:40.325691 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf161750_370c_41dc_b88a_cfc1d6db4252.slice/crio-22cedf5018ea3c239d8e22501e471ebdea1cf8ef6d17b5c18e2edb88d99de7de WatchSource:0}: Error finding container 22cedf5018ea3c239d8e22501e471ebdea1cf8ef6d17b5c18e2edb88d99de7de: Status 404 returned error can't find the container with id 22cedf5018ea3c239d8e22501e471ebdea1cf8ef6d17b5c18e2edb88d99de7de Jul 23 12:10:41.313368 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:10:41.313337 2561 generic.go:356] "Generic (PLEG): container finished" podID="af161750-370c-41dc-b88a-cfc1d6db4252" containerID="8b0c18d4819d9003369e7d0db8b5381e4d8b297801d214bd0e982dd9106d837a" exitCode=0 Jul 23 12:10:41.313763 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:10:41.313401 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-66b55d3d-rxzww" event={"ID":"af161750-370c-41dc-b88a-cfc1d6db4252","Type":"ContainerDied","Data":"8b0c18d4819d9003369e7d0db8b5381e4d8b297801d214bd0e982dd9106d837a"} Jul 23 12:10:41.313763 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:10:41.313442 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-66b55d3d-rxzww" event={"ID":"af161750-370c-41dc-b88a-cfc1d6db4252","Type":"ContainerStarted","Data":"22cedf5018ea3c239d8e22501e471ebdea1cf8ef6d17b5c18e2edb88d99de7de"} Jul 23 12:10:42.452219 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:10:42.452195 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-66b55d3d-rxzww" Jul 23 12:10:42.567023 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:10:42.566983 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af161750-370c-41dc-b88a-cfc1d6db4252-maas-api-service-ca\") pod \"af161750-370c-41dc-b88a-cfc1d6db4252\" (UID: \"af161750-370c-41dc-b88a-cfc1d6db4252\") " Jul 23 12:10:42.567374 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:10:42.567350 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af161750-370c-41dc-b88a-cfc1d6db4252-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "af161750-370c-41dc-b88a-cfc1d6db4252" (UID: "af161750-370c-41dc-b88a-cfc1d6db4252"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 12:10:42.668563 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:10:42.668470 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af161750-370c-41dc-b88a-cfc1d6db4252-maas-api-service-ca\") on node \"ip-10-0-134-236.ec2.internal\" DevicePath \"\"" Jul 23 12:10:43.325275 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:10:43.325247 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-66b55d3d-rxzww" Jul 23 12:10:43.325450 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:10:43.325289 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-66b55d3d-rxzww" event={"ID":"af161750-370c-41dc-b88a-cfc1d6db4252","Type":"ContainerDied","Data":"22cedf5018ea3c239d8e22501e471ebdea1cf8ef6d17b5c18e2edb88d99de7de"} Jul 23 12:10:43.325450 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:10:43.325366 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="22cedf5018ea3c239d8e22501e471ebdea1cf8ef6d17b5c18e2edb88d99de7de" Jul 23 12:10:53.643607 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:10:53.643573 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 12:10:59.976645 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:10:59.976609 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-66b55d3d-rxzww"] Jul 23 12:10:59.979190 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:10:59.979166 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-66b55d3d-rxzww"] Jul 23 12:11:01.454262 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:11:01.454229 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af161750-370c-41dc-b88a-cfc1d6db4252" path="/var/lib/kubelet/pods/af161750-370c-41dc-b88a-cfc1d6db4252/volumes" Jul 23 12:11:04.141482 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:11:04.141450 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 12:11:07.327992 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:11:07.327958 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-66b55d3d/e2e-lifecycle-model-66b55d3d-kserve-86bb59bf5f-smqhv"] Jul 23 12:11:22.348960 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:11:22.348893 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-66b55d3d/e2e-lifecycle-model-66b55d3d-kserve-86bb59bf5f-smqhv" podUID="85fcb1cf-fe93-4233-b5b9-22e3628e3021" containerName="main" containerID="cri-o://7cbc12cdeaef4a1950c556315707b77d4677bc4c71d915ba6574ee49caead5b9" gracePeriod=45 Jul 23 12:11:22.540973 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:11:22.540938 2561 generic.go:356] "Generic (PLEG): container finished" podID="85fcb1cf-fe93-4233-b5b9-22e3628e3021" containerID="7cbc12cdeaef4a1950c556315707b77d4677bc4c71d915ba6574ee49caead5b9" exitCode=0 Jul 23 12:11:22.541131 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:11:22.540945 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-66b55d3d/e2e-lifecycle-model-66b55d3d-kserve-86bb59bf5f-smqhv" event={"ID":"85fcb1cf-fe93-4233-b5b9-22e3628e3021","Type":"ContainerDied","Data":"7cbc12cdeaef4a1950c556315707b77d4677bc4c71d915ba6574ee49caead5b9"} Jul 23 12:11:22.602683 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:11:22.602617 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-66b55d3d/e2e-lifecycle-model-66b55d3d-kserve-86bb59bf5f-smqhv" Jul 23 12:11:22.619800 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:11:22.619766 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85fcb1cf-fe93-4233-b5b9-22e3628e3021-home\") pod \"85fcb1cf-fe93-4233-b5b9-22e3628e3021\" (UID: \"85fcb1cf-fe93-4233-b5b9-22e3628e3021\") " Jul 23 12:11:22.619987 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:11:22.619822 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85fcb1cf-fe93-4233-b5b9-22e3628e3021-tmp-dir\") pod \"85fcb1cf-fe93-4233-b5b9-22e3628e3021\" (UID: \"85fcb1cf-fe93-4233-b5b9-22e3628e3021\") " Jul 23 12:11:22.619987 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:11:22.619843 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85fcb1cf-fe93-4233-b5b9-22e3628e3021-model-cache\") pod \"85fcb1cf-fe93-4233-b5b9-22e3628e3021\" (UID: \"85fcb1cf-fe93-4233-b5b9-22e3628e3021\") " Jul 23 12:11:22.619987 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:11:22.619865 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sphzr\" (UniqueName: \"kubernetes.io/projected/85fcb1cf-fe93-4233-b5b9-22e3628e3021-kube-api-access-sphzr\") pod \"85fcb1cf-fe93-4233-b5b9-22e3628e3021\" (UID: \"85fcb1cf-fe93-4233-b5b9-22e3628e3021\") " Jul 23 12:11:22.619987 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:11:22.619885 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85fcb1cf-fe93-4233-b5b9-22e3628e3021-tls-certs\") pod \"85fcb1cf-fe93-4233-b5b9-22e3628e3021\" (UID: \"85fcb1cf-fe93-4233-b5b9-22e3628e3021\") " Jul 23 12:11:22.619987 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:11:22.619900 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85fcb1cf-fe93-4233-b5b9-22e3628e3021-dshm\") pod \"85fcb1cf-fe93-4233-b5b9-22e3628e3021\" (UID: \"85fcb1cf-fe93-4233-b5b9-22e3628e3021\") " Jul 23 12:11:22.620249 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:11:22.620123 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85fcb1cf-fe93-4233-b5b9-22e3628e3021-model-cache" (OuterVolumeSpecName: "model-cache") pod "85fcb1cf-fe93-4233-b5b9-22e3628e3021" (UID: "85fcb1cf-fe93-4233-b5b9-22e3628e3021"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:11:22.620249 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:11:22.620131 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85fcb1cf-fe93-4233-b5b9-22e3628e3021-home" (OuterVolumeSpecName: "home") pod "85fcb1cf-fe93-4233-b5b9-22e3628e3021" (UID: "85fcb1cf-fe93-4233-b5b9-22e3628e3021"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:11:22.620249 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:11:22.620232 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85fcb1cf-fe93-4233-b5b9-22e3628e3021-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "85fcb1cf-fe93-4233-b5b9-22e3628e3021" (UID: "85fcb1cf-fe93-4233-b5b9-22e3628e3021"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:11:22.622980 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:11:22.622945 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85fcb1cf-fe93-4233-b5b9-22e3628e3021-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "85fcb1cf-fe93-4233-b5b9-22e3628e3021" (UID: "85fcb1cf-fe93-4233-b5b9-22e3628e3021"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 12:11:22.622980 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:11:22.622952 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85fcb1cf-fe93-4233-b5b9-22e3628e3021-kube-api-access-sphzr" (OuterVolumeSpecName: "kube-api-access-sphzr") pod "85fcb1cf-fe93-4233-b5b9-22e3628e3021" (UID: "85fcb1cf-fe93-4233-b5b9-22e3628e3021"). InnerVolumeSpecName "kube-api-access-sphzr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 12:11:22.623190 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:11:22.623117 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85fcb1cf-fe93-4233-b5b9-22e3628e3021-dshm" (OuterVolumeSpecName: "dshm") pod "85fcb1cf-fe93-4233-b5b9-22e3628e3021" (UID: "85fcb1cf-fe93-4233-b5b9-22e3628e3021"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:11:22.720799 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:11:22.720760 2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85fcb1cf-fe93-4233-b5b9-22e3628e3021-model-cache\") on node \"ip-10-0-134-236.ec2.internal\" DevicePath \"\"" Jul 23 12:11:22.720799 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:11:22.720791 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sphzr\" (UniqueName: \"kubernetes.io/projected/85fcb1cf-fe93-4233-b5b9-22e3628e3021-kube-api-access-sphzr\") on node \"ip-10-0-134-236.ec2.internal\" DevicePath \"\"" Jul 23 12:11:22.720799 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:11:22.720804 2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85fcb1cf-fe93-4233-b5b9-22e3628e3021-tls-certs\") on node \"ip-10-0-134-236.ec2.internal\" DevicePath \"\"" Jul 23 12:11:22.720799 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:11:22.720812 2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85fcb1cf-fe93-4233-b5b9-22e3628e3021-dshm\") on node \"ip-10-0-134-236.ec2.internal\" DevicePath \"\"" Jul 23 12:11:22.721118 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:11:22.720820 2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85fcb1cf-fe93-4233-b5b9-22e3628e3021-home\") on node \"ip-10-0-134-236.ec2.internal\" DevicePath \"\"" Jul 23 12:11:22.721118 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:11:22.720851 2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85fcb1cf-fe93-4233-b5b9-22e3628e3021-tmp-dir\") on node \"ip-10-0-134-236.ec2.internal\" DevicePath \"\"" Jul 23 12:11:23.547760 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:11:23.547721 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-66b55d3d/e2e-lifecycle-model-66b55d3d-kserve-86bb59bf5f-smqhv" Jul 23 12:11:23.547760 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:11:23.547747 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-66b55d3d/e2e-lifecycle-model-66b55d3d-kserve-86bb59bf5f-smqhv" event={"ID":"85fcb1cf-fe93-4233-b5b9-22e3628e3021","Type":"ContainerDied","Data":"004c558095113c179b4a649c0d27e833ef560b281eab781ecd3236faaee9e1ad"} Jul 23 12:11:23.548294 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:11:23.547798 2561 scope.go:117] "RemoveContainer" containerID="7cbc12cdeaef4a1950c556315707b77d4677bc4c71d915ba6574ee49caead5b9" Jul 23 12:11:23.567306 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:11:23.567271 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-66b55d3d/e2e-lifecycle-model-66b55d3d-kserve-86bb59bf5f-smqhv"] Jul 23 12:11:23.574674 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:11:23.574636 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-66b55d3d/e2e-lifecycle-model-66b55d3d-kserve-86bb59bf5f-smqhv"] Jul 23 12:11:25.455270 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:11:25.455220 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85fcb1cf-fe93-4233-b5b9-22e3628e3021" path="/var/lib/kubelet/pods/85fcb1cf-fe93-4233-b5b9-22e3628e3021/volumes" Jul 23 12:11:33.561688 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:11:33.561649 2561 scope.go:117] "RemoveContainer" containerID="249fd5081e20a7345b9469110da0f9ac45673c64c43a2b2adfa4f13fca9b5cde" Jul 23 12:11:55.144257 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:11:55.144219 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 12:12:07.765659 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:12:07.765620 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 12:12:33.587483 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:12:33.587453 2561 scope.go:117] "RemoveContainer" containerID="5647360ab0b73ef66635e4fb60f10ca2fece4aed9eaf6de71d6a70659b792441" Jul 23 12:12:33.595745 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:12:33.595728 2561 scope.go:117] "RemoveContainer" containerID="54d5f1ceb458927f99884153d440ede7cb87c35bf551184eec6f1eeb1f5a3107" Jul 23 12:12:55.241389 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:12:55.241314 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 12:12:58.939523 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:12:58.939490 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-480488-f8f8959cf-l69jf"] Jul 23 12:12:58.940029 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:12:58.940014 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85fcb1cf-fe93-4233-b5b9-22e3628e3021" containerName="main" Jul 23 12:12:58.940080 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:12:58.940033 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="85fcb1cf-fe93-4233-b5b9-22e3628e3021" containerName="main" Jul 23 12:12:58.940080 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:12:58.940058 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af161750-370c-41dc-b88a-cfc1d6db4252" containerName="revoke-keys" Jul 23 12:12:58.940080 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:12:58.940067 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="af161750-370c-41dc-b88a-cfc1d6db4252" containerName="revoke-keys" Jul 23 12:12:58.940173 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:12:58.940134 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="af161750-370c-41dc-b88a-cfc1d6db4252" containerName="revoke-keys" Jul 23 12:12:58.940173 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:12:58.940148 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="85fcb1cf-fe93-4233-b5b9-22e3628e3021" containerName="main" Jul 23 12:12:58.943446 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:12:58.943427 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-480488-f8f8959cf-l69jf" Jul 23 12:12:58.946345 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:12:58.946318 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-480488-serving-cert\"" Jul 23 12:12:58.946460 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:12:58.946377 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 23 12:12:58.946460 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:12:58.946332 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 23 12:12:58.947270 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:12:58.947255 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-c9lpl\"" Jul 23 12:12:58.956931 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:12:58.956892 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-480488-f8f8959cf-l69jf"] Jul 23 12:12:59.027546 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:12:59.027515 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6z2c\" (UniqueName: \"kubernetes.io/projected/ef219f43-23ff-4589-840c-248c4f0ae5b7-kube-api-access-x6z2c\") pod \"maas-api-e2e-inf-b-480488-f8f8959cf-l69jf\" (UID: \"ef219f43-23ff-4589-840c-248c4f0ae5b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-480488-f8f8959cf-l69jf" Jul 23 12:12:59.027728 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:12:59.027571 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef219f43-23ff-4589-840c-248c4f0ae5b7-openshift-service-ca\") pod \"maas-api-e2e-inf-b-480488-f8f8959cf-l69jf\" (UID: \"ef219f43-23ff-4589-840c-248c4f0ae5b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-480488-f8f8959cf-l69jf" Jul 23 12:12:59.027728 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:12:59.027677 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef219f43-23ff-4589-840c-248c4f0ae5b7-maas-api-tls\") pod \"maas-api-e2e-inf-b-480488-f8f8959cf-l69jf\" (UID: \"ef219f43-23ff-4589-840c-248c4f0ae5b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-480488-f8f8959cf-l69jf" Jul 23 12:12:59.128535 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:12:59.128507 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef219f43-23ff-4589-840c-248c4f0ae5b7-maas-api-tls\") pod \"maas-api-e2e-inf-b-480488-f8f8959cf-l69jf\" (UID: \"ef219f43-23ff-4589-840c-248c4f0ae5b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-480488-f8f8959cf-l69jf" Jul 23 12:12:59.128687 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:12:59.128546 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6z2c\" (UniqueName: \"kubernetes.io/projected/ef219f43-23ff-4589-840c-248c4f0ae5b7-kube-api-access-x6z2c\") pod \"maas-api-e2e-inf-b-480488-f8f8959cf-l69jf\" (UID: \"ef219f43-23ff-4589-840c-248c4f0ae5b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-480488-f8f8959cf-l69jf" Jul 23 12:12:59.128687 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:12:59.128583 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef219f43-23ff-4589-840c-248c4f0ae5b7-openshift-service-ca\") pod \"maas-api-e2e-inf-b-480488-f8f8959cf-l69jf\" (UID: \"ef219f43-23ff-4589-840c-248c4f0ae5b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-480488-f8f8959cf-l69jf" Jul 23 12:12:59.129160 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:12:59.129143 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef219f43-23ff-4589-840c-248c4f0ae5b7-openshift-service-ca\") pod \"maas-api-e2e-inf-b-480488-f8f8959cf-l69jf\" (UID: \"ef219f43-23ff-4589-840c-248c4f0ae5b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-480488-f8f8959cf-l69jf" Jul 23 12:12:59.131327 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:12:59.131304 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef219f43-23ff-4589-840c-248c4f0ae5b7-maas-api-tls\") pod \"maas-api-e2e-inf-b-480488-f8f8959cf-l69jf\" (UID: \"ef219f43-23ff-4589-840c-248c4f0ae5b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-480488-f8f8959cf-l69jf" Jul 23 12:12:59.140852 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:12:59.140832 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6z2c\" (UniqueName: \"kubernetes.io/projected/ef219f43-23ff-4589-840c-248c4f0ae5b7-kube-api-access-x6z2c\") pod \"maas-api-e2e-inf-b-480488-f8f8959cf-l69jf\" (UID: \"ef219f43-23ff-4589-840c-248c4f0ae5b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-480488-f8f8959cf-l69jf" Jul 23 12:12:59.258752 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:12:59.258694 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-480488-f8f8959cf-l69jf" Jul 23 12:12:59.401029 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:12:59.401003 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-480488-f8f8959cf-l69jf"] Jul 23 12:12:59.402334 ip-10-0-134-236 kubenswrapper[2561]: W0723 12:12:59.402302 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef219f43_23ff_4589_840c_248c4f0ae5b7.slice/crio-2a1522b80b7ac93c85a558b13b3e833d881c5cedbd7a095390c4c4d4e14aad6e WatchSource:0}: Error finding container 2a1522b80b7ac93c85a558b13b3e833d881c5cedbd7a095390c4c4d4e14aad6e: Status 404 returned error can't find the container with id 2a1522b80b7ac93c85a558b13b3e833d881c5cedbd7a095390c4c4d4e14aad6e Jul 23 12:12:59.403767 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:12:59.403752 2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 12:13:00.042168 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:13:00.042132 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-480488-f8f8959cf-l69jf" event={"ID":"ef219f43-23ff-4589-840c-248c4f0ae5b7","Type":"ContainerStarted","Data":"2a1522b80b7ac93c85a558b13b3e833d881c5cedbd7a095390c4c4d4e14aad6e"} Jul 23 12:13:02.054882 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:13:02.054841 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-480488-f8f8959cf-l69jf" event={"ID":"ef219f43-23ff-4589-840c-248c4f0ae5b7","Type":"ContainerStarted","Data":"f7610299c08866f72bdf6cfc848bfe30a7ea50cc0c7f872656084826563554eb"} Jul 23 12:13:02.055283 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:13:02.054939 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-480488-f8f8959cf-l69jf" Jul 23 12:13:02.075823 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:13:02.075777 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-480488-f8f8959cf-l69jf" podStartSLOduration=1.921941009 podStartE2EDuration="4.075765267s" totalImagesPullingTime="2.153824258s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 12:12:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 12:13:02.073941809 +0000 UTC m=+2249.145631802" watchObservedRunningTime="2026-07-23 12:13:02.075765267 +0000 UTC m=+2249.147455258" Jul 23 12:13:08.066137 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:13:08.066108 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-480488-f8f8959cf-l69jf" Jul 23 12:13:08.257201 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:13:08.257168 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-9d4f98/test-model-9d4f98-kserve-796df8544f-5lh69"] Jul 23 12:13:08.260347 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:13:08.260328 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9d4f98/test-model-9d4f98-kserve-796df8544f-5lh69" Jul 23 12:13:08.263097 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:13:08.263073 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-9d4f98\"/\"kube-root-ca.crt\"" Jul 23 12:13:08.263224 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:13:08.263119 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-9d4f98\"/\"test-model-9d4f98-kserve-self-signed-certs\"" Jul 23 12:13:08.263224 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:13:08.263144 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-9d4f98\"/\"openshift-service-ca.crt\"" Jul 23 12:13:08.263410 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:13:08.263396 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-9d4f98\"/\"default-dockercfg-xpgm2\"" Jul 23 12:13:08.269747 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:13:08.269725 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-9d4f98/test-model-9d4f98-kserve-796df8544f-5lh69"] Jul 23 12:13:08.406821 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:13:08.406747 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9042247-ffd4-4bd8-86f1-5af19cfdd65b-dshm\") pod \"test-model-9d4f98-kserve-796df8544f-5lh69\" (UID: \"d9042247-ffd4-4bd8-86f1-5af19cfdd65b\") " pod="ai-tenant-e2e-inf-a-9d4f98/test-model-9d4f98-kserve-796df8544f-5lh69" Jul 23 12:13:08.406821 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:13:08.406780 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9042247-ffd4-4bd8-86f1-5af19cfdd65b-home\") pod \"test-model-9d4f98-kserve-796df8544f-5lh69\" (UID: \"d9042247-ffd4-4bd8-86f1-5af19cfdd65b\") " pod="ai-tenant-e2e-inf-a-9d4f98/test-model-9d4f98-kserve-796df8544f-5lh69" Jul 23 12:13:08.406821 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:13:08.406797 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9042247-ffd4-4bd8-86f1-5af19cfdd65b-tmp-dir\") pod \"test-model-9d4f98-kserve-796df8544f-5lh69\" (UID: \"d9042247-ffd4-4bd8-86f1-5af19cfdd65b\") " pod="ai-tenant-e2e-inf-a-9d4f98/test-model-9d4f98-kserve-796df8544f-5lh69" Jul 23 12:13:08.406821 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:13:08.406816 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9042247-ffd4-4bd8-86f1-5af19cfdd65b-tls-certs\") pod \"test-model-9d4f98-kserve-796df8544f-5lh69\" (UID: \"d9042247-ffd4-4bd8-86f1-5af19cfdd65b\") " pod="ai-tenant-e2e-inf-a-9d4f98/test-model-9d4f98-kserve-796df8544f-5lh69" Jul 23 12:13:08.407084 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:13:08.406937 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28rgc\" (UniqueName: \"kubernetes.io/projected/d9042247-ffd4-4bd8-86f1-5af19cfdd65b-kube-api-access-28rgc\") pod \"test-model-9d4f98-kserve-796df8544f-5lh69\" (UID: \"d9042247-ffd4-4bd8-86f1-5af19cfdd65b\") " pod="ai-tenant-e2e-inf-a-9d4f98/test-model-9d4f98-kserve-796df8544f-5lh69" Jul 23 12:13:08.407084 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:13:08.407010 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9042247-ffd4-4bd8-86f1-5af19cfdd65b-model-cache\") pod \"test-model-9d4f98-kserve-796df8544f-5lh69\" (UID: \"d9042247-ffd4-4bd8-86f1-5af19cfdd65b\") " pod="ai-tenant-e2e-inf-a-9d4f98/test-model-9d4f98-kserve-796df8544f-5lh69" Jul 23 12:13:08.508063 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:13:08.508041 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9042247-ffd4-4bd8-86f1-5af19cfdd65b-tls-certs\") pod \"test-model-9d4f98-kserve-796df8544f-5lh69\" (UID: \"d9042247-ffd4-4bd8-86f1-5af19cfdd65b\") " pod="ai-tenant-e2e-inf-a-9d4f98/test-model-9d4f98-kserve-796df8544f-5lh69" Jul 23 12:13:08.508192 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:13:08.508090 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28rgc\" (UniqueName: \"kubernetes.io/projected/d9042247-ffd4-4bd8-86f1-5af19cfdd65b-kube-api-access-28rgc\") pod \"test-model-9d4f98-kserve-796df8544f-5lh69\" (UID: \"d9042247-ffd4-4bd8-86f1-5af19cfdd65b\") " pod="ai-tenant-e2e-inf-a-9d4f98/test-model-9d4f98-kserve-796df8544f-5lh69" Jul 23 12:13:08.508192 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:13:08.508143 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9042247-ffd4-4bd8-86f1-5af19cfdd65b-model-cache\") pod \"test-model-9d4f98-kserve-796df8544f-5lh69\" (UID: \"d9042247-ffd4-4bd8-86f1-5af19cfdd65b\") " pod="ai-tenant-e2e-inf-a-9d4f98/test-model-9d4f98-kserve-796df8544f-5lh69" Jul 23 12:13:08.508192 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:13:08.508165 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9042247-ffd4-4bd8-86f1-5af19cfdd65b-dshm\") pod \"test-model-9d4f98-kserve-796df8544f-5lh69\" (UID: \"d9042247-ffd4-4bd8-86f1-5af19cfdd65b\") " pod="ai-tenant-e2e-inf-a-9d4f98/test-model-9d4f98-kserve-796df8544f-5lh69" Jul 23 12:13:08.508320 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:13:08.508193 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9042247-ffd4-4bd8-86f1-5af19cfdd65b-home\") pod \"test-model-9d4f98-kserve-796df8544f-5lh69\" (UID: \"d9042247-ffd4-4bd8-86f1-5af19cfdd65b\") " pod="ai-tenant-e2e-inf-a-9d4f98/test-model-9d4f98-kserve-796df8544f-5lh69" Jul 23 12:13:08.508373 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:13:08.508341 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9042247-ffd4-4bd8-86f1-5af19cfdd65b-tmp-dir\") pod \"test-model-9d4f98-kserve-796df8544f-5lh69\" (UID: \"d9042247-ffd4-4bd8-86f1-5af19cfdd65b\") " pod="ai-tenant-e2e-inf-a-9d4f98/test-model-9d4f98-kserve-796df8544f-5lh69" Jul 23 12:13:08.508615 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:13:08.508579 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9042247-ffd4-4bd8-86f1-5af19cfdd65b-model-cache\") pod \"test-model-9d4f98-kserve-796df8544f-5lh69\" (UID: \"d9042247-ffd4-4bd8-86f1-5af19cfdd65b\") " pod="ai-tenant-e2e-inf-a-9d4f98/test-model-9d4f98-kserve-796df8544f-5lh69" Jul 23 12:13:08.508615 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:13:08.508615 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9042247-ffd4-4bd8-86f1-5af19cfdd65b-home\") pod \"test-model-9d4f98-kserve-796df8544f-5lh69\" (UID: \"d9042247-ffd4-4bd8-86f1-5af19cfdd65b\") " pod="ai-tenant-e2e-inf-a-9d4f98/test-model-9d4f98-kserve-796df8544f-5lh69" Jul 23 12:13:08.508776 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:13:08.508670 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9042247-ffd4-4bd8-86f1-5af19cfdd65b-tmp-dir\") pod \"test-model-9d4f98-kserve-796df8544f-5lh69\" (UID: \"d9042247-ffd4-4bd8-86f1-5af19cfdd65b\") " pod="ai-tenant-e2e-inf-a-9d4f98/test-model-9d4f98-kserve-796df8544f-5lh69" Jul 23 12:13:08.510592 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:13:08.510572 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9042247-ffd4-4bd8-86f1-5af19cfdd65b-dshm\") pod \"test-model-9d4f98-kserve-796df8544f-5lh69\" (UID: \"d9042247-ffd4-4bd8-86f1-5af19cfdd65b\") " pod="ai-tenant-e2e-inf-a-9d4f98/test-model-9d4f98-kserve-796df8544f-5lh69" Jul 23 12:13:08.510592 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:13:08.510587 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9042247-ffd4-4bd8-86f1-5af19cfdd65b-tls-certs\") pod \"test-model-9d4f98-kserve-796df8544f-5lh69\" (UID: \"d9042247-ffd4-4bd8-86f1-5af19cfdd65b\") " pod="ai-tenant-e2e-inf-a-9d4f98/test-model-9d4f98-kserve-796df8544f-5lh69" Jul 23 12:13:08.519375 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:13:08.519352 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28rgc\" (UniqueName: \"kubernetes.io/projected/d9042247-ffd4-4bd8-86f1-5af19cfdd65b-kube-api-access-28rgc\") pod \"test-model-9d4f98-kserve-796df8544f-5lh69\" (UID: \"d9042247-ffd4-4bd8-86f1-5af19cfdd65b\") " pod="ai-tenant-e2e-inf-a-9d4f98/test-model-9d4f98-kserve-796df8544f-5lh69" Jul 23 12:13:08.570999 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:13:08.570979 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9d4f98/test-model-9d4f98-kserve-796df8544f-5lh69" Jul 23 12:13:08.702752 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:13:08.702729 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-9d4f98/test-model-9d4f98-kserve-796df8544f-5lh69"] Jul 23 12:13:08.703936 ip-10-0-134-236 kubenswrapper[2561]: W0723 12:13:08.703898 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd9042247_ffd4_4bd8_86f1_5af19cfdd65b.slice/crio-c035158a58fb317af97128b0c7a7db9ddf1cd34712062531e1742e07cf59e151 WatchSource:0}: Error finding container c035158a58fb317af97128b0c7a7db9ddf1cd34712062531e1742e07cf59e151: Status 404 returned error can't find the container with id c035158a58fb317af97128b0c7a7db9ddf1cd34712062531e1742e07cf59e151 Jul 23 12:13:09.090165 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:13:09.090131 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9d4f98/test-model-9d4f98-kserve-796df8544f-5lh69" event={"ID":"d9042247-ffd4-4bd8-86f1-5af19cfdd65b","Type":"ContainerStarted","Data":"2f3bc1ddcc8f8a1ed3b0a5836e34114741b337da37f440d16173fdad2469afa6"} Jul 23 12:13:09.090165 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:13:09.090170 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9d4f98/test-model-9d4f98-kserve-796df8544f-5lh69" event={"ID":"d9042247-ffd4-4bd8-86f1-5af19cfdd65b","Type":"ContainerStarted","Data":"c035158a58fb317af97128b0c7a7db9ddf1cd34712062531e1742e07cf59e151"} Jul 23 12:13:09.111230 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:13:09.111183 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-9d4f98/test-model-9d4f98-kserve-796df8544f-5lh69" podStartSLOduration=1.111167578 podStartE2EDuration="1.111167578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 12:13:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 12:13:09.108639077 +0000 UTC m=+2256.180329081" watchObservedRunningTime="2026-07-23 12:13:09.111167578 +0000 UTC m=+2256.182857566" Jul 23 12:13:14.250796 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:13:14.250760 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 12:13:18.572043 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:13:18.572011 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-9d4f98/test-model-9d4f98-kserve-796df8544f-5lh69" Jul 23 12:13:18.572491 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:13:18.572154 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-9d4f98/test-model-9d4f98-kserve-796df8544f-5lh69" Jul 23 12:13:18.584584 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:13:18.584563 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-9d4f98/test-model-9d4f98-kserve-796df8544f-5lh69" Jul 23 12:13:19.156151 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:13:19.156121 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-9d4f98/test-model-9d4f98-kserve-796df8544f-5lh69" Jul 23 12:13:36.340770 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:13:36.340732 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 12:13:51.143291 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:13:51.143245 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 12:14:12.990765 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:12.990731 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-9d4f98/test-model-9d4f98-kserve-796df8544f-5lh69"] Jul 23 12:14:14.168603 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:14.168569 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9d4f98-f66vq"] Jul 23 12:14:14.172145 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:14.172121 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9d4f98-f66vq" Jul 23 12:14:14.176280 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:14.176257 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q8m5p\"" Jul 23 12:14:14.185211 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:14.185182 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9d4f98-f66vq"] Jul 23 12:14:14.369705 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:14.369659 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad3566f4-f605-4673-ad91-24603ed9f68d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9d4f98-f66vq\" (UID: \"ad3566f4-f605-4673-ad91-24603ed9f68d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9d4f98-f66vq" Jul 23 12:14:14.470549 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:14.470463 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad3566f4-f605-4673-ad91-24603ed9f68d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9d4f98-f66vq\" (UID: \"ad3566f4-f605-4673-ad91-24603ed9f68d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9d4f98-f66vq" Jul 23 12:14:14.471082 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:14.471061 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad3566f4-f605-4673-ad91-24603ed9f68d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9d4f98-f66vq\" (UID: \"ad3566f4-f605-4673-ad91-24603ed9f68d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9d4f98-f66vq" Jul 23 12:14:14.482498 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:14.482476 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9d4f98-f66vq" Jul 23 12:14:14.621578 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:14.621554 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9d4f98-f66vq"] Jul 23 12:14:14.623438 ip-10-0-134-236 kubenswrapper[2561]: W0723 12:14:14.623399 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad3566f4_f605_4673_ad91_24603ed9f68d.slice/crio-ccbddc4df0f1fad4651636e11c808f6cbc598ebea2ac3d842df976418cd3afaa WatchSource:0}: Error finding container ccbddc4df0f1fad4651636e11c808f6cbc598ebea2ac3d842df976418cd3afaa: Status 404 returned error can't find the container with id ccbddc4df0f1fad4651636e11c808f6cbc598ebea2ac3d842df976418cd3afaa Jul 23 12:14:15.437396 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:15.437362 2561 generic.go:356] "Generic (PLEG): container finished" podID="ad3566f4-f605-4673-ad91-24603ed9f68d" containerID="ec7a5f5976db8d94e3b8aea63a1d1bbca322283a103a26e8ea5c2975fef61f06" exitCode=0 Jul 23 12:14:15.437781 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:15.437451 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9d4f98-f66vq" event={"ID":"ad3566f4-f605-4673-ad91-24603ed9f68d","Type":"ContainerDied","Data":"ec7a5f5976db8d94e3b8aea63a1d1bbca322283a103a26e8ea5c2975fef61f06"} Jul 23 12:14:15.437781 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:15.437538 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9d4f98-f66vq" event={"ID":"ad3566f4-f605-4673-ad91-24603ed9f68d","Type":"ContainerStarted","Data":"ccbddc4df0f1fad4651636e11c808f6cbc598ebea2ac3d842df976418cd3afaa"} Jul 23 12:14:16.582794 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:16.582768 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9d4f98-f66vq" Jul 23 12:14:16.586524 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:16.586494 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad3566f4-f605-4673-ad91-24603ed9f68d-maas-api-service-ca\") pod \"ad3566f4-f605-4673-ad91-24603ed9f68d\" (UID: \"ad3566f4-f605-4673-ad91-24603ed9f68d\") " Jul 23 12:14:16.586769 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:16.586743 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad3566f4-f605-4673-ad91-24603ed9f68d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ad3566f4-f605-4673-ad91-24603ed9f68d" (UID: "ad3566f4-f605-4673-ad91-24603ed9f68d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 12:14:16.687119 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:16.687078 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad3566f4-f605-4673-ad91-24603ed9f68d-maas-api-service-ca\") on node \"ip-10-0-134-236.ec2.internal\" DevicePath \"\"" Jul 23 12:14:17.452344 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:17.452318 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9d4f98-f66vq" Jul 23 12:14:17.456073 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:17.456042 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9d4f98-f66vq" event={"ID":"ad3566f4-f605-4673-ad91-24603ed9f68d","Type":"ContainerDied","Data":"ccbddc4df0f1fad4651636e11c808f6cbc598ebea2ac3d842df976418cd3afaa"} Jul 23 12:14:17.456242 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:17.456089 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ccbddc4df0f1fad4651636e11c808f6cbc598ebea2ac3d842df976418cd3afaa" Jul 23 12:14:18.148341 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:18.148303 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 12:14:28.011518 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:28.011420 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-9d4f98/test-model-9d4f98-kserve-796df8544f-5lh69" podUID="d9042247-ffd4-4bd8-86f1-5af19cfdd65b" containerName="main" containerID="cri-o://2f3bc1ddcc8f8a1ed3b0a5836e34114741b337da37f440d16173fdad2469afa6" gracePeriod=45 Jul 23 12:14:28.272043 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:28.271965 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9d4f98/test-model-9d4f98-kserve-796df8544f-5lh69" Jul 23 12:14:28.384669 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:28.384630 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9042247-ffd4-4bd8-86f1-5af19cfdd65b-tls-certs\") pod \"d9042247-ffd4-4bd8-86f1-5af19cfdd65b\" (UID: \"d9042247-ffd4-4bd8-86f1-5af19cfdd65b\") " Jul 23 12:14:28.384669 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:28.384674 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-28rgc\" (UniqueName: \"kubernetes.io/projected/d9042247-ffd4-4bd8-86f1-5af19cfdd65b-kube-api-access-28rgc\") pod \"d9042247-ffd4-4bd8-86f1-5af19cfdd65b\" (UID: \"d9042247-ffd4-4bd8-86f1-5af19cfdd65b\") " Jul 23 12:14:28.384974 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:28.384704 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9042247-ffd4-4bd8-86f1-5af19cfdd65b-dshm\") pod \"d9042247-ffd4-4bd8-86f1-5af19cfdd65b\" (UID: \"d9042247-ffd4-4bd8-86f1-5af19cfdd65b\") " Jul 23 12:14:28.384974 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:28.384734 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9042247-ffd4-4bd8-86f1-5af19cfdd65b-model-cache\") pod \"d9042247-ffd4-4bd8-86f1-5af19cfdd65b\" (UID: \"d9042247-ffd4-4bd8-86f1-5af19cfdd65b\") " Jul 23 12:14:28.384974 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:28.384802 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9042247-ffd4-4bd8-86f1-5af19cfdd65b-home\") pod \"d9042247-ffd4-4bd8-86f1-5af19cfdd65b\" (UID: \"d9042247-ffd4-4bd8-86f1-5af19cfdd65b\") " Jul 23 12:14:28.384974 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:28.384843 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9042247-ffd4-4bd8-86f1-5af19cfdd65b-tmp-dir\") pod \"d9042247-ffd4-4bd8-86f1-5af19cfdd65b\" (UID: \"d9042247-ffd4-4bd8-86f1-5af19cfdd65b\") " Jul 23 12:14:28.385208 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:28.385087 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9042247-ffd4-4bd8-86f1-5af19cfdd65b-model-cache" (OuterVolumeSpecName: "model-cache") pod "d9042247-ffd4-4bd8-86f1-5af19cfdd65b" (UID: "d9042247-ffd4-4bd8-86f1-5af19cfdd65b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:14:28.385208 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:28.385132 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9042247-ffd4-4bd8-86f1-5af19cfdd65b-home" (OuterVolumeSpecName: "home") pod "d9042247-ffd4-4bd8-86f1-5af19cfdd65b" (UID: "d9042247-ffd4-4bd8-86f1-5af19cfdd65b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:14:28.385208 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:28.385198 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9042247-ffd4-4bd8-86f1-5af19cfdd65b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d9042247-ffd4-4bd8-86f1-5af19cfdd65b" (UID: "d9042247-ffd4-4bd8-86f1-5af19cfdd65b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:14:28.387130 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:28.387101 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9042247-ffd4-4bd8-86f1-5af19cfdd65b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d9042247-ffd4-4bd8-86f1-5af19cfdd65b" (UID: "d9042247-ffd4-4bd8-86f1-5af19cfdd65b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 12:14:28.387246 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:28.387151 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9042247-ffd4-4bd8-86f1-5af19cfdd65b-kube-api-access-28rgc" (OuterVolumeSpecName: "kube-api-access-28rgc") pod "d9042247-ffd4-4bd8-86f1-5af19cfdd65b" (UID: "d9042247-ffd4-4bd8-86f1-5af19cfdd65b"). InnerVolumeSpecName "kube-api-access-28rgc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 12:14:28.387246 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:28.387166 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9042247-ffd4-4bd8-86f1-5af19cfdd65b-dshm" (OuterVolumeSpecName: "dshm") pod "d9042247-ffd4-4bd8-86f1-5af19cfdd65b" (UID: "d9042247-ffd4-4bd8-86f1-5af19cfdd65b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:14:28.486151 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:28.486111 2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9042247-ffd4-4bd8-86f1-5af19cfdd65b-model-cache\") on node \"ip-10-0-134-236.ec2.internal\" DevicePath \"\"" Jul 23 12:14:28.486151 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:28.486141 2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9042247-ffd4-4bd8-86f1-5af19cfdd65b-home\") on node \"ip-10-0-134-236.ec2.internal\" DevicePath \"\"" Jul 23 12:14:28.486151 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:28.486150 2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9042247-ffd4-4bd8-86f1-5af19cfdd65b-tmp-dir\") on node \"ip-10-0-134-236.ec2.internal\" DevicePath \"\"" Jul 23 12:14:28.486151 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:28.486158 2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9042247-ffd4-4bd8-86f1-5af19cfdd65b-tls-certs\") on node \"ip-10-0-134-236.ec2.internal\" DevicePath \"\"" Jul 23 12:14:28.486437 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:28.486167 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28rgc\" (UniqueName: \"kubernetes.io/projected/d9042247-ffd4-4bd8-86f1-5af19cfdd65b-kube-api-access-28rgc\") on node \"ip-10-0-134-236.ec2.internal\" DevicePath \"\"" Jul 23 12:14:28.486437 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:28.486176 2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9042247-ffd4-4bd8-86f1-5af19cfdd65b-dshm\") on node \"ip-10-0-134-236.ec2.internal\" DevicePath \"\"" Jul 23 12:14:28.515258 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:28.515219 2561 generic.go:356] "Generic (PLEG): container finished" podID="d9042247-ffd4-4bd8-86f1-5af19cfdd65b" containerID="2f3bc1ddcc8f8a1ed3b0a5836e34114741b337da37f440d16173fdad2469afa6" exitCode=0 Jul 23 12:14:28.515447 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:28.515262 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9d4f98/test-model-9d4f98-kserve-796df8544f-5lh69" Jul 23 12:14:28.515447 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:28.515304 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9d4f98/test-model-9d4f98-kserve-796df8544f-5lh69" event={"ID":"d9042247-ffd4-4bd8-86f1-5af19cfdd65b","Type":"ContainerDied","Data":"2f3bc1ddcc8f8a1ed3b0a5836e34114741b337da37f440d16173fdad2469afa6"} Jul 23 12:14:28.515447 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:28.515346 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9d4f98/test-model-9d4f98-kserve-796df8544f-5lh69" event={"ID":"d9042247-ffd4-4bd8-86f1-5af19cfdd65b","Type":"ContainerDied","Data":"c035158a58fb317af97128b0c7a7db9ddf1cd34712062531e1742e07cf59e151"} Jul 23 12:14:28.515447 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:28.515367 2561 scope.go:117] "RemoveContainer" containerID="2f3bc1ddcc8f8a1ed3b0a5836e34114741b337da37f440d16173fdad2469afa6" Jul 23 12:14:28.525120 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:28.525102 2561 scope.go:117] "RemoveContainer" containerID="2f3bc1ddcc8f8a1ed3b0a5836e34114741b337da37f440d16173fdad2469afa6" Jul 23 12:14:28.525405 ip-10-0-134-236 kubenswrapper[2561]: E0723 12:14:28.525384 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f3bc1ddcc8f8a1ed3b0a5836e34114741b337da37f440d16173fdad2469afa6\": container with ID starting with 2f3bc1ddcc8f8a1ed3b0a5836e34114741b337da37f440d16173fdad2469afa6 not found: ID does not exist" containerID="2f3bc1ddcc8f8a1ed3b0a5836e34114741b337da37f440d16173fdad2469afa6" Jul 23 12:14:28.525456 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:28.525415 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f3bc1ddcc8f8a1ed3b0a5836e34114741b337da37f440d16173fdad2469afa6"} err="failed to get container status \"2f3bc1ddcc8f8a1ed3b0a5836e34114741b337da37f440d16173fdad2469afa6\": rpc error: code = NotFound desc = could not find container \"2f3bc1ddcc8f8a1ed3b0a5836e34114741b337da37f440d16173fdad2469afa6\": container with ID starting with 2f3bc1ddcc8f8a1ed3b0a5836e34114741b337da37f440d16173fdad2469afa6 not found: ID does not exist" Jul 23 12:14:28.539278 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:28.539237 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-9d4f98/test-model-9d4f98-kserve-796df8544f-5lh69"] Jul 23 12:14:28.543492 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:28.543465 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-9d4f98/test-model-9d4f98-kserve-796df8544f-5lh69"] Jul 23 12:14:29.279018 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:29.278988 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9d4f98-f66vq"] Jul 23 12:14:29.283200 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:29.283174 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9d4f98-f66vq"] Jul 23 12:14:29.454529 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:29.454499 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad3566f4-f605-4673-ad91-24603ed9f68d" path="/var/lib/kubelet/pods/ad3566f4-f605-4673-ad91-24603ed9f68d/volumes" Jul 23 12:14:29.454889 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:29.454876 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9042247-ffd4-4bd8-86f1-5af19cfdd65b" path="/var/lib/kubelet/pods/d9042247-ffd4-4bd8-86f1-5af19cfdd65b/volumes" Jul 23 12:14:34.785827 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:34.785790 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 12:14:38.036038 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:38.035994 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-480488-9jqjx"] Jul 23 12:14:38.036449 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:38.036358 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad3566f4-f605-4673-ad91-24603ed9f68d" containerName="revoke-keys" Jul 23 12:14:38.036449 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:38.036374 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad3566f4-f605-4673-ad91-24603ed9f68d" containerName="revoke-keys" Jul 23 12:14:38.036449 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:38.036396 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9042247-ffd4-4bd8-86f1-5af19cfdd65b" containerName="main" Jul 23 12:14:38.036449 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:38.036401 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9042247-ffd4-4bd8-86f1-5af19cfdd65b" containerName="main" Jul 23 12:14:38.036585 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:38.036460 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9042247-ffd4-4bd8-86f1-5af19cfdd65b" containerName="main" Jul 23 12:14:38.036585 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:38.036474 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad3566f4-f605-4673-ad91-24603ed9f68d" containerName="revoke-keys" Jul 23 12:14:38.040842 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:38.040822 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-480488-9jqjx" Jul 23 12:14:38.044189 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:38.044173 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q8m5p\"" Jul 23 12:14:38.048967 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:38.048941 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-480488-9jqjx"] Jul 23 12:14:38.171392 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:38.171358 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4aad368f-9761-4896-87d3-cbf8ec40d96e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-480488-9jqjx\" (UID: \"4aad368f-9761-4896-87d3-cbf8ec40d96e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-480488-9jqjx" Jul 23 12:14:38.272796 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:38.272752 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4aad368f-9761-4896-87d3-cbf8ec40d96e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-480488-9jqjx\" (UID: \"4aad368f-9761-4896-87d3-cbf8ec40d96e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-480488-9jqjx" Jul 23 12:14:38.273509 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:38.273479 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4aad368f-9761-4896-87d3-cbf8ec40d96e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-480488-9jqjx\" (UID: \"4aad368f-9761-4896-87d3-cbf8ec40d96e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-480488-9jqjx" Jul 23 12:14:38.353034 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:38.352990 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-480488-9jqjx" Jul 23 12:14:38.513023 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:38.512995 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-480488-9jqjx"] Jul 23 12:14:38.515423 ip-10-0-134-236 kubenswrapper[2561]: W0723 12:14:38.515396 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4aad368f_9761_4896_87d3_cbf8ec40d96e.slice/crio-41252bf014f56a23da587f5fe0ac48fc1d68c580dbd38212b56b3dea9dcc0d34 WatchSource:0}: Error finding container 41252bf014f56a23da587f5fe0ac48fc1d68c580dbd38212b56b3dea9dcc0d34: Status 404 returned error can't find the container with id 41252bf014f56a23da587f5fe0ac48fc1d68c580dbd38212b56b3dea9dcc0d34 Jul 23 12:14:38.569523 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:38.569487 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-480488-9jqjx" event={"ID":"4aad368f-9761-4896-87d3-cbf8ec40d96e","Type":"ContainerStarted","Data":"41252bf014f56a23da587f5fe0ac48fc1d68c580dbd38212b56b3dea9dcc0d34"} Jul 23 12:14:39.577046 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:39.577005 2561 generic.go:356] "Generic (PLEG): container finished" podID="4aad368f-9761-4896-87d3-cbf8ec40d96e" containerID="dea89109ef7e804891d63f886cde0be889f4d16d1e582f1ccb09417544ac4ac6" exitCode=0 Jul 23 12:14:39.577494 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:39.577096 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-480488-9jqjx" event={"ID":"4aad368f-9761-4896-87d3-cbf8ec40d96e","Type":"ContainerDied","Data":"dea89109ef7e804891d63f886cde0be889f4d16d1e582f1ccb09417544ac4ac6"} Jul 23 12:14:40.729237 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:40.729199 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-480488-9jqjx" Jul 23 12:14:40.794744 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:40.794703 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4aad368f-9761-4896-87d3-cbf8ec40d96e-maas-api-service-ca\") pod \"4aad368f-9761-4896-87d3-cbf8ec40d96e\" (UID: \"4aad368f-9761-4896-87d3-cbf8ec40d96e\") " Jul 23 12:14:40.795147 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:40.795126 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4aad368f-9761-4896-87d3-cbf8ec40d96e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4aad368f-9761-4896-87d3-cbf8ec40d96e" (UID: "4aad368f-9761-4896-87d3-cbf8ec40d96e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 12:14:40.896261 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:40.896155 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4aad368f-9761-4896-87d3-cbf8ec40d96e-maas-api-service-ca\") on node \"ip-10-0-134-236.ec2.internal\" DevicePath \"\"" Jul 23 12:14:41.589674 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:41.589644 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-480488-9jqjx" Jul 23 12:14:41.589855 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:41.589648 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-480488-9jqjx" event={"ID":"4aad368f-9761-4896-87d3-cbf8ec40d96e","Type":"ContainerDied","Data":"41252bf014f56a23da587f5fe0ac48fc1d68c580dbd38212b56b3dea9dcc0d34"} Jul 23 12:14:41.589855 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:41.589772 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41252bf014f56a23da587f5fe0ac48fc1d68c580dbd38212b56b3dea9dcc0d34" Jul 23 12:14:48.663305 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:48.663265 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-480488-f8f8959cf-l69jf"] Jul 23 12:14:48.664362 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:48.663503 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-480488-f8f8959cf-l69jf" podUID="ef219f43-23ff-4589-840c-248c4f0ae5b7" containerName="maas-api" containerID="cri-o://f7610299c08866f72bdf6cfc848bfe30a7ea50cc0c7f872656084826563554eb" gracePeriod=30 Jul 23 12:14:48.925040 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:48.924944 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-480488-f8f8959cf-l69jf" Jul 23 12:14:49.075492 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:49.075455 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef219f43-23ff-4589-840c-248c4f0ae5b7-openshift-service-ca\") pod \"ef219f43-23ff-4589-840c-248c4f0ae5b7\" (UID: \"ef219f43-23ff-4589-840c-248c4f0ae5b7\") " Jul 23 12:14:49.075690 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:49.075544 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef219f43-23ff-4589-840c-248c4f0ae5b7-maas-api-tls\") pod \"ef219f43-23ff-4589-840c-248c4f0ae5b7\" (UID: \"ef219f43-23ff-4589-840c-248c4f0ae5b7\") " Jul 23 12:14:49.075690 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:49.075626 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x6z2c\" (UniqueName: \"kubernetes.io/projected/ef219f43-23ff-4589-840c-248c4f0ae5b7-kube-api-access-x6z2c\") pod \"ef219f43-23ff-4589-840c-248c4f0ae5b7\" (UID: \"ef219f43-23ff-4589-840c-248c4f0ae5b7\") " Jul 23 12:14:49.075877 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:49.075840 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef219f43-23ff-4589-840c-248c4f0ae5b7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ef219f43-23ff-4589-840c-248c4f0ae5b7" (UID: "ef219f43-23ff-4589-840c-248c4f0ae5b7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 12:14:49.078168 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:49.078127 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef219f43-23ff-4589-840c-248c4f0ae5b7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ef219f43-23ff-4589-840c-248c4f0ae5b7" (UID: "ef219f43-23ff-4589-840c-248c4f0ae5b7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 12:14:49.078316 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:49.078165 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef219f43-23ff-4589-840c-248c4f0ae5b7-kube-api-access-x6z2c" (OuterVolumeSpecName: "kube-api-access-x6z2c") pod "ef219f43-23ff-4589-840c-248c4f0ae5b7" (UID: "ef219f43-23ff-4589-840c-248c4f0ae5b7"). InnerVolumeSpecName "kube-api-access-x6z2c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 12:14:49.176669 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:49.176571 2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef219f43-23ff-4589-840c-248c4f0ae5b7-openshift-service-ca\") on node \"ip-10-0-134-236.ec2.internal\" DevicePath \"\"" Jul 23 12:14:49.176669 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:49.176607 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef219f43-23ff-4589-840c-248c4f0ae5b7-maas-api-tls\") on node \"ip-10-0-134-236.ec2.internal\" DevicePath \"\"" Jul 23 12:14:49.176669 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:49.176616 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6z2c\" (UniqueName: \"kubernetes.io/projected/ef219f43-23ff-4589-840c-248c4f0ae5b7-kube-api-access-x6z2c\") on node \"ip-10-0-134-236.ec2.internal\" DevicePath \"\"" Jul 23 12:14:49.637193 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:49.637160 2561 generic.go:356] "Generic (PLEG): container finished" podID="ef219f43-23ff-4589-840c-248c4f0ae5b7" containerID="f7610299c08866f72bdf6cfc848bfe30a7ea50cc0c7f872656084826563554eb" exitCode=0 Jul 23 12:14:49.637417 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:49.637210 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-480488-f8f8959cf-l69jf" event={"ID":"ef219f43-23ff-4589-840c-248c4f0ae5b7","Type":"ContainerDied","Data":"f7610299c08866f72bdf6cfc848bfe30a7ea50cc0c7f872656084826563554eb"} Jul 23 12:14:49.637417 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:49.637222 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-480488-f8f8959cf-l69jf" Jul 23 12:14:49.637417 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:49.637253 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-480488-f8f8959cf-l69jf" event={"ID":"ef219f43-23ff-4589-840c-248c4f0ae5b7","Type":"ContainerDied","Data":"2a1522b80b7ac93c85a558b13b3e833d881c5cedbd7a095390c4c4d4e14aad6e"} Jul 23 12:14:49.637417 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:49.637276 2561 scope.go:117] "RemoveContainer" containerID="f7610299c08866f72bdf6cfc848bfe30a7ea50cc0c7f872656084826563554eb" Jul 23 12:14:49.646750 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:49.646726 2561 scope.go:117] "RemoveContainer" containerID="f7610299c08866f72bdf6cfc848bfe30a7ea50cc0c7f872656084826563554eb" Jul 23 12:14:49.647096 ip-10-0-134-236 kubenswrapper[2561]: E0723 12:14:49.647071 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7610299c08866f72bdf6cfc848bfe30a7ea50cc0c7f872656084826563554eb\": container with ID starting with f7610299c08866f72bdf6cfc848bfe30a7ea50cc0c7f872656084826563554eb not found: ID does not exist" containerID="f7610299c08866f72bdf6cfc848bfe30a7ea50cc0c7f872656084826563554eb" Jul 23 12:14:49.647218 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:49.647103 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7610299c08866f72bdf6cfc848bfe30a7ea50cc0c7f872656084826563554eb"} err="failed to get container status \"f7610299c08866f72bdf6cfc848bfe30a7ea50cc0c7f872656084826563554eb\": rpc error: code = NotFound desc = could not find container \"f7610299c08866f72bdf6cfc848bfe30a7ea50cc0c7f872656084826563554eb\": container with ID starting with f7610299c08866f72bdf6cfc848bfe30a7ea50cc0c7f872656084826563554eb not found: ID does not exist" Jul 23 12:14:49.656815 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:49.656773 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-480488-f8f8959cf-l69jf"] Jul 23 12:14:49.661241 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:49.661204 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-480488-f8f8959cf-l69jf"] Jul 23 12:14:51.455777 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:51.455730 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef219f43-23ff-4589-840c-248c4f0ae5b7" path="/var/lib/kubelet/pods/ef219f43-23ff-4589-840c-248c4f0ae5b7/volumes" Jul 23 12:14:51.946635 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:51.946595 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 12:14:53.192965 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:53.192921 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-480488-9jqjx"] Jul 23 12:14:53.195769 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:53.195735 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-480488-9jqjx"] Jul 23 12:14:53.454633 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:14:53.454550 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4aad368f-9761-4896-87d3-cbf8ec40d96e" path="/var/lib/kubelet/pods/4aad368f-9761-4896-87d3-cbf8ec40d96e/volumes" Jul 23 12:15:00.148580 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:00.148539 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29746815-9m4r5"] Jul 23 12:15:00.149115 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:00.148878 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4aad368f-9761-4896-87d3-cbf8ec40d96e" containerName="revoke-keys" Jul 23 12:15:00.149115 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:00.148888 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="4aad368f-9761-4896-87d3-cbf8ec40d96e" containerName="revoke-keys" Jul 23 12:15:00.149115 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:00.148904 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef219f43-23ff-4589-840c-248c4f0ae5b7" containerName="maas-api" Jul 23 12:15:00.149115 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:00.148924 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef219f43-23ff-4589-840c-248c4f0ae5b7" containerName="maas-api" Jul 23 12:15:00.149115 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:00.148980 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef219f43-23ff-4589-840c-248c4f0ae5b7" containerName="maas-api" Jul 23 12:15:00.149115 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:00.148989 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="4aad368f-9761-4896-87d3-cbf8ec40d96e" containerName="revoke-keys" Jul 23 12:15:00.153312 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:00.153291 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746815-9m4r5" Jul 23 12:15:00.156548 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:00.156516 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q8m5p\"" Jul 23 12:15:00.160229 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:00.160199 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29746815-9m4r5"] Jul 23 12:15:00.166529 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:00.166498 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746815-9m4r5" Jul 23 12:15:00.317570 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:00.317542 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29746815-9m4r5"] Jul 23 12:15:00.319619 ip-10-0-134-236 kubenswrapper[2561]: W0723 12:15:00.319586 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc10604aa_cbbb_40e5_8c7d_8e1c16cab1db.slice/crio-97f13e492eae6345ae55cba4e17d5047464a6b5cbf046fb10f7701f6c3be58fc WatchSource:0}: Error finding container 97f13e492eae6345ae55cba4e17d5047464a6b5cbf046fb10f7701f6c3be58fc: Status 404 returned error can't find the container with id 97f13e492eae6345ae55cba4e17d5047464a6b5cbf046fb10f7701f6c3be58fc Jul 23 12:15:00.698193 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:00.698154 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746815-9m4r5" event={"ID":"c10604aa-cbbb-40e5-8c7d-8e1c16cab1db","Type":"ContainerStarted","Data":"b6226ecf7f6c93ebca16402ce9ddf43524478b1746448660353f092ba4865d0b"} Jul 23 12:15:00.698408 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:00.698208 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746815-9m4r5" event={"ID":"c10604aa-cbbb-40e5-8c7d-8e1c16cab1db","Type":"ContainerStarted","Data":"97f13e492eae6345ae55cba4e17d5047464a6b5cbf046fb10f7701f6c3be58fc"} Jul 23 12:15:00.718608 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:00.718552 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746815-9m4r5" podStartSLOduration=0.718530058 podStartE2EDuration="718.530058ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 12:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 12:15:00.71625097 +0000 UTC m=+2367.787940958" watchObservedRunningTime="2026-07-23 12:15:00.718530058 +0000 UTC m=+2367.790220026" Jul 23 12:15:01.705555 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:01.705462 2561 generic.go:356] "Generic (PLEG): container finished" podID="c10604aa-cbbb-40e5-8c7d-8e1c16cab1db" containerID="b6226ecf7f6c93ebca16402ce9ddf43524478b1746448660353f092ba4865d0b" exitCode=0 Jul 23 12:15:01.705555 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:01.705524 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746815-9m4r5" event={"ID":"c10604aa-cbbb-40e5-8c7d-8e1c16cab1db","Type":"ContainerDied","Data":"b6226ecf7f6c93ebca16402ce9ddf43524478b1746448660353f092ba4865d0b"} Jul 23 12:15:02.873251 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:02.873220 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746815-9m4r5" Jul 23 12:15:03.719830 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:03.719743 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746815-9m4r5" Jul 23 12:15:03.719830 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:03.719754 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746815-9m4r5" event={"ID":"c10604aa-cbbb-40e5-8c7d-8e1c16cab1db","Type":"ContainerDied","Data":"97f13e492eae6345ae55cba4e17d5047464a6b5cbf046fb10f7701f6c3be58fc"} Jul 23 12:15:03.719830 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:03.719801 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="97f13e492eae6345ae55cba4e17d5047464a6b5cbf046fb10f7701f6c3be58fc" Jul 23 12:15:04.840723 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:04.840683 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-091828-df7f748f7-8j742"] Jul 23 12:15:04.841221 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:04.841204 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c10604aa-cbbb-40e5-8c7d-8e1c16cab1db" containerName="cleanup" Jul 23 12:15:04.841272 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:04.841224 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c10604aa-cbbb-40e5-8c7d-8e1c16cab1db" containerName="cleanup" Jul 23 12:15:04.841315 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:04.841304 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="c10604aa-cbbb-40e5-8c7d-8e1c16cab1db" containerName="cleanup" Jul 23 12:15:04.844600 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:04.844575 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-091828-df7f748f7-8j742" Jul 23 12:15:04.847748 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:04.847704 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 23 12:15:04.848039 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:04.848022 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 23 12:15:04.848702 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:04.848681 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-c9lpl\"" Jul 23 12:15:04.849593 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:04.849573 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-091828-serving-cert\"" Jul 23 12:15:04.862024 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:04.861989 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-091828-df7f748f7-8j742"] Jul 23 12:15:04.919011 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:04.918972 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1add3517-ad57-4bb6-89af-2c07bd572329-maas-api-tls\") pod \"maas-api-e2e-api-a-091828-df7f748f7-8j742\" (UID: \"1add3517-ad57-4bb6-89af-2c07bd572329\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-091828-df7f748f7-8j742" Jul 23 12:15:04.919209 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:04.919084 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2fbr\" (UniqueName: \"kubernetes.io/projected/1add3517-ad57-4bb6-89af-2c07bd572329-kube-api-access-w2fbr\") pod \"maas-api-e2e-api-a-091828-df7f748f7-8j742\" (UID: \"1add3517-ad57-4bb6-89af-2c07bd572329\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-091828-df7f748f7-8j742" Jul 23 12:15:04.919209 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:04.919123 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1add3517-ad57-4bb6-89af-2c07bd572329-openshift-service-ca\") pod \"maas-api-e2e-api-a-091828-df7f748f7-8j742\" (UID: \"1add3517-ad57-4bb6-89af-2c07bd572329\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-091828-df7f748f7-8j742" Jul 23 12:15:05.020378 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:05.020338 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2fbr\" (UniqueName: \"kubernetes.io/projected/1add3517-ad57-4bb6-89af-2c07bd572329-kube-api-access-w2fbr\") pod \"maas-api-e2e-api-a-091828-df7f748f7-8j742\" (UID: \"1add3517-ad57-4bb6-89af-2c07bd572329\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-091828-df7f748f7-8j742" Jul 23 12:15:05.020581 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:05.020399 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1add3517-ad57-4bb6-89af-2c07bd572329-openshift-service-ca\") pod \"maas-api-e2e-api-a-091828-df7f748f7-8j742\" (UID: \"1add3517-ad57-4bb6-89af-2c07bd572329\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-091828-df7f748f7-8j742" Jul 23 12:15:05.020581 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:05.020441 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1add3517-ad57-4bb6-89af-2c07bd572329-maas-api-tls\") pod \"maas-api-e2e-api-a-091828-df7f748f7-8j742\" (UID: \"1add3517-ad57-4bb6-89af-2c07bd572329\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-091828-df7f748f7-8j742" Jul 23 12:15:05.021113 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:05.021093 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1add3517-ad57-4bb6-89af-2c07bd572329-openshift-service-ca\") pod \"maas-api-e2e-api-a-091828-df7f748f7-8j742\" (UID: \"1add3517-ad57-4bb6-89af-2c07bd572329\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-091828-df7f748f7-8j742" Jul 23 12:15:05.023113 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:05.023090 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1add3517-ad57-4bb6-89af-2c07bd572329-maas-api-tls\") pod \"maas-api-e2e-api-a-091828-df7f748f7-8j742\" (UID: \"1add3517-ad57-4bb6-89af-2c07bd572329\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-091828-df7f748f7-8j742" Jul 23 12:15:05.055134 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:05.055091 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2fbr\" (UniqueName: \"kubernetes.io/projected/1add3517-ad57-4bb6-89af-2c07bd572329-kube-api-access-w2fbr\") pod \"maas-api-e2e-api-a-091828-df7f748f7-8j742\" (UID: \"1add3517-ad57-4bb6-89af-2c07bd572329\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-091828-df7f748f7-8j742" Jul 23 12:15:05.157058 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:05.156948 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-091828-df7f748f7-8j742" Jul 23 12:15:05.340564 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:05.340527 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-091828-df7f748f7-8j742"] Jul 23 12:15:05.342194 ip-10-0-134-236 kubenswrapper[2561]: W0723 12:15:05.342166 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1add3517_ad57_4bb6_89af_2c07bd572329.slice/crio-34e7de4f74807cf9b654f8080b06c76992965aa925fea6b0f80ce54c7d6f55fa WatchSource:0}: Error finding container 34e7de4f74807cf9b654f8080b06c76992965aa925fea6b0f80ce54c7d6f55fa: Status 404 returned error can't find the container with id 34e7de4f74807cf9b654f8080b06c76992965aa925fea6b0f80ce54c7d6f55fa Jul 23 12:15:05.732628 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:05.732595 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-091828-df7f748f7-8j742" event={"ID":"1add3517-ad57-4bb6-89af-2c07bd572329","Type":"ContainerStarted","Data":"34e7de4f74807cf9b654f8080b06c76992965aa925fea6b0f80ce54c7d6f55fa"} Jul 23 12:15:06.740244 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:06.740204 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-091828-df7f748f7-8j742" event={"ID":"1add3517-ad57-4bb6-89af-2c07bd572329","Type":"ContainerStarted","Data":"b63a9c67cb16539d3e7d1208ff23efd381fb797ca24b1596e4cc790603f4e460"} Jul 23 12:15:06.740757 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:06.740319 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-091828-df7f748f7-8j742" Jul 23 12:15:06.797463 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:06.797405 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-091828-df7f748f7-8j742" podStartSLOduration=2.1045825479999998 podStartE2EDuration="2.797387253s" totalImagesPullingTime="692.804705ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 12:15:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 12:15:06.795364187 +0000 UTC m=+2373.867054179" watchObservedRunningTime="2026-07-23 12:15:06.797387253 +0000 UTC m=+2373.869077243" Jul 23 12:15:12.217191 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:12.217150 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 12:15:12.750540 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:12.750512 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-091828-df7f748f7-8j742" Jul 23 12:15:14.021160 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:14.021119 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c21064-f78b8c9d8-b2xbh"] Jul 23 12:15:14.024719 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:14.024692 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c21064-f78b8c9d8-b2xbh" Jul 23 12:15:14.028700 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:14.028675 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-c21064-serving-cert\"" Jul 23 12:15:14.044934 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:14.044876 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c21064-f78b8c9d8-b2xbh"] Jul 23 12:15:14.094565 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:14.094512 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhnr8\" (UniqueName: \"kubernetes.io/projected/43e3f4b3-7799-4e2f-a24b-bada7975f114-kube-api-access-jhnr8\") pod \"maas-api-e2e-api-b-c21064-f78b8c9d8-b2xbh\" (UID: \"43e3f4b3-7799-4e2f-a24b-bada7975f114\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c21064-f78b8c9d8-b2xbh" Jul 23 12:15:14.094810 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:14.094645 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43e3f4b3-7799-4e2f-a24b-bada7975f114-maas-api-tls\") pod \"maas-api-e2e-api-b-c21064-f78b8c9d8-b2xbh\" (UID: \"43e3f4b3-7799-4e2f-a24b-bada7975f114\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c21064-f78b8c9d8-b2xbh" Jul 23 12:15:14.094810 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:14.094675 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43e3f4b3-7799-4e2f-a24b-bada7975f114-openshift-service-ca\") pod \"maas-api-e2e-api-b-c21064-f78b8c9d8-b2xbh\" (UID: \"43e3f4b3-7799-4e2f-a24b-bada7975f114\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c21064-f78b8c9d8-b2xbh" Jul 23 12:15:14.195729 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:14.195696 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43e3f4b3-7799-4e2f-a24b-bada7975f114-maas-api-tls\") pod \"maas-api-e2e-api-b-c21064-f78b8c9d8-b2xbh\" (UID: \"43e3f4b3-7799-4e2f-a24b-bada7975f114\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c21064-f78b8c9d8-b2xbh" Jul 23 12:15:14.195729 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:14.195735 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43e3f4b3-7799-4e2f-a24b-bada7975f114-openshift-service-ca\") pod \"maas-api-e2e-api-b-c21064-f78b8c9d8-b2xbh\" (UID: \"43e3f4b3-7799-4e2f-a24b-bada7975f114\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c21064-f78b8c9d8-b2xbh" Jul 23 12:15:14.196051 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:14.195807 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhnr8\" (UniqueName: \"kubernetes.io/projected/43e3f4b3-7799-4e2f-a24b-bada7975f114-kube-api-access-jhnr8\") pod \"maas-api-e2e-api-b-c21064-f78b8c9d8-b2xbh\" (UID: \"43e3f4b3-7799-4e2f-a24b-bada7975f114\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c21064-f78b8c9d8-b2xbh" Jul 23 12:15:14.196558 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:14.196535 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43e3f4b3-7799-4e2f-a24b-bada7975f114-openshift-service-ca\") pod \"maas-api-e2e-api-b-c21064-f78b8c9d8-b2xbh\" (UID: \"43e3f4b3-7799-4e2f-a24b-bada7975f114\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c21064-f78b8c9d8-b2xbh" Jul 23 12:15:14.198529 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:14.198505 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43e3f4b3-7799-4e2f-a24b-bada7975f114-maas-api-tls\") pod \"maas-api-e2e-api-b-c21064-f78b8c9d8-b2xbh\" (UID: \"43e3f4b3-7799-4e2f-a24b-bada7975f114\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c21064-f78b8c9d8-b2xbh" Jul 23 12:15:14.217770 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:14.217736 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhnr8\" (UniqueName: \"kubernetes.io/projected/43e3f4b3-7799-4e2f-a24b-bada7975f114-kube-api-access-jhnr8\") pod \"maas-api-e2e-api-b-c21064-f78b8c9d8-b2xbh\" (UID: \"43e3f4b3-7799-4e2f-a24b-bada7975f114\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c21064-f78b8c9d8-b2xbh" Jul 23 12:15:14.336213 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:14.336169 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c21064-f78b8c9d8-b2xbh" Jul 23 12:15:14.543682 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:14.543563 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c21064-f78b8c9d8-b2xbh"] Jul 23 12:15:14.546861 ip-10-0-134-236 kubenswrapper[2561]: W0723 12:15:14.546827 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43e3f4b3_7799_4e2f_a24b_bada7975f114.slice/crio-e9fd715e6912201f1682e773273eb068ed147397ab50f40d651080da62d27f91 WatchSource:0}: Error finding container e9fd715e6912201f1682e773273eb068ed147397ab50f40d651080da62d27f91: Status 404 returned error can't find the container with id e9fd715e6912201f1682e773273eb068ed147397ab50f40d651080da62d27f91 Jul 23 12:15:14.788504 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:14.788413 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c21064-f78b8c9d8-b2xbh" event={"ID":"43e3f4b3-7799-4e2f-a24b-bada7975f114","Type":"ContainerStarted","Data":"e9fd715e6912201f1682e773273eb068ed147397ab50f40d651080da62d27f91"} Jul 23 12:15:15.795779 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:15.795740 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c21064-f78b8c9d8-b2xbh" event={"ID":"43e3f4b3-7799-4e2f-a24b-bada7975f114","Type":"ContainerStarted","Data":"a648bc66af8453689419bba3cef3f0cfa9220998c6063a53e6b3ae9d94cc49c3"} Jul 23 12:15:15.796250 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:15.795873 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c21064-f78b8c9d8-b2xbh" Jul 23 12:15:15.827651 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:15.827596 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c21064-f78b8c9d8-b2xbh" podStartSLOduration=2.525302915 podStartE2EDuration="2.827577608s" totalImagesPullingTime="302.274693ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 12:15:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 12:15:15.826683393 +0000 UTC m=+2382.898373414" watchObservedRunningTime="2026-07-23 12:15:15.827577608 +0000 UTC m=+2382.899267599" Jul 23 12:15:21.807900 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:21.807865 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c21064-f78b8c9d8-b2xbh" Jul 23 12:15:26.574563 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:26.574525 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-091828-kls2j"] Jul 23 12:15:26.578086 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:26.578063 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-091828-kls2j" Jul 23 12:15:26.580930 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:26.580891 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q8m5p\"" Jul 23 12:15:26.592142 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:26.592107 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-091828-kls2j"] Jul 23 12:15:26.711632 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:26.711584 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/effa348e-6dff-43b1-a5e8-02d1b4ea3fc4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-091828-kls2j\" (UID: \"effa348e-6dff-43b1-a5e8-02d1b4ea3fc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-091828-kls2j" Jul 23 12:15:26.812545 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:26.812501 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/effa348e-6dff-43b1-a5e8-02d1b4ea3fc4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-091828-kls2j\" (UID: \"effa348e-6dff-43b1-a5e8-02d1b4ea3fc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-091828-kls2j" Jul 23 12:15:26.813165 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:26.813143 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/effa348e-6dff-43b1-a5e8-02d1b4ea3fc4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-091828-kls2j\" (UID: \"effa348e-6dff-43b1-a5e8-02d1b4ea3fc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-091828-kls2j" Jul 23 12:15:26.891268 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:26.891166 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-091828-kls2j" Jul 23 12:15:27.049119 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:27.049081 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-091828-kls2j"] Jul 23 12:15:27.052547 ip-10-0-134-236 kubenswrapper[2561]: W0723 12:15:27.052517 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeffa348e_6dff_43b1_a5e8_02d1b4ea3fc4.slice/crio-67f221e52629ceacad1d0582644251f37c2b0e7c0c1dd575e905966369aab637 WatchSource:0}: Error finding container 67f221e52629ceacad1d0582644251f37c2b0e7c0c1dd575e905966369aab637: Status 404 returned error can't find the container with id 67f221e52629ceacad1d0582644251f37c2b0e7c0c1dd575e905966369aab637 Jul 23 12:15:27.865778 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:27.865728 2561 generic.go:356] "Generic (PLEG): container finished" podID="effa348e-6dff-43b1-a5e8-02d1b4ea3fc4" containerID="b0817567f45f22c9a3ee4fe4343553c2282de23e1a431ee6ffa3cf2846bb4ee2" exitCode=0 Jul 23 12:15:27.866269 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:27.865821 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-091828-kls2j" event={"ID":"effa348e-6dff-43b1-a5e8-02d1b4ea3fc4","Type":"ContainerDied","Data":"b0817567f45f22c9a3ee4fe4343553c2282de23e1a431ee6ffa3cf2846bb4ee2"} Jul 23 12:15:27.866269 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:27.865890 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-091828-kls2j" event={"ID":"effa348e-6dff-43b1-a5e8-02d1b4ea3fc4","Type":"ContainerStarted","Data":"67f221e52629ceacad1d0582644251f37c2b0e7c0c1dd575e905966369aab637"} Jul 23 12:15:29.012549 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:29.012519 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-091828-kls2j" Jul 23 12:15:29.133034 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:29.132900 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/effa348e-6dff-43b1-a5e8-02d1b4ea3fc4-maas-api-service-ca\") pod \"effa348e-6dff-43b1-a5e8-02d1b4ea3fc4\" (UID: \"effa348e-6dff-43b1-a5e8-02d1b4ea3fc4\") " Jul 23 12:15:29.133248 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:29.133224 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/effa348e-6dff-43b1-a5e8-02d1b4ea3fc4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "effa348e-6dff-43b1-a5e8-02d1b4ea3fc4" (UID: "effa348e-6dff-43b1-a5e8-02d1b4ea3fc4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 12:15:29.234395 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:29.234351 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/effa348e-6dff-43b1-a5e8-02d1b4ea3fc4-maas-api-service-ca\") on node \"ip-10-0-134-236.ec2.internal\" DevicePath \"\"" Jul 23 12:15:29.878049 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:29.878015 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-091828-kls2j" Jul 23 12:15:29.878229 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:29.878018 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-091828-kls2j" event={"ID":"effa348e-6dff-43b1-a5e8-02d1b4ea3fc4","Type":"ContainerDied","Data":"67f221e52629ceacad1d0582644251f37c2b0e7c0c1dd575e905966369aab637"} Jul 23 12:15:29.878229 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:29.878143 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67f221e52629ceacad1d0582644251f37c2b0e7c0c1dd575e905966369aab637" Jul 23 12:15:30.853340 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:30.853302 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 12:15:33.553818 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:33.553790 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-7slpj_65247e1f-fc10-4b40-af67-5e646f352263/console-operator/1.log" Jul 23 12:15:33.557231 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:33.557203 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-7slpj_65247e1f-fc10-4b40-af67-5e646f352263/console-operator/1.log" Jul 23 12:15:33.639287 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:33.639256 2561 scope.go:117] "RemoveContainer" containerID="9df8b216b8550ce15a18ffc6aa08adc09d327a2de232972e67f2d7fc9104d3fb" Jul 23 12:15:42.039267 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:42.039231 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-091828-df7f748f7-8j742"] Jul 23 12:15:42.039803 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:42.039493 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-091828-df7f748f7-8j742" podUID="1add3517-ad57-4bb6-89af-2c07bd572329" containerName="maas-api" containerID="cri-o://b63a9c67cb16539d3e7d1208ff23efd381fb797ca24b1596e4cc790603f4e460" gracePeriod=30 Jul 23 12:15:42.292513 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:42.292450 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-091828-df7f748f7-8j742" Jul 23 12:15:42.460839 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:42.460806 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1add3517-ad57-4bb6-89af-2c07bd572329-maas-api-tls\") pod \"1add3517-ad57-4bb6-89af-2c07bd572329\" (UID: \"1add3517-ad57-4bb6-89af-2c07bd572329\") " Jul 23 12:15:42.461069 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:42.460858 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w2fbr\" (UniqueName: \"kubernetes.io/projected/1add3517-ad57-4bb6-89af-2c07bd572329-kube-api-access-w2fbr\") pod \"1add3517-ad57-4bb6-89af-2c07bd572329\" (UID: \"1add3517-ad57-4bb6-89af-2c07bd572329\") " Jul 23 12:15:42.461069 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:42.461024 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1add3517-ad57-4bb6-89af-2c07bd572329-openshift-service-ca\") pod \"1add3517-ad57-4bb6-89af-2c07bd572329\" (UID: \"1add3517-ad57-4bb6-89af-2c07bd572329\") " Jul 23 12:15:42.461336 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:42.461310 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1add3517-ad57-4bb6-89af-2c07bd572329-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1add3517-ad57-4bb6-89af-2c07bd572329" (UID: "1add3517-ad57-4bb6-89af-2c07bd572329"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 12:15:42.463105 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:42.463083 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1add3517-ad57-4bb6-89af-2c07bd572329-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1add3517-ad57-4bb6-89af-2c07bd572329" (UID: "1add3517-ad57-4bb6-89af-2c07bd572329"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 12:15:42.463191 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:42.463178 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1add3517-ad57-4bb6-89af-2c07bd572329-kube-api-access-w2fbr" (OuterVolumeSpecName: "kube-api-access-w2fbr") pod "1add3517-ad57-4bb6-89af-2c07bd572329" (UID: "1add3517-ad57-4bb6-89af-2c07bd572329"). InnerVolumeSpecName "kube-api-access-w2fbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 12:15:42.562549 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:42.562516 2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1add3517-ad57-4bb6-89af-2c07bd572329-openshift-service-ca\") on node \"ip-10-0-134-236.ec2.internal\" DevicePath \"\"" Jul 23 12:15:42.562549 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:42.562550 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1add3517-ad57-4bb6-89af-2c07bd572329-maas-api-tls\") on node \"ip-10-0-134-236.ec2.internal\" DevicePath \"\"" Jul 23 12:15:42.562750 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:42.562562 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2fbr\" (UniqueName: \"kubernetes.io/projected/1add3517-ad57-4bb6-89af-2c07bd572329-kube-api-access-w2fbr\") on node \"ip-10-0-134-236.ec2.internal\" DevicePath \"\"" Jul 23 12:15:42.948781 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:42.948677 2561 generic.go:356] "Generic (PLEG): container finished" podID="1add3517-ad57-4bb6-89af-2c07bd572329" containerID="b63a9c67cb16539d3e7d1208ff23efd381fb797ca24b1596e4cc790603f4e460" exitCode=0 Jul 23 12:15:42.948781 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:42.948726 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-091828-df7f748f7-8j742" Jul 23 12:15:42.948781 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:42.948767 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-091828-df7f748f7-8j742" event={"ID":"1add3517-ad57-4bb6-89af-2c07bd572329","Type":"ContainerDied","Data":"b63a9c67cb16539d3e7d1208ff23efd381fb797ca24b1596e4cc790603f4e460"} Jul 23 12:15:42.949094 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:42.948823 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-091828-df7f748f7-8j742" event={"ID":"1add3517-ad57-4bb6-89af-2c07bd572329","Type":"ContainerDied","Data":"34e7de4f74807cf9b654f8080b06c76992965aa925fea6b0f80ce54c7d6f55fa"} Jul 23 12:15:42.949094 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:42.948849 2561 scope.go:117] "RemoveContainer" containerID="b63a9c67cb16539d3e7d1208ff23efd381fb797ca24b1596e4cc790603f4e460" Jul 23 12:15:42.960624 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:42.960605 2561 scope.go:117] "RemoveContainer" containerID="b63a9c67cb16539d3e7d1208ff23efd381fb797ca24b1596e4cc790603f4e460" Jul 23 12:15:42.960932 ip-10-0-134-236 kubenswrapper[2561]: E0723 12:15:42.960895 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b63a9c67cb16539d3e7d1208ff23efd381fb797ca24b1596e4cc790603f4e460\": container with ID starting with b63a9c67cb16539d3e7d1208ff23efd381fb797ca24b1596e4cc790603f4e460 not found: ID does not exist" containerID="b63a9c67cb16539d3e7d1208ff23efd381fb797ca24b1596e4cc790603f4e460" Jul 23 12:15:42.960997 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:42.960941 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b63a9c67cb16539d3e7d1208ff23efd381fb797ca24b1596e4cc790603f4e460"} err="failed to get container status \"b63a9c67cb16539d3e7d1208ff23efd381fb797ca24b1596e4cc790603f4e460\": rpc error: code = NotFound desc = could not find container \"b63a9c67cb16539d3e7d1208ff23efd381fb797ca24b1596e4cc790603f4e460\": container with ID starting with b63a9c67cb16539d3e7d1208ff23efd381fb797ca24b1596e4cc790603f4e460 not found: ID does not exist" Jul 23 12:15:42.973936 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:42.973894 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-091828-df7f748f7-8j742"] Jul 23 12:15:42.978588 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:42.978564 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-091828-df7f748f7-8j742"] Jul 23 12:15:43.455761 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:43.455719 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1add3517-ad57-4bb6-89af-2c07bd572329" path="/var/lib/kubelet/pods/1add3517-ad57-4bb6-89af-2c07bd572329/volumes" Jul 23 12:15:46.672610 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:46.672566 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-091828-kls2j"] Jul 23 12:15:46.674923 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:46.674886 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-091828-kls2j"] Jul 23 12:15:47.456997 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:47.456943 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="effa348e-6dff-43b1-a5e8-02d1b4ea3fc4" path="/var/lib/kubelet/pods/effa348e-6dff-43b1-a5e8-02d1b4ea3fc4/volumes" Jul 23 12:15:47.547187 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:47.547151 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 12:15:53.873786 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:53.873703 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c21064-4wzv5"] Jul 23 12:15:53.874181 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:53.874102 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1add3517-ad57-4bb6-89af-2c07bd572329" containerName="maas-api" Jul 23 12:15:53.874181 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:53.874117 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1add3517-ad57-4bb6-89af-2c07bd572329" containerName="maas-api" Jul 23 12:15:53.874181 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:53.874127 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="effa348e-6dff-43b1-a5e8-02d1b4ea3fc4" containerName="revoke-keys" Jul 23 12:15:53.874181 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:53.874133 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="effa348e-6dff-43b1-a5e8-02d1b4ea3fc4" containerName="revoke-keys" Jul 23 12:15:53.874314 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:53.874200 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1add3517-ad57-4bb6-89af-2c07bd572329" containerName="maas-api" Jul 23 12:15:53.874314 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:53.874208 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="effa348e-6dff-43b1-a5e8-02d1b4ea3fc4" containerName="revoke-keys" Jul 23 12:15:53.878855 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:53.878826 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c21064-4wzv5" Jul 23 12:15:53.884340 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:53.884317 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q8m5p\"" Jul 23 12:15:53.907134 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:53.907106 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c21064-4wzv5"] Jul 23 12:15:53.956177 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:53.956137 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c586b652-a72c-44ed-a3ac-923cc1e06aec-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c21064-4wzv5\" (UID: \"c586b652-a72c-44ed-a3ac-923cc1e06aec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c21064-4wzv5" Jul 23 12:15:54.057405 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:54.057370 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c586b652-a72c-44ed-a3ac-923cc1e06aec-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c21064-4wzv5\" (UID: \"c586b652-a72c-44ed-a3ac-923cc1e06aec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c21064-4wzv5" Jul 23 12:15:54.058034 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:54.058013 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c586b652-a72c-44ed-a3ac-923cc1e06aec-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c21064-4wzv5\" (UID: \"c586b652-a72c-44ed-a3ac-923cc1e06aec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c21064-4wzv5" Jul 23 12:15:54.188856 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:54.188766 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c21064-4wzv5" Jul 23 12:15:54.334561 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:54.334529 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c21064-4wzv5"] Jul 23 12:15:54.336310 ip-10-0-134-236 kubenswrapper[2561]: W0723 12:15:54.336277 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc586b652_a72c_44ed_a3ac_923cc1e06aec.slice/crio-83abf3586903a623afd9b3ecc26c0cc8ffdf52fdaa86559fe6852d4ad2493510 WatchSource:0}: Error finding container 83abf3586903a623afd9b3ecc26c0cc8ffdf52fdaa86559fe6852d4ad2493510: Status 404 returned error can't find the container with id 83abf3586903a623afd9b3ecc26c0cc8ffdf52fdaa86559fe6852d4ad2493510 Jul 23 12:15:55.013229 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:55.013186 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c21064-4wzv5" event={"ID":"c586b652-a72c-44ed-a3ac-923cc1e06aec","Type":"ContainerStarted","Data":"d2152d21d4008bf61c4eef5f7fb0d44417a54d852eab22c3452ece5de9083004"} Jul 23 12:15:55.013676 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:55.013243 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c21064-4wzv5" event={"ID":"c586b652-a72c-44ed-a3ac-923cc1e06aec","Type":"ContainerStarted","Data":"83abf3586903a623afd9b3ecc26c0cc8ffdf52fdaa86559fe6852d4ad2493510"} Jul 23 12:15:55.031469 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:55.031425 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c21064-4wzv5" podStartSLOduration=2.031408071 podStartE2EDuration="2.031408071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 12:15:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 12:15:55.028331362 +0000 UTC m=+2422.100021353" watchObservedRunningTime="2026-07-23 12:15:55.031408071 +0000 UTC m=+2422.103098086" Jul 23 12:15:56.020628 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:56.020590 2561 generic.go:356] "Generic (PLEG): container finished" podID="c586b652-a72c-44ed-a3ac-923cc1e06aec" containerID="d2152d21d4008bf61c4eef5f7fb0d44417a54d852eab22c3452ece5de9083004" exitCode=0 Jul 23 12:15:56.021055 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:56.020682 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c21064-4wzv5" event={"ID":"c586b652-a72c-44ed-a3ac-923cc1e06aec","Type":"ContainerDied","Data":"d2152d21d4008bf61c4eef5f7fb0d44417a54d852eab22c3452ece5de9083004"} Jul 23 12:15:57.160768 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:57.160743 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c21064-4wzv5" Jul 23 12:15:57.182508 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:57.182476 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c586b652-a72c-44ed-a3ac-923cc1e06aec-maas-api-service-ca\") pod \"c586b652-a72c-44ed-a3ac-923cc1e06aec\" (UID: \"c586b652-a72c-44ed-a3ac-923cc1e06aec\") " Jul 23 12:15:57.182859 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:57.182832 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c586b652-a72c-44ed-a3ac-923cc1e06aec-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c586b652-a72c-44ed-a3ac-923cc1e06aec" (UID: "c586b652-a72c-44ed-a3ac-923cc1e06aec"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 12:15:57.283725 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:57.283629 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c586b652-a72c-44ed-a3ac-923cc1e06aec-maas-api-service-ca\") on node \"ip-10-0-134-236.ec2.internal\" DevicePath \"\"" Jul 23 12:15:58.033589 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:58.033555 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c21064-4wzv5" Jul 23 12:15:58.033772 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:58.033592 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c21064-4wzv5" event={"ID":"c586b652-a72c-44ed-a3ac-923cc1e06aec","Type":"ContainerDied","Data":"83abf3586903a623afd9b3ecc26c0cc8ffdf52fdaa86559fe6852d4ad2493510"} Jul 23 12:15:58.033772 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:15:58.033641 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83abf3586903a623afd9b3ecc26c0cc8ffdf52fdaa86559fe6852d4ad2493510" Jul 23 12:16:09.540731 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:16:09.540697 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c21064-f78b8c9d8-b2xbh"] Jul 23 12:16:09.541329 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:16:09.540966 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c21064-f78b8c9d8-b2xbh" podUID="43e3f4b3-7799-4e2f-a24b-bada7975f114" containerName="maas-api" containerID="cri-o://a648bc66af8453689419bba3cef3f0cfa9220998c6063a53e6b3ae9d94cc49c3" gracePeriod=30 Jul 23 12:16:09.806258 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:16:09.806216 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c21064-f78b8c9d8-b2xbh" Jul 23 12:16:09.898019 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:16:09.897978 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43e3f4b3-7799-4e2f-a24b-bada7975f114-maas-api-tls\") pod \"43e3f4b3-7799-4e2f-a24b-bada7975f114\" (UID: \"43e3f4b3-7799-4e2f-a24b-bada7975f114\") " Jul 23 12:16:09.898207 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:16:09.898080 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jhnr8\" (UniqueName: \"kubernetes.io/projected/43e3f4b3-7799-4e2f-a24b-bada7975f114-kube-api-access-jhnr8\") pod \"43e3f4b3-7799-4e2f-a24b-bada7975f114\" (UID: \"43e3f4b3-7799-4e2f-a24b-bada7975f114\") " Jul 23 12:16:09.898207 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:16:09.898117 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43e3f4b3-7799-4e2f-a24b-bada7975f114-openshift-service-ca\") pod \"43e3f4b3-7799-4e2f-a24b-bada7975f114\" (UID: \"43e3f4b3-7799-4e2f-a24b-bada7975f114\") " Jul 23 12:16:09.898571 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:16:09.898541 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43e3f4b3-7799-4e2f-a24b-bada7975f114-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "43e3f4b3-7799-4e2f-a24b-bada7975f114" (UID: "43e3f4b3-7799-4e2f-a24b-bada7975f114"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 12:16:09.900364 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:16:09.900334 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43e3f4b3-7799-4e2f-a24b-bada7975f114-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "43e3f4b3-7799-4e2f-a24b-bada7975f114" (UID: "43e3f4b3-7799-4e2f-a24b-bada7975f114"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 12:16:09.900364 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:16:09.900352 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43e3f4b3-7799-4e2f-a24b-bada7975f114-kube-api-access-jhnr8" (OuterVolumeSpecName: "kube-api-access-jhnr8") pod "43e3f4b3-7799-4e2f-a24b-bada7975f114" (UID: "43e3f4b3-7799-4e2f-a24b-bada7975f114"). InnerVolumeSpecName "kube-api-access-jhnr8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 12:16:09.999184 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:16:09.999141 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43e3f4b3-7799-4e2f-a24b-bada7975f114-maas-api-tls\") on node \"ip-10-0-134-236.ec2.internal\" DevicePath \"\"" Jul 23 12:16:09.999184 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:16:09.999174 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhnr8\" (UniqueName: \"kubernetes.io/projected/43e3f4b3-7799-4e2f-a24b-bada7975f114-kube-api-access-jhnr8\") on node \"ip-10-0-134-236.ec2.internal\" DevicePath \"\"" Jul 23 12:16:09.999184 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:16:09.999185 2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43e3f4b3-7799-4e2f-a24b-bada7975f114-openshift-service-ca\") on node \"ip-10-0-134-236.ec2.internal\" DevicePath \"\"" Jul 23 12:16:10.097321 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:16:10.097284 2561 generic.go:356] "Generic (PLEG): container finished" podID="43e3f4b3-7799-4e2f-a24b-bada7975f114" containerID="a648bc66af8453689419bba3cef3f0cfa9220998c6063a53e6b3ae9d94cc49c3" exitCode=0 Jul 23 12:16:10.097490 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:16:10.097333 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c21064-f78b8c9d8-b2xbh" Jul 23 12:16:10.097490 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:16:10.097375 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c21064-f78b8c9d8-b2xbh" event={"ID":"43e3f4b3-7799-4e2f-a24b-bada7975f114","Type":"ContainerDied","Data":"a648bc66af8453689419bba3cef3f0cfa9220998c6063a53e6b3ae9d94cc49c3"} Jul 23 12:16:10.097490 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:16:10.097454 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c21064-f78b8c9d8-b2xbh" event={"ID":"43e3f4b3-7799-4e2f-a24b-bada7975f114","Type":"ContainerDied","Data":"e9fd715e6912201f1682e773273eb068ed147397ab50f40d651080da62d27f91"} Jul 23 12:16:10.097490 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:16:10.097478 2561 scope.go:117] "RemoveContainer" containerID="a648bc66af8453689419bba3cef3f0cfa9220998c6063a53e6b3ae9d94cc49c3" Jul 23 12:16:10.107773 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:16:10.107746 2561 scope.go:117] "RemoveContainer" containerID="a648bc66af8453689419bba3cef3f0cfa9220998c6063a53e6b3ae9d94cc49c3" Jul 23 12:16:10.108037 ip-10-0-134-236 kubenswrapper[2561]: E0723 12:16:10.108020 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a648bc66af8453689419bba3cef3f0cfa9220998c6063a53e6b3ae9d94cc49c3\": container with ID starting with a648bc66af8453689419bba3cef3f0cfa9220998c6063a53e6b3ae9d94cc49c3 not found: ID does not exist" containerID="a648bc66af8453689419bba3cef3f0cfa9220998c6063a53e6b3ae9d94cc49c3" Jul 23 12:16:10.108111 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:16:10.108050 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a648bc66af8453689419bba3cef3f0cfa9220998c6063a53e6b3ae9d94cc49c3"} err="failed to get container status \"a648bc66af8453689419bba3cef3f0cfa9220998c6063a53e6b3ae9d94cc49c3\": rpc error: code = NotFound desc = could not find container \"a648bc66af8453689419bba3cef3f0cfa9220998c6063a53e6b3ae9d94cc49c3\": container with ID starting with a648bc66af8453689419bba3cef3f0cfa9220998c6063a53e6b3ae9d94cc49c3 not found: ID does not exist" Jul 23 12:16:10.120028 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:16:10.119903 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c21064-f78b8c9d8-b2xbh"] Jul 23 12:16:10.124031 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:16:10.124002 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c21064-f78b8c9d8-b2xbh"] Jul 23 12:16:11.454476 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:16:11.454439 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43e3f4b3-7799-4e2f-a24b-bada7975f114" path="/var/lib/kubelet/pods/43e3f4b3-7799-4e2f-a24b-bada7975f114/volumes" Jul 23 12:16:14.026656 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:16:14.026622 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c21064-4wzv5"] Jul 23 12:16:14.030100 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:16:14.030075 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c21064-4wzv5"] Jul 23 12:16:14.847738 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:16:14.847701 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 12:16:15.461176 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:16:15.461137 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c586b652-a72c-44ed-a3ac-923cc1e06aec" path="/var/lib/kubelet/pods/c586b652-a72c-44ed-a3ac-923cc1e06aec/volumes" Jul 23 12:16:25.157273 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:16:25.157231 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-290648-c5686d489-4c5pj"] Jul 23 12:16:25.157848 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:16:25.157702 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c586b652-a72c-44ed-a3ac-923cc1e06aec" containerName="revoke-keys" Jul 23 12:16:25.157848 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:16:25.157722 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c586b652-a72c-44ed-a3ac-923cc1e06aec" containerName="revoke-keys" Jul 23 12:16:25.157848 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:16:25.157753 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43e3f4b3-7799-4e2f-a24b-bada7975f114" containerName="maas-api" Jul 23 12:16:25.157848 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:16:25.157762 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="43e3f4b3-7799-4e2f-a24b-bada7975f114" containerName="maas-api" Jul 23 12:16:25.158096 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:16:25.157850 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="c586b652-a72c-44ed-a3ac-923cc1e06aec" containerName="revoke-keys" Jul 23 12:16:25.158096 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:16:25.157863 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="43e3f4b3-7799-4e2f-a24b-bada7975f114" containerName="maas-api" Jul 23 12:16:25.161386 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:16:25.161364 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-290648-c5686d489-4c5pj" Jul 23 12:16:25.164731 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:16:25.164705 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-290648-serving-cert\"" Jul 23 12:16:25.164866 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:16:25.164778 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-c9lpl\"" Jul 23 12:16:25.165132 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:16:25.165117 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 23 12:16:25.165779 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:16:25.165763 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 23 12:16:25.174623 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:16:25.174600 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-290648-c5686d489-4c5pj"] Jul 23 12:16:25.234757 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:16:25.234720 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8tb8\" (UniqueName: \"kubernetes.io/projected/e49f098b-169d-45f5-82b5-ded09a4eb10c-kube-api-access-h8tb8\") pod \"maas-api-e2e-shared-a-290648-c5686d489-4c5pj\" (UID: \"e49f098b-169d-45f5-82b5-ded09a4eb10c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-290648-c5686d489-4c5pj" Jul 23 12:16:25.235025 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:16:25.234787 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e49f098b-169d-45f5-82b5-ded09a4eb10c-maas-api-tls\") pod \"maas-api-e2e-shared-a-290648-c5686d489-4c5pj\" (UID: \"e49f098b-169d-45f5-82b5-ded09a4eb10c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-290648-c5686d489-4c5pj" Jul 23 12:16:25.235025 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:16:25.234815 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e49f098b-169d-45f5-82b5-ded09a4eb10c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-290648-c5686d489-4c5pj\" (UID: \"e49f098b-169d-45f5-82b5-ded09a4eb10c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-290648-c5686d489-4c5pj" Jul 23 12:16:25.336291 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:16:25.336250 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e49f098b-169d-45f5-82b5-ded09a4eb10c-maas-api-tls\") pod \"maas-api-e2e-shared-a-290648-c5686d489-4c5pj\" (UID: \"e49f098b-169d-45f5-82b5-ded09a4eb10c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-290648-c5686d489-4c5pj" Jul 23 12:16:25.336485 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:16:25.336301 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e49f098b-169d-45f5-82b5-ded09a4eb10c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-290648-c5686d489-4c5pj\" (UID: \"e49f098b-169d-45f5-82b5-ded09a4eb10c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-290648-c5686d489-4c5pj" Jul 23 12:16:25.336485 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:16:25.336341 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8tb8\" (UniqueName: \"kubernetes.io/projected/e49f098b-169d-45f5-82b5-ded09a4eb10c-kube-api-access-h8tb8\") pod \"maas-api-e2e-shared-a-290648-c5686d489-4c5pj\" (UID: \"e49f098b-169d-45f5-82b5-ded09a4eb10c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-290648-c5686d489-4c5pj" Jul 23 12:16:25.337021 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:16:25.337000 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e49f098b-169d-45f5-82b5-ded09a4eb10c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-290648-c5686d489-4c5pj\" (UID: \"e49f098b-169d-45f5-82b5-ded09a4eb10c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-290648-c5686d489-4c5pj" Jul 23 12:16:25.338807 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:16:25.338786 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e49f098b-169d-45f5-82b5-ded09a4eb10c-maas-api-tls\") pod \"maas-api-e2e-shared-a-290648-c5686d489-4c5pj\" (UID: \"e49f098b-169d-45f5-82b5-ded09a4eb10c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-290648-c5686d489-4c5pj" Jul 23 12:16:25.347042 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:16:25.347014 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8tb8\" (UniqueName: \"kubernetes.io/projected/e49f098b-169d-45f5-82b5-ded09a4eb10c-kube-api-access-h8tb8\") pod \"maas-api-e2e-shared-a-290648-c5686d489-4c5pj\" (UID: \"e49f098b-169d-45f5-82b5-ded09a4eb10c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-290648-c5686d489-4c5pj" Jul 23 12:16:25.472374 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:16:25.472302 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-290648-c5686d489-4c5pj" Jul 23 12:16:25.613712 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:16:25.613479 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-290648-c5686d489-4c5pj"] Jul 23 12:16:25.616265 ip-10-0-134-236 kubenswrapper[2561]: W0723 12:16:25.616241 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode49f098b_169d_45f5_82b5_ded09a4eb10c.slice/crio-ee816dbf515c862e189be2588b9b1511829fdb991b414fb9ac2d3edecfe20f0b WatchSource:0}: Error finding container ee816dbf515c862e189be2588b9b1511829fdb991b414fb9ac2d3edecfe20f0b: Status 404 returned error can't find the container with id ee816dbf515c862e189be2588b9b1511829fdb991b414fb9ac2d3edecfe20f0b Jul 23 12:16:26.187199 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:16:26.187161 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-290648-c5686d489-4c5pj" event={"ID":"e49f098b-169d-45f5-82b5-ded09a4eb10c","Type":"ContainerStarted","Data":"ee816dbf515c862e189be2588b9b1511829fdb991b414fb9ac2d3edecfe20f0b"} Jul 23 12:16:27.193627 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:16:27.193596 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-290648-c5686d489-4c5pj" event={"ID":"e49f098b-169d-45f5-82b5-ded09a4eb10c","Type":"ContainerStarted","Data":"3587b49a728fea415ea057e085aa027b31949602317425845966e17279509f74"} Jul 23 12:16:27.194064 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:16:27.193657 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-290648-c5686d489-4c5pj" Jul 23 12:16:27.214117 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:16:27.214064 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-290648-c5686d489-4c5pj" podStartSLOduration=1.495137228 podStartE2EDuration="2.214047042s" totalImagesPullingTime="718.909814ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 12:16:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 12:16:27.210596493 +0000 UTC m=+2454.282286481" watchObservedRunningTime="2026-07-23 12:16:27.214047042 +0000 UTC m=+2454.285737090" Jul 23 12:16:30.550149 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:16:30.550111 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 12:16:33.204774 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:16:33.204747 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-290648-c5686d489-4c5pj" Jul 23 12:16:33.275957 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:16:33.275903 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-1e1abd-76d9bdf959-qsfnf"] Jul 23 12:16:33.281004 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:16:33.280978 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1e1abd-76d9bdf959-qsfnf" Jul 23 12:16:33.284478 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:16:33.284450 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-1e1abd-serving-cert\"" Jul 23 12:16:33.299520 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:16:33.299493 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-1e1abd-76d9bdf959-qsfnf"] Jul 23 12:16:33.409182 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:16:33.409145 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b6b9b16-f931-4154-ac0f-320cafa0c447-openshift-service-ca\") pod \"maas-api-e2e-shared-b-1e1abd-76d9bdf959-qsfnf\" (UID: \"8b6b9b16-f931-4154-ac0f-320cafa0c447\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1e1abd-76d9bdf959-qsfnf" Jul 23 12:16:33.409368 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:16:33.409209 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b6b9b16-f931-4154-ac0f-320cafa0c447-maas-api-tls\") pod \"maas-api-e2e-shared-b-1e1abd-76d9bdf959-qsfnf\" (UID: \"8b6b9b16-f931-4154-ac0f-320cafa0c447\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1e1abd-76d9bdf959-qsfnf" Jul 23 12:16:33.409368 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:16:33.409244 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97nfw\" (UniqueName: \"kubernetes.io/projected/8b6b9b16-f931-4154-ac0f-320cafa0c447-kube-api-access-97nfw\") pod \"maas-api-e2e-shared-b-1e1abd-76d9bdf959-qsfnf\" (UID: \"8b6b9b16-f931-4154-ac0f-320cafa0c447\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1e1abd-76d9bdf959-qsfnf" Jul 23 12:16:33.510105 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:16:33.510009 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b6b9b16-f931-4154-ac0f-320cafa0c447-maas-api-tls\") pod \"maas-api-e2e-shared-b-1e1abd-76d9bdf959-qsfnf\" (UID: \"8b6b9b16-f931-4154-ac0f-320cafa0c447\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1e1abd-76d9bdf959-qsfnf" Jul 23 12:16:33.510105 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:16:33.510058 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97nfw\" (UniqueName: \"kubernetes.io/projected/8b6b9b16-f931-4154-ac0f-320cafa0c447-kube-api-access-97nfw\") pod \"maas-api-e2e-shared-b-1e1abd-76d9bdf959-qsfnf\" (UID: \"8b6b9b16-f931-4154-ac0f-320cafa0c447\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1e1abd-76d9bdf959-qsfnf" Jul 23 12:16:33.510370 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:16:33.510116 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b6b9b16-f931-4154-ac0f-320cafa0c447-openshift-service-ca\") pod \"maas-api-e2e-shared-b-1e1abd-76d9bdf959-qsfnf\" (UID: \"8b6b9b16-f931-4154-ac0f-320cafa0c447\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1e1abd-76d9bdf959-qsfnf" Jul 23 12:16:33.510739 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:16:33.510721 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b6b9b16-f931-4154-ac0f-320cafa0c447-openshift-service-ca\") pod \"maas-api-e2e-shared-b-1e1abd-76d9bdf959-qsfnf\" (UID: \"8b6b9b16-f931-4154-ac0f-320cafa0c447\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1e1abd-76d9bdf959-qsfnf" Jul 23 12:16:33.512990 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:16:33.512972 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-1e1abd-serving-cert\"" Jul 23 12:16:33.518533 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:16:33.518513 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97nfw\" (UniqueName: \"kubernetes.io/projected/8b6b9b16-f931-4154-ac0f-320cafa0c447-kube-api-access-97nfw\") pod \"maas-api-e2e-shared-b-1e1abd-76d9bdf959-qsfnf\" (UID: \"8b6b9b16-f931-4154-ac0f-320cafa0c447\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1e1abd-76d9bdf959-qsfnf" Jul 23 12:16:33.520252 ip-10-0-134-236 kubenswrapper[2561]: E0723 12:16:33.520231 2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-1e1abd-serving-cert: secret "maas-api-e2e-shared-b-1e1abd-serving-cert" not found Jul 23 12:16:33.520323 ip-10-0-134-236 kubenswrapper[2561]: E0723 12:16:33.520300 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b6b9b16-f931-4154-ac0f-320cafa0c447-maas-api-tls podName:8b6b9b16-f931-4154-ac0f-320cafa0c447 nodeName:}" failed. No retries permitted until 2026-07-23 12:16:34.020281024 +0000 UTC m=+2461.091970999 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8b6b9b16-f931-4154-ac0f-320cafa0c447-maas-api-tls") pod "maas-api-e2e-shared-b-1e1abd-76d9bdf959-qsfnf" (UID: "8b6b9b16-f931-4154-ac0f-320cafa0c447") : secret "maas-api-e2e-shared-b-1e1abd-serving-cert" not found Jul 23 12:16:34.116406 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:16:34.116367 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b6b9b16-f931-4154-ac0f-320cafa0c447-maas-api-tls\") pod \"maas-api-e2e-shared-b-1e1abd-76d9bdf959-qsfnf\" (UID: \"8b6b9b16-f931-4154-ac0f-320cafa0c447\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1e1abd-76d9bdf959-qsfnf" Jul 23 12:16:34.119033 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:16:34.119002 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b6b9b16-f931-4154-ac0f-320cafa0c447-maas-api-tls\") pod \"maas-api-e2e-shared-b-1e1abd-76d9bdf959-qsfnf\" (UID: \"8b6b9b16-f931-4154-ac0f-320cafa0c447\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1e1abd-76d9bdf959-qsfnf" Jul 23 12:16:34.192159 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:16:34.192127 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1e1abd-76d9bdf959-qsfnf" Jul 23 12:16:34.335901 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:16:34.335879 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-1e1abd-76d9bdf959-qsfnf"] Jul 23 12:16:34.337822 ip-10-0-134-236 kubenswrapper[2561]: W0723 12:16:34.337793 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b6b9b16_f931_4154_ac0f_320cafa0c447.slice/crio-d82bde6a3d18499ccfe10244dc5b285c56261aff735bcfdcd00830a3e4a8f01b WatchSource:0}: Error finding container d82bde6a3d18499ccfe10244dc5b285c56261aff735bcfdcd00830a3e4a8f01b: Status 404 returned error can't find the container with id d82bde6a3d18499ccfe10244dc5b285c56261aff735bcfdcd00830a3e4a8f01b Jul 23 12:16:35.236965 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:16:35.236930 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1e1abd-76d9bdf959-qsfnf" event={"ID":"8b6b9b16-f931-4154-ac0f-320cafa0c447","Type":"ContainerStarted","Data":"d82bde6a3d18499ccfe10244dc5b285c56261aff735bcfdcd00830a3e4a8f01b"} Jul 23 12:16:36.243691 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:16:36.243651 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1e1abd-76d9bdf959-qsfnf" event={"ID":"8b6b9b16-f931-4154-ac0f-320cafa0c447","Type":"ContainerStarted","Data":"e814d56c2e0234f4cf91e031c13989686d5f5aa8a1c97c0e237e71fd52d972e2"} Jul 23 12:16:36.244150 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:16:36.243848 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1e1abd-76d9bdf959-qsfnf" Jul 23 12:16:36.263075 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:16:36.263026 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1e1abd-76d9bdf959-qsfnf" podStartSLOduration=2.101644907 podStartE2EDuration="3.263009802s" totalImagesPullingTime="1.161364895s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 12:16:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 12:16:36.260176848 +0000 UTC m=+2463.331866841" watchObservedRunningTime="2026-07-23 12:16:36.263009802 +0000 UTC m=+2463.334699792" Jul 23 12:16:42.254512 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:16:42.254480 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1e1abd-76d9bdf959-qsfnf" Jul 23 12:16:47.454995 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:16:47.454962 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 12:16:59.644137 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:16:59.644094 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 12:17:28.844466 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:17:28.844382 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 12:17:33.684509 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:17:33.684476 2561 scope.go:117] "RemoveContainer" containerID="8b0c18d4819d9003369e7d0db8b5381e4d8b297801d214bd0e982dd9106d837a" Jul 23 12:17:47.444527 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:17:47.444491 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 12:18:06.655304 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:18:06.655262 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 12:18:25.541889 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:18:25.541858 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 12:18:42.052068 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:18:42.052028 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 12:19:05.056478 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:19:05.056393 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 12:19:21.844895 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:19:21.844860 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 12:19:41.442602 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:19:41.442567 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 12:20:09.938127 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:20:09.938084 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 12:20:33.587616 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:20:33.587541 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-7slpj_65247e1f-fc10-4b40-af67-5e646f352263/console-operator/1.log" Jul 23 12:20:33.592259 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:20:33.592236 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-7slpj_65247e1f-fc10-4b40-af67-5e646f352263/console-operator/1.log" Jul 23 12:20:33.722207 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:20:33.722175 2561 scope.go:117] "RemoveContainer" containerID="ec7a5f5976db8d94e3b8aea63a1d1bbca322283a103a26e8ea5c2975fef61f06" Jul 23 12:20:34.244162 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:20:34.244128 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-1e1abd/sub-test-model-1e1abd-kserve-5fc6f78587-8kkcl"] Jul 23 12:20:34.248879 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:20:34.248855 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1e1abd/sub-test-model-1e1abd-kserve-5fc6f78587-8kkcl" Jul 23 12:20:34.252562 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:20:34.252541 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-1e1abd\"/\"sub-test-model-1e1abd-kserve-self-signed-certs\"" Jul 23 12:20:34.252723 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:20:34.252542 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-1e1abd\"/\"kube-root-ca.crt\"" Jul 23 12:20:34.252723 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:20:34.252542 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-1e1abd\"/\"openshift-service-ca.crt\"" Jul 23 12:20:34.252723 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:20:34.252542 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-1e1abd\"/\"default-dockercfg-fr5ck\"" Jul 23 12:20:34.258826 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:20:34.258806 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-1e1abd/sub-test-model-1e1abd-kserve-5fc6f78587-8kkcl"] Jul 23 12:20:34.300962 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:20:34.300899 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5cabd3c7-da57-4e6f-8ff5-24b877b9ab1c-model-cache\") pod \"sub-test-model-1e1abd-kserve-5fc6f78587-8kkcl\" (UID: \"5cabd3c7-da57-4e6f-8ff5-24b877b9ab1c\") " pod="ai-tenant-e2e-shared-b-1e1abd/sub-test-model-1e1abd-kserve-5fc6f78587-8kkcl" Jul 23 12:20:34.300962 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:20:34.300966 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cabd3c7-da57-4e6f-8ff5-24b877b9ab1c-tls-certs\") pod \"sub-test-model-1e1abd-kserve-5fc6f78587-8kkcl\" (UID: \"5cabd3c7-da57-4e6f-8ff5-24b877b9ab1c\") " pod="ai-tenant-e2e-shared-b-1e1abd/sub-test-model-1e1abd-kserve-5fc6f78587-8kkcl" Jul 23 12:20:34.301171 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:20:34.300992 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cabd3c7-da57-4e6f-8ff5-24b877b9ab1c-tmp-dir\") pod \"sub-test-model-1e1abd-kserve-5fc6f78587-8kkcl\" (UID: \"5cabd3c7-da57-4e6f-8ff5-24b877b9ab1c\") " pod="ai-tenant-e2e-shared-b-1e1abd/sub-test-model-1e1abd-kserve-5fc6f78587-8kkcl" Jul 23 12:20:34.301171 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:20:34.301008 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvq84\" (UniqueName: \"kubernetes.io/projected/5cabd3c7-da57-4e6f-8ff5-24b877b9ab1c-kube-api-access-cvq84\") pod \"sub-test-model-1e1abd-kserve-5fc6f78587-8kkcl\" (UID: \"5cabd3c7-da57-4e6f-8ff5-24b877b9ab1c\") " pod="ai-tenant-e2e-shared-b-1e1abd/sub-test-model-1e1abd-kserve-5fc6f78587-8kkcl" Jul 23 12:20:34.301171 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:20:34.301026 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5cabd3c7-da57-4e6f-8ff5-24b877b9ab1c-home\") pod \"sub-test-model-1e1abd-kserve-5fc6f78587-8kkcl\" (UID: \"5cabd3c7-da57-4e6f-8ff5-24b877b9ab1c\") " pod="ai-tenant-e2e-shared-b-1e1abd/sub-test-model-1e1abd-kserve-5fc6f78587-8kkcl" Jul 23 12:20:34.301171 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:20:34.301123 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5cabd3c7-da57-4e6f-8ff5-24b877b9ab1c-dshm\") pod \"sub-test-model-1e1abd-kserve-5fc6f78587-8kkcl\" (UID: \"5cabd3c7-da57-4e6f-8ff5-24b877b9ab1c\") " pod="ai-tenant-e2e-shared-b-1e1abd/sub-test-model-1e1abd-kserve-5fc6f78587-8kkcl" Jul 23 12:20:34.402083 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:20:34.402046 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5cabd3c7-da57-4e6f-8ff5-24b877b9ab1c-dshm\") pod \"sub-test-model-1e1abd-kserve-5fc6f78587-8kkcl\" (UID: \"5cabd3c7-da57-4e6f-8ff5-24b877b9ab1c\") " pod="ai-tenant-e2e-shared-b-1e1abd/sub-test-model-1e1abd-kserve-5fc6f78587-8kkcl" Jul 23 12:20:34.402274 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:20:34.402099 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5cabd3c7-da57-4e6f-8ff5-24b877b9ab1c-model-cache\") pod \"sub-test-model-1e1abd-kserve-5fc6f78587-8kkcl\" (UID: \"5cabd3c7-da57-4e6f-8ff5-24b877b9ab1c\") " pod="ai-tenant-e2e-shared-b-1e1abd/sub-test-model-1e1abd-kserve-5fc6f78587-8kkcl" Jul 23 12:20:34.402274 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:20:34.402123 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cabd3c7-da57-4e6f-8ff5-24b877b9ab1c-tls-certs\") pod \"sub-test-model-1e1abd-kserve-5fc6f78587-8kkcl\" (UID: \"5cabd3c7-da57-4e6f-8ff5-24b877b9ab1c\") " pod="ai-tenant-e2e-shared-b-1e1abd/sub-test-model-1e1abd-kserve-5fc6f78587-8kkcl" Jul 23 12:20:34.402274 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:20:34.402146 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cabd3c7-da57-4e6f-8ff5-24b877b9ab1c-tmp-dir\") pod \"sub-test-model-1e1abd-kserve-5fc6f78587-8kkcl\" (UID: \"5cabd3c7-da57-4e6f-8ff5-24b877b9ab1c\") " pod="ai-tenant-e2e-shared-b-1e1abd/sub-test-model-1e1abd-kserve-5fc6f78587-8kkcl" Jul 23 12:20:34.402274 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:20:34.402163 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvq84\" (UniqueName: \"kubernetes.io/projected/5cabd3c7-da57-4e6f-8ff5-24b877b9ab1c-kube-api-access-cvq84\") pod \"sub-test-model-1e1abd-kserve-5fc6f78587-8kkcl\" (UID: \"5cabd3c7-da57-4e6f-8ff5-24b877b9ab1c\") " pod="ai-tenant-e2e-shared-b-1e1abd/sub-test-model-1e1abd-kserve-5fc6f78587-8kkcl" Jul 23 12:20:34.402274 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:20:34.402179 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5cabd3c7-da57-4e6f-8ff5-24b877b9ab1c-home\") pod \"sub-test-model-1e1abd-kserve-5fc6f78587-8kkcl\" (UID: \"5cabd3c7-da57-4e6f-8ff5-24b877b9ab1c\") " pod="ai-tenant-e2e-shared-b-1e1abd/sub-test-model-1e1abd-kserve-5fc6f78587-8kkcl" Jul 23 12:20:34.402531 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:20:34.402510 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cabd3c7-da57-4e6f-8ff5-24b877b9ab1c-tmp-dir\") pod \"sub-test-model-1e1abd-kserve-5fc6f78587-8kkcl\" (UID: \"5cabd3c7-da57-4e6f-8ff5-24b877b9ab1c\") " pod="ai-tenant-e2e-shared-b-1e1abd/sub-test-model-1e1abd-kserve-5fc6f78587-8kkcl" Jul 23 12:20:34.402588 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:20:34.402558 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5cabd3c7-da57-4e6f-8ff5-24b877b9ab1c-home\") pod \"sub-test-model-1e1abd-kserve-5fc6f78587-8kkcl\" (UID: \"5cabd3c7-da57-4e6f-8ff5-24b877b9ab1c\") " pod="ai-tenant-e2e-shared-b-1e1abd/sub-test-model-1e1abd-kserve-5fc6f78587-8kkcl" Jul 23 12:20:34.402630 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:20:34.402606 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5cabd3c7-da57-4e6f-8ff5-24b877b9ab1c-model-cache\") pod \"sub-test-model-1e1abd-kserve-5fc6f78587-8kkcl\" (UID: \"5cabd3c7-da57-4e6f-8ff5-24b877b9ab1c\") " pod="ai-tenant-e2e-shared-b-1e1abd/sub-test-model-1e1abd-kserve-5fc6f78587-8kkcl" Jul 23 12:20:34.404522 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:20:34.404494 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5cabd3c7-da57-4e6f-8ff5-24b877b9ab1c-dshm\") pod \"sub-test-model-1e1abd-kserve-5fc6f78587-8kkcl\" (UID: \"5cabd3c7-da57-4e6f-8ff5-24b877b9ab1c\") " pod="ai-tenant-e2e-shared-b-1e1abd/sub-test-model-1e1abd-kserve-5fc6f78587-8kkcl" Jul 23 12:20:34.404816 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:20:34.404797 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cabd3c7-da57-4e6f-8ff5-24b877b9ab1c-tls-certs\") pod \"sub-test-model-1e1abd-kserve-5fc6f78587-8kkcl\" (UID: \"5cabd3c7-da57-4e6f-8ff5-24b877b9ab1c\") " pod="ai-tenant-e2e-shared-b-1e1abd/sub-test-model-1e1abd-kserve-5fc6f78587-8kkcl" Jul 23 12:20:34.412825 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:20:34.412803 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvq84\" (UniqueName: \"kubernetes.io/projected/5cabd3c7-da57-4e6f-8ff5-24b877b9ab1c-kube-api-access-cvq84\") pod \"sub-test-model-1e1abd-kserve-5fc6f78587-8kkcl\" (UID: \"5cabd3c7-da57-4e6f-8ff5-24b877b9ab1c\") " pod="ai-tenant-e2e-shared-b-1e1abd/sub-test-model-1e1abd-kserve-5fc6f78587-8kkcl" Jul 23 12:20:34.559155 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:20:34.559067 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1e1abd/sub-test-model-1e1abd-kserve-5fc6f78587-8kkcl" Jul 23 12:20:34.696357 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:20:34.696302 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-1e1abd/sub-test-model-1e1abd-kserve-5fc6f78587-8kkcl"] Jul 23 12:20:34.698757 ip-10-0-134-236 kubenswrapper[2561]: W0723 12:20:34.698731 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5cabd3c7_da57_4e6f_8ff5_24b877b9ab1c.slice/crio-58f27a61382656fe47a21ed0ac75a8d070f2b4fab3f1005d4922b6360d105892 WatchSource:0}: Error finding container 58f27a61382656fe47a21ed0ac75a8d070f2b4fab3f1005d4922b6360d105892: Status 404 returned error can't find the container with id 58f27a61382656fe47a21ed0ac75a8d070f2b4fab3f1005d4922b6360d105892 Jul 23 12:20:35.501259 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:20:35.501218 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1e1abd/sub-test-model-1e1abd-kserve-5fc6f78587-8kkcl" event={"ID":"5cabd3c7-da57-4e6f-8ff5-24b877b9ab1c","Type":"ContainerStarted","Data":"3a0d7b3f25bfda8f30fa541ebe39fa2a9db0484daf7c813aa15547e85d3f35de"} Jul 23 12:20:35.501259 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:20:35.501260 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1e1abd/sub-test-model-1e1abd-kserve-5fc6f78587-8kkcl" event={"ID":"5cabd3c7-da57-4e6f-8ff5-24b877b9ab1c","Type":"ContainerStarted","Data":"58f27a61382656fe47a21ed0ac75a8d070f2b4fab3f1005d4922b6360d105892"} Jul 23 12:20:35.521420 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:20:35.521371 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-1e1abd/sub-test-model-1e1abd-kserve-5fc6f78587-8kkcl" podStartSLOduration=1.521356044 podStartE2EDuration="1.521356044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 12:20:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 12:20:35.519099877 +0000 UTC m=+2702.590789870" watchObservedRunningTime="2026-07-23 12:20:35.521356044 +0000 UTC m=+2702.593046035" Jul 23 12:20:38.339665 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:20:38.339630 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 12:20:44.559292 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:20:44.559241 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-1e1abd/sub-test-model-1e1abd-kserve-5fc6f78587-8kkcl" Jul 23 12:20:44.559900 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:20:44.559310 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-1e1abd/sub-test-model-1e1abd-kserve-5fc6f78587-8kkcl" Jul 23 12:20:44.572395 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:20:44.572371 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-1e1abd/sub-test-model-1e1abd-kserve-5fc6f78587-8kkcl" Jul 23 12:20:45.563958 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:20:45.563902 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-1e1abd/sub-test-model-1e1abd-kserve-5fc6f78587-8kkcl" Jul 23 12:21:04.846013 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:21:04.845971 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 12:21:23.741487 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:21:23.741450 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 12:21:32.334263 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:21:32.334224 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-1e1abd/sub-test-model-1e1abd-kserve-5fc6f78587-8kkcl"] Jul 23 12:21:33.742840 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:21:33.742809 2561 scope.go:117] "RemoveContainer" containerID="dea89109ef7e804891d63f886cde0be889f4d16d1e582f1ccb09417544ac4ac6" Jul 23 12:21:33.751969 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:21:33.751948 2561 scope.go:117] "RemoveContainer" containerID="b0817567f45f22c9a3ee4fe4343553c2282de23e1a431ee6ffa3cf2846bb4ee2" Jul 23 12:21:45.936311 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:21:45.936275 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 12:21:47.354810 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:21:47.354747 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-1e1abd/sub-test-model-1e1abd-kserve-5fc6f78587-8kkcl" podUID="5cabd3c7-da57-4e6f-8ff5-24b877b9ab1c" containerName="main" containerID="cri-o://3a0d7b3f25bfda8f30fa541ebe39fa2a9db0484daf7c813aa15547e85d3f35de" gracePeriod=45 Jul 23 12:21:47.616398 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:21:47.616318 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1e1abd/sub-test-model-1e1abd-kserve-5fc6f78587-8kkcl" Jul 23 12:21:47.763139 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:21:47.763102 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5cabd3c7-da57-4e6f-8ff5-24b877b9ab1c-dshm\") pod \"5cabd3c7-da57-4e6f-8ff5-24b877b9ab1c\" (UID: \"5cabd3c7-da57-4e6f-8ff5-24b877b9ab1c\") " Jul 23 12:21:47.763345 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:21:47.763177 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5cabd3c7-da57-4e6f-8ff5-24b877b9ab1c-home\") pod \"5cabd3c7-da57-4e6f-8ff5-24b877b9ab1c\" (UID: \"5cabd3c7-da57-4e6f-8ff5-24b877b9ab1c\") " Jul 23 12:21:47.763345 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:21:47.763254 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5cabd3c7-da57-4e6f-8ff5-24b877b9ab1c-model-cache\") pod \"5cabd3c7-da57-4e6f-8ff5-24b877b9ab1c\" (UID: \"5cabd3c7-da57-4e6f-8ff5-24b877b9ab1c\") " Jul 23 12:21:47.763345 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:21:47.763284 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cabd3c7-da57-4e6f-8ff5-24b877b9ab1c-tmp-dir\") pod \"5cabd3c7-da57-4e6f-8ff5-24b877b9ab1c\" (UID: \"5cabd3c7-da57-4e6f-8ff5-24b877b9ab1c\") " Jul 23 12:21:47.763345 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:21:47.763325 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cabd3c7-da57-4e6f-8ff5-24b877b9ab1c-tls-certs\") pod \"5cabd3c7-da57-4e6f-8ff5-24b877b9ab1c\" (UID: \"5cabd3c7-da57-4e6f-8ff5-24b877b9ab1c\") " Jul 23 12:21:47.763563 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:21:47.763353 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cvq84\" (UniqueName: \"kubernetes.io/projected/5cabd3c7-da57-4e6f-8ff5-24b877b9ab1c-kube-api-access-cvq84\") pod \"5cabd3c7-da57-4e6f-8ff5-24b877b9ab1c\" (UID: \"5cabd3c7-da57-4e6f-8ff5-24b877b9ab1c\") " Jul 23 12:21:47.763563 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:21:47.763504 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cabd3c7-da57-4e6f-8ff5-24b877b9ab1c-home" (OuterVolumeSpecName: "home") pod "5cabd3c7-da57-4e6f-8ff5-24b877b9ab1c" (UID: "5cabd3c7-da57-4e6f-8ff5-24b877b9ab1c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:21:47.763563 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:21:47.763503 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cabd3c7-da57-4e6f-8ff5-24b877b9ab1c-model-cache" (OuterVolumeSpecName: "model-cache") pod "5cabd3c7-da57-4e6f-8ff5-24b877b9ab1c" (UID: "5cabd3c7-da57-4e6f-8ff5-24b877b9ab1c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:21:47.763723 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:21:47.763538 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cabd3c7-da57-4e6f-8ff5-24b877b9ab1c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5cabd3c7-da57-4e6f-8ff5-24b877b9ab1c" (UID: "5cabd3c7-da57-4e6f-8ff5-24b877b9ab1c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:21:47.763774 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:21:47.763755 2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5cabd3c7-da57-4e6f-8ff5-24b877b9ab1c-model-cache\") on node \"ip-10-0-134-236.ec2.internal\" DevicePath \"\"" Jul 23 12:21:47.763829 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:21:47.763773 2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cabd3c7-da57-4e6f-8ff5-24b877b9ab1c-tmp-dir\") on node \"ip-10-0-134-236.ec2.internal\" DevicePath \"\"" Jul 23 12:21:47.763829 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:21:47.763786 2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5cabd3c7-da57-4e6f-8ff5-24b877b9ab1c-home\") on node \"ip-10-0-134-236.ec2.internal\" DevicePath \"\"" Jul 23 12:21:47.765736 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:21:47.765705 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cabd3c7-da57-4e6f-8ff5-24b877b9ab1c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5cabd3c7-da57-4e6f-8ff5-24b877b9ab1c" (UID: "5cabd3c7-da57-4e6f-8ff5-24b877b9ab1c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 12:21:47.765845 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:21:47.765744 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cabd3c7-da57-4e6f-8ff5-24b877b9ab1c-kube-api-access-cvq84" (OuterVolumeSpecName: "kube-api-access-cvq84") pod "5cabd3c7-da57-4e6f-8ff5-24b877b9ab1c" (UID: "5cabd3c7-da57-4e6f-8ff5-24b877b9ab1c"). InnerVolumeSpecName "kube-api-access-cvq84". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 12:21:47.765845 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:21:47.765761 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cabd3c7-da57-4e6f-8ff5-24b877b9ab1c-dshm" (OuterVolumeSpecName: "dshm") pod "5cabd3c7-da57-4e6f-8ff5-24b877b9ab1c" (UID: "5cabd3c7-da57-4e6f-8ff5-24b877b9ab1c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:21:47.864373 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:21:47.864333 2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5cabd3c7-da57-4e6f-8ff5-24b877b9ab1c-dshm\") on node \"ip-10-0-134-236.ec2.internal\" DevicePath \"\"" Jul 23 12:21:47.864373 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:21:47.864361 2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cabd3c7-da57-4e6f-8ff5-24b877b9ab1c-tls-certs\") on node \"ip-10-0-134-236.ec2.internal\" DevicePath \"\"" Jul 23 12:21:47.864373 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:21:47.864374 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvq84\" (UniqueName: \"kubernetes.io/projected/5cabd3c7-da57-4e6f-8ff5-24b877b9ab1c-kube-api-access-cvq84\") on node \"ip-10-0-134-236.ec2.internal\" DevicePath \"\"" Jul 23 12:21:47.875634 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:21:47.875546 2561 generic.go:356] "Generic (PLEG): container finished" podID="5cabd3c7-da57-4e6f-8ff5-24b877b9ab1c" containerID="3a0d7b3f25bfda8f30fa541ebe39fa2a9db0484daf7c813aa15547e85d3f35de" exitCode=0 Jul 23 12:21:47.875634 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:21:47.875600 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1e1abd/sub-test-model-1e1abd-kserve-5fc6f78587-8kkcl" Jul 23 12:21:47.875634 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:21:47.875621 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1e1abd/sub-test-model-1e1abd-kserve-5fc6f78587-8kkcl" event={"ID":"5cabd3c7-da57-4e6f-8ff5-24b877b9ab1c","Type":"ContainerDied","Data":"3a0d7b3f25bfda8f30fa541ebe39fa2a9db0484daf7c813aa15547e85d3f35de"} Jul 23 12:21:47.875871 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:21:47.875660 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1e1abd/sub-test-model-1e1abd-kserve-5fc6f78587-8kkcl" event={"ID":"5cabd3c7-da57-4e6f-8ff5-24b877b9ab1c","Type":"ContainerDied","Data":"58f27a61382656fe47a21ed0ac75a8d070f2b4fab3f1005d4922b6360d105892"} Jul 23 12:21:47.875871 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:21:47.875692 2561 scope.go:117] "RemoveContainer" containerID="3a0d7b3f25bfda8f30fa541ebe39fa2a9db0484daf7c813aa15547e85d3f35de" Jul 23 12:21:47.885658 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:21:47.885639 2561 scope.go:117] "RemoveContainer" containerID="3a0d7b3f25bfda8f30fa541ebe39fa2a9db0484daf7c813aa15547e85d3f35de" Jul 23 12:21:47.885974 ip-10-0-134-236 kubenswrapper[2561]: E0723 12:21:47.885955 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a0d7b3f25bfda8f30fa541ebe39fa2a9db0484daf7c813aa15547e85d3f35de\": container with ID starting with 3a0d7b3f25bfda8f30fa541ebe39fa2a9db0484daf7c813aa15547e85d3f35de not found: ID does not exist" containerID="3a0d7b3f25bfda8f30fa541ebe39fa2a9db0484daf7c813aa15547e85d3f35de" Jul 23 12:21:47.886024 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:21:47.885985 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a0d7b3f25bfda8f30fa541ebe39fa2a9db0484daf7c813aa15547e85d3f35de"} err="failed to get container status \"3a0d7b3f25bfda8f30fa541ebe39fa2a9db0484daf7c813aa15547e85d3f35de\": rpc error: code = NotFound desc = could not find container \"3a0d7b3f25bfda8f30fa541ebe39fa2a9db0484daf7c813aa15547e85d3f35de\": container with ID starting with 3a0d7b3f25bfda8f30fa541ebe39fa2a9db0484daf7c813aa15547e85d3f35de not found: ID does not exist" Jul 23 12:21:47.899259 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:21:47.899228 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-1e1abd/sub-test-model-1e1abd-kserve-5fc6f78587-8kkcl"] Jul 23 12:21:47.903676 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:21:47.903645 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-1e1abd/sub-test-model-1e1abd-kserve-5fc6f78587-8kkcl"] Jul 23 12:21:49.455011 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:21:49.454975 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5cabd3c7-da57-4e6f-8ff5-24b877b9ab1c" path="/var/lib/kubelet/pods/5cabd3c7-da57-4e6f-8ff5-24b877b9ab1c/volumes" Jul 23 12:22:13.844631 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:22:13.844597 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-1e1abd/rate-test-model-1e1abd-kserve-74476555d7-cxkqb"] Jul 23 12:22:13.847093 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:22:13.844985 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cabd3c7-da57-4e6f-8ff5-24b877b9ab1c" containerName="main" Jul 23 12:22:13.847093 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:22:13.844998 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cabd3c7-da57-4e6f-8ff5-24b877b9ab1c" containerName="main" Jul 23 12:22:13.847093 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:22:13.845061 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="5cabd3c7-da57-4e6f-8ff5-24b877b9ab1c" containerName="main" Jul 23 12:22:13.848000 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:22:13.847984 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1e1abd/rate-test-model-1e1abd-kserve-74476555d7-cxkqb" Jul 23 12:22:13.851696 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:22:13.851665 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-1e1abd\"/\"kube-root-ca.crt\"" Jul 23 12:22:13.851836 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:22:13.851710 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-1e1abd\"/\"openshift-service-ca.crt\"" Jul 23 12:22:13.851836 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:22:13.851718 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-1e1abd\"/\"default-dockercfg-fr5ck\"" Jul 23 12:22:13.851836 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:22:13.851673 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-1e1abd\"/\"rate-test-model-1e1abd-kserve-self-signed-certs\"" Jul 23 12:22:13.858119 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:22:13.858100 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-1e1abd/rate-test-model-1e1abd-kserve-74476555d7-cxkqb"] Jul 23 12:22:13.890459 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:22:13.890432 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7915910-0559-46db-bfb8-51771724d937-home\") pod \"rate-test-model-1e1abd-kserve-74476555d7-cxkqb\" (UID: \"f7915910-0559-46db-bfb8-51771724d937\") " pod="ai-tenant-e2e-shared-b-1e1abd/rate-test-model-1e1abd-kserve-74476555d7-cxkqb" Jul 23 12:22:13.890608 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:22:13.890488 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7915910-0559-46db-bfb8-51771724d937-dshm\") pod \"rate-test-model-1e1abd-kserve-74476555d7-cxkqb\" (UID: \"f7915910-0559-46db-bfb8-51771724d937\") " pod="ai-tenant-e2e-shared-b-1e1abd/rate-test-model-1e1abd-kserve-74476555d7-cxkqb" Jul 23 12:22:13.890608 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:22:13.890516 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7915910-0559-46db-bfb8-51771724d937-model-cache\") pod \"rate-test-model-1e1abd-kserve-74476555d7-cxkqb\" (UID: \"f7915910-0559-46db-bfb8-51771724d937\") " pod="ai-tenant-e2e-shared-b-1e1abd/rate-test-model-1e1abd-kserve-74476555d7-cxkqb" Jul 23 12:22:13.890608 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:22:13.890575 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7915910-0559-46db-bfb8-51771724d937-tls-certs\") pod \"rate-test-model-1e1abd-kserve-74476555d7-cxkqb\" (UID: \"f7915910-0559-46db-bfb8-51771724d937\") " pod="ai-tenant-e2e-shared-b-1e1abd/rate-test-model-1e1abd-kserve-74476555d7-cxkqb" Jul 23 12:22:13.890608 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:22:13.890602 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvbcj\" (UniqueName: \"kubernetes.io/projected/f7915910-0559-46db-bfb8-51771724d937-kube-api-access-pvbcj\") pod \"rate-test-model-1e1abd-kserve-74476555d7-cxkqb\" (UID: \"f7915910-0559-46db-bfb8-51771724d937\") " pod="ai-tenant-e2e-shared-b-1e1abd/rate-test-model-1e1abd-kserve-74476555d7-cxkqb" Jul 23 12:22:13.890757 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:22:13.890624 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7915910-0559-46db-bfb8-51771724d937-tmp-dir\") pod \"rate-test-model-1e1abd-kserve-74476555d7-cxkqb\" (UID: \"f7915910-0559-46db-bfb8-51771724d937\") " pod="ai-tenant-e2e-shared-b-1e1abd/rate-test-model-1e1abd-kserve-74476555d7-cxkqb" Jul 23 12:22:13.991241 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:22:13.991204 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7915910-0559-46db-bfb8-51771724d937-tls-certs\") pod \"rate-test-model-1e1abd-kserve-74476555d7-cxkqb\" (UID: \"f7915910-0559-46db-bfb8-51771724d937\") " pod="ai-tenant-e2e-shared-b-1e1abd/rate-test-model-1e1abd-kserve-74476555d7-cxkqb" Jul 23 12:22:13.991241 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:22:13.991243 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvbcj\" (UniqueName: \"kubernetes.io/projected/f7915910-0559-46db-bfb8-51771724d937-kube-api-access-pvbcj\") pod \"rate-test-model-1e1abd-kserve-74476555d7-cxkqb\" (UID: \"f7915910-0559-46db-bfb8-51771724d937\") " pod="ai-tenant-e2e-shared-b-1e1abd/rate-test-model-1e1abd-kserve-74476555d7-cxkqb" Jul 23 12:22:13.991477 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:22:13.991260 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7915910-0559-46db-bfb8-51771724d937-tmp-dir\") pod \"rate-test-model-1e1abd-kserve-74476555d7-cxkqb\" (UID: \"f7915910-0559-46db-bfb8-51771724d937\") " pod="ai-tenant-e2e-shared-b-1e1abd/rate-test-model-1e1abd-kserve-74476555d7-cxkqb" Jul 23 12:22:13.991477 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:22:13.991302 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7915910-0559-46db-bfb8-51771724d937-home\") pod \"rate-test-model-1e1abd-kserve-74476555d7-cxkqb\" (UID: \"f7915910-0559-46db-bfb8-51771724d937\") " pod="ai-tenant-e2e-shared-b-1e1abd/rate-test-model-1e1abd-kserve-74476555d7-cxkqb" Jul 23 12:22:13.991477 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:22:13.991360 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7915910-0559-46db-bfb8-51771724d937-dshm\") pod \"rate-test-model-1e1abd-kserve-74476555d7-cxkqb\" (UID: \"f7915910-0559-46db-bfb8-51771724d937\") " pod="ai-tenant-e2e-shared-b-1e1abd/rate-test-model-1e1abd-kserve-74476555d7-cxkqb" Jul 23 12:22:13.991477 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:22:13.991396 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7915910-0559-46db-bfb8-51771724d937-model-cache\") pod \"rate-test-model-1e1abd-kserve-74476555d7-cxkqb\" (UID: \"f7915910-0559-46db-bfb8-51771724d937\") " pod="ai-tenant-e2e-shared-b-1e1abd/rate-test-model-1e1abd-kserve-74476555d7-cxkqb" Jul 23 12:22:13.991854 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:22:13.991822 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7915910-0559-46db-bfb8-51771724d937-home\") pod \"rate-test-model-1e1abd-kserve-74476555d7-cxkqb\" (UID: \"f7915910-0559-46db-bfb8-51771724d937\") " pod="ai-tenant-e2e-shared-b-1e1abd/rate-test-model-1e1abd-kserve-74476555d7-cxkqb" Jul 23 12:22:13.992024 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:22:13.991828 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7915910-0559-46db-bfb8-51771724d937-model-cache\") pod \"rate-test-model-1e1abd-kserve-74476555d7-cxkqb\" (UID: \"f7915910-0559-46db-bfb8-51771724d937\") " pod="ai-tenant-e2e-shared-b-1e1abd/rate-test-model-1e1abd-kserve-74476555d7-cxkqb" Jul 23 12:22:13.992024 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:22:13.991874 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7915910-0559-46db-bfb8-51771724d937-tmp-dir\") pod \"rate-test-model-1e1abd-kserve-74476555d7-cxkqb\" (UID: \"f7915910-0559-46db-bfb8-51771724d937\") " pod="ai-tenant-e2e-shared-b-1e1abd/rate-test-model-1e1abd-kserve-74476555d7-cxkqb" Jul 23 12:22:13.993767 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:22:13.993746 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7915910-0559-46db-bfb8-51771724d937-dshm\") pod \"rate-test-model-1e1abd-kserve-74476555d7-cxkqb\" (UID: \"f7915910-0559-46db-bfb8-51771724d937\") " pod="ai-tenant-e2e-shared-b-1e1abd/rate-test-model-1e1abd-kserve-74476555d7-cxkqb" Jul 23 12:22:13.994165 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:22:13.994144 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7915910-0559-46db-bfb8-51771724d937-tls-certs\") pod \"rate-test-model-1e1abd-kserve-74476555d7-cxkqb\" (UID: \"f7915910-0559-46db-bfb8-51771724d937\") " pod="ai-tenant-e2e-shared-b-1e1abd/rate-test-model-1e1abd-kserve-74476555d7-cxkqb" Jul 23 12:22:14.000719 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:22:14.000695 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvbcj\" (UniqueName: \"kubernetes.io/projected/f7915910-0559-46db-bfb8-51771724d937-kube-api-access-pvbcj\") pod \"rate-test-model-1e1abd-kserve-74476555d7-cxkqb\" (UID: \"f7915910-0559-46db-bfb8-51771724d937\") " pod="ai-tenant-e2e-shared-b-1e1abd/rate-test-model-1e1abd-kserve-74476555d7-cxkqb" Jul 23 12:22:14.157626 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:22:14.157539 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1e1abd/rate-test-model-1e1abd-kserve-74476555d7-cxkqb" Jul 23 12:22:14.301902 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:22:14.301878 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-1e1abd/rate-test-model-1e1abd-kserve-74476555d7-cxkqb"] Jul 23 12:22:14.303718 ip-10-0-134-236 kubenswrapper[2561]: W0723 12:22:14.303682 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf7915910_0559_46db_bfb8_51771724d937.slice/crio-c2fcb9f5ae04a7e79eaecb51e2e77d3c1a28805d5eb88fbf17608db170886f0d WatchSource:0}: Error finding container c2fcb9f5ae04a7e79eaecb51e2e77d3c1a28805d5eb88fbf17608db170886f0d: Status 404 returned error can't find the container with id c2fcb9f5ae04a7e79eaecb51e2e77d3c1a28805d5eb88fbf17608db170886f0d Jul 23 12:22:15.018321 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:22:15.018282 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1e1abd/rate-test-model-1e1abd-kserve-74476555d7-cxkqb" event={"ID":"f7915910-0559-46db-bfb8-51771724d937","Type":"ContainerStarted","Data":"d1620a6937d38e8ce9da53746c4a5462c6dcbc8fe94ab7e5ca1c37cbe20a808c"} Jul 23 12:22:15.018735 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:22:15.018329 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1e1abd/rate-test-model-1e1abd-kserve-74476555d7-cxkqb" event={"ID":"f7915910-0559-46db-bfb8-51771724d937","Type":"ContainerStarted","Data":"c2fcb9f5ae04a7e79eaecb51e2e77d3c1a28805d5eb88fbf17608db170886f0d"} Jul 23 12:22:17.740296 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:22:17.740237 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-1e1abd/rate-test-model-1e1abd-kserve-74476555d7-cxkqb" podStartSLOduration=4.740216973 podStartE2EDuration="4.740216973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 12:22:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 12:22:15.044314194 +0000 UTC m=+2802.116004189" watchObservedRunningTime="2026-07-23 12:22:17.740216973 +0000 UTC m=+2804.811906963" Jul 23 12:22:17.740706 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:22:17.740375 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 12:22:24.157871 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:22:24.157836 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-1e1abd/rate-test-model-1e1abd-kserve-74476555d7-cxkqb" Jul 23 12:22:24.158363 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:22:24.157898 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-1e1abd/rate-test-model-1e1abd-kserve-74476555d7-cxkqb" Jul 23 12:22:24.171286 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:22:24.171260 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-1e1abd/rate-test-model-1e1abd-kserve-74476555d7-cxkqb" Jul 23 12:22:25.079898 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:22:25.079867 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-1e1abd/rate-test-model-1e1abd-kserve-74476555d7-cxkqb" Jul 23 12:22:33.776187 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:22:33.776153 2561 scope.go:117] "RemoveContainer" containerID="d2152d21d4008bf61c4eef5f7fb0d44417a54d852eab22c3452ece5de9083004" Jul 23 12:22:37.952358 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:22:37.952327 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 12:22:54.937070 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:22:54.937027 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 12:23:02.353436 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:02.353398 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-1e1abd/rate-test-model-1e1abd-kserve-74476555d7-cxkqb"] Jul 23 12:23:11.443127 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:11.443088 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 12:23:17.374731 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:17.374673 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-1e1abd/rate-test-model-1e1abd-kserve-74476555d7-cxkqb" podUID="f7915910-0559-46db-bfb8-51771724d937" containerName="main" containerID="cri-o://d1620a6937d38e8ce9da53746c4a5462c6dcbc8fe94ab7e5ca1c37cbe20a808c" gracePeriod=45 Jul 23 12:23:17.621575 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:17.621551 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1e1abd/rate-test-model-1e1abd-kserve-74476555d7-cxkqb" Jul 23 12:23:17.729802 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:17.729700 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7915910-0559-46db-bfb8-51771724d937-dshm\") pod \"f7915910-0559-46db-bfb8-51771724d937\" (UID: \"f7915910-0559-46db-bfb8-51771724d937\") " Jul 23 12:23:17.729802 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:17.729737 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7915910-0559-46db-bfb8-51771724d937-model-cache\") pod \"f7915910-0559-46db-bfb8-51771724d937\" (UID: \"f7915910-0559-46db-bfb8-51771724d937\") " Jul 23 12:23:17.729802 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:17.729789 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7915910-0559-46db-bfb8-51771724d937-tls-certs\") pod \"f7915910-0559-46db-bfb8-51771724d937\" (UID: \"f7915910-0559-46db-bfb8-51771724d937\") " Jul 23 12:23:17.730142 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:17.729808 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7915910-0559-46db-bfb8-51771724d937-tmp-dir\") pod \"f7915910-0559-46db-bfb8-51771724d937\" (UID: \"f7915910-0559-46db-bfb8-51771724d937\") " Jul 23 12:23:17.730142 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:17.729825 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pvbcj\" (UniqueName: \"kubernetes.io/projected/f7915910-0559-46db-bfb8-51771724d937-kube-api-access-pvbcj\") pod \"f7915910-0559-46db-bfb8-51771724d937\" (UID: \"f7915910-0559-46db-bfb8-51771724d937\") " Jul 23 12:23:17.730142 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:17.729938 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7915910-0559-46db-bfb8-51771724d937-home\") pod \"f7915910-0559-46db-bfb8-51771724d937\" (UID: \"f7915910-0559-46db-bfb8-51771724d937\") " Jul 23 12:23:17.730285 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:17.730142 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7915910-0559-46db-bfb8-51771724d937-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f7915910-0559-46db-bfb8-51771724d937" (UID: "f7915910-0559-46db-bfb8-51771724d937"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:23:17.730285 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:17.730171 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7915910-0559-46db-bfb8-51771724d937-model-cache" (OuterVolumeSpecName: "model-cache") pod "f7915910-0559-46db-bfb8-51771724d937" (UID: "f7915910-0559-46db-bfb8-51771724d937"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:23:17.730285 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:17.730210 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7915910-0559-46db-bfb8-51771724d937-home" (OuterVolumeSpecName: "home") pod "f7915910-0559-46db-bfb8-51771724d937" (UID: "f7915910-0559-46db-bfb8-51771724d937"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:23:17.730285 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:17.730257 2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7915910-0559-46db-bfb8-51771724d937-model-cache\") on node \"ip-10-0-134-236.ec2.internal\" DevicePath \"\"" Jul 23 12:23:17.730285 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:17.730269 2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7915910-0559-46db-bfb8-51771724d937-tmp-dir\") on node \"ip-10-0-134-236.ec2.internal\" DevicePath \"\"" Jul 23 12:23:17.732184 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:17.732157 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7915910-0559-46db-bfb8-51771724d937-dshm" (OuterVolumeSpecName: "dshm") pod "f7915910-0559-46db-bfb8-51771724d937" (UID: "f7915910-0559-46db-bfb8-51771724d937"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:23:17.732287 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:17.732225 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7915910-0559-46db-bfb8-51771724d937-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f7915910-0559-46db-bfb8-51771724d937" (UID: "f7915910-0559-46db-bfb8-51771724d937"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 12:23:17.732331 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:17.732316 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7915910-0559-46db-bfb8-51771724d937-kube-api-access-pvbcj" (OuterVolumeSpecName: "kube-api-access-pvbcj") pod "f7915910-0559-46db-bfb8-51771724d937" (UID: "f7915910-0559-46db-bfb8-51771724d937"). InnerVolumeSpecName "kube-api-access-pvbcj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 12:23:17.831023 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:17.830985 2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7915910-0559-46db-bfb8-51771724d937-dshm\") on node \"ip-10-0-134-236.ec2.internal\" DevicePath \"\"" Jul 23 12:23:17.831023 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:17.831017 2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7915910-0559-46db-bfb8-51771724d937-tls-certs\") on node \"ip-10-0-134-236.ec2.internal\" DevicePath \"\"" Jul 23 12:23:17.831023 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:17.831027 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvbcj\" (UniqueName: \"kubernetes.io/projected/f7915910-0559-46db-bfb8-51771724d937-kube-api-access-pvbcj\") on node \"ip-10-0-134-236.ec2.internal\" DevicePath \"\"" Jul 23 12:23:17.831280 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:17.831039 2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7915910-0559-46db-bfb8-51771724d937-home\") on node \"ip-10-0-134-236.ec2.internal\" DevicePath \"\"" Jul 23 12:23:18.341221 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:18.341191 2561 generic.go:356] "Generic (PLEG): container finished" podID="f7915910-0559-46db-bfb8-51771724d937" containerID="d1620a6937d38e8ce9da53746c4a5462c6dcbc8fe94ab7e5ca1c37cbe20a808c" exitCode=0 Jul 23 12:23:18.341430 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:18.341233 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1e1abd/rate-test-model-1e1abd-kserve-74476555d7-cxkqb" Jul 23 12:23:18.341430 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:18.341276 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1e1abd/rate-test-model-1e1abd-kserve-74476555d7-cxkqb" event={"ID":"f7915910-0559-46db-bfb8-51771724d937","Type":"ContainerDied","Data":"d1620a6937d38e8ce9da53746c4a5462c6dcbc8fe94ab7e5ca1c37cbe20a808c"} Jul 23 12:23:18.341430 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:18.341324 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1e1abd/rate-test-model-1e1abd-kserve-74476555d7-cxkqb" event={"ID":"f7915910-0559-46db-bfb8-51771724d937","Type":"ContainerDied","Data":"c2fcb9f5ae04a7e79eaecb51e2e77d3c1a28805d5eb88fbf17608db170886f0d"} Jul 23 12:23:18.341430 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:18.341344 2561 scope.go:117] "RemoveContainer" containerID="d1620a6937d38e8ce9da53746c4a5462c6dcbc8fe94ab7e5ca1c37cbe20a808c" Jul 23 12:23:18.350787 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:18.350771 2561 scope.go:117] "RemoveContainer" containerID="d1620a6937d38e8ce9da53746c4a5462c6dcbc8fe94ab7e5ca1c37cbe20a808c" Jul 23 12:23:18.351046 ip-10-0-134-236 kubenswrapper[2561]: E0723 12:23:18.351027 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1620a6937d38e8ce9da53746c4a5462c6dcbc8fe94ab7e5ca1c37cbe20a808c\": container with ID starting with d1620a6937d38e8ce9da53746c4a5462c6dcbc8fe94ab7e5ca1c37cbe20a808c not found: ID does not exist" containerID="d1620a6937d38e8ce9da53746c4a5462c6dcbc8fe94ab7e5ca1c37cbe20a808c" Jul 23 12:23:18.351117 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:18.351055 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1620a6937d38e8ce9da53746c4a5462c6dcbc8fe94ab7e5ca1c37cbe20a808c"} err="failed to get container status \"d1620a6937d38e8ce9da53746c4a5462c6dcbc8fe94ab7e5ca1c37cbe20a808c\": rpc error: code = NotFound desc = could not find container \"d1620a6937d38e8ce9da53746c4a5462c6dcbc8fe94ab7e5ca1c37cbe20a808c\": container with ID starting with d1620a6937d38e8ce9da53746c4a5462c6dcbc8fe94ab7e5ca1c37cbe20a808c not found: ID does not exist" Jul 23 12:23:18.364901 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:18.364869 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-1e1abd/rate-test-model-1e1abd-kserve-74476555d7-cxkqb"] Jul 23 12:23:18.373558 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:18.373534 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-1e1abd/rate-test-model-1e1abd-kserve-74476555d7-cxkqb"] Jul 23 12:23:19.455018 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:19.454972 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7915910-0559-46db-bfb8-51771724d937" path="/var/lib/kubelet/pods/f7915910-0559-46db-bfb8-51771724d937/volumes" Jul 23 12:23:21.913051 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:21.913007 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-290648-csqcl"] Jul 23 12:23:21.913451 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:21.913359 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7915910-0559-46db-bfb8-51771724d937" containerName="main" Jul 23 12:23:21.913451 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:21.913371 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7915910-0559-46db-bfb8-51771724d937" containerName="main" Jul 23 12:23:21.913451 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:21.913432 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7915910-0559-46db-bfb8-51771724d937" containerName="main" Jul 23 12:23:21.918055 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:21.918037 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-290648-csqcl" Jul 23 12:23:21.921051 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:21.921020 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q8m5p\"" Jul 23 12:23:21.927637 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:21.927609 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-290648-csqcl"] Jul 23 12:23:21.968442 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:21.968408 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/860638d1-933d-4f90-bd50-1b47995cf168-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-290648-csqcl\" (UID: \"860638d1-933d-4f90-bd50-1b47995cf168\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-290648-csqcl" Jul 23 12:23:22.069938 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:22.069873 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/860638d1-933d-4f90-bd50-1b47995cf168-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-290648-csqcl\" (UID: \"860638d1-933d-4f90-bd50-1b47995cf168\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-290648-csqcl" Jul 23 12:23:22.070527 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:22.070504 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/860638d1-933d-4f90-bd50-1b47995cf168-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-290648-csqcl\" (UID: \"860638d1-933d-4f90-bd50-1b47995cf168\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-290648-csqcl" Jul 23 12:23:22.229201 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:22.229111 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-290648-csqcl" Jul 23 12:23:22.366074 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:22.366050 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-290648-csqcl"] Jul 23 12:23:23.369425 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:23.369326 2561 generic.go:356] "Generic (PLEG): container finished" podID="860638d1-933d-4f90-bd50-1b47995cf168" containerID="34fc74e6e672a5d8c12ac8a2d7ff631216cde877c3366ea422a2f8147422ac48" exitCode=0 Jul 23 12:23:23.369808 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:23.369419 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-290648-csqcl" event={"ID":"860638d1-933d-4f90-bd50-1b47995cf168","Type":"ContainerDied","Data":"34fc74e6e672a5d8c12ac8a2d7ff631216cde877c3366ea422a2f8147422ac48"} Jul 23 12:23:23.369808 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:23.369473 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-290648-csqcl" event={"ID":"860638d1-933d-4f90-bd50-1b47995cf168","Type":"ContainerStarted","Data":"574f4941705920a35bc38b7b2a95c4ba8a175825fd03f36552d8b25c49140410"} Jul 23 12:23:24.513427 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:24.513400 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-290648-csqcl" Jul 23 12:23:24.593570 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:24.593539 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/860638d1-933d-4f90-bd50-1b47995cf168-maas-api-service-ca\") pod \"860638d1-933d-4f90-bd50-1b47995cf168\" (UID: \"860638d1-933d-4f90-bd50-1b47995cf168\") " Jul 23 12:23:24.593994 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:24.593958 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/860638d1-933d-4f90-bd50-1b47995cf168-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "860638d1-933d-4f90-bd50-1b47995cf168" (UID: "860638d1-933d-4f90-bd50-1b47995cf168"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 12:23:24.694414 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:24.694324 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/860638d1-933d-4f90-bd50-1b47995cf168-maas-api-service-ca\") on node \"ip-10-0-134-236.ec2.internal\" DevicePath \"\"" Jul 23 12:23:25.381177 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:25.381139 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-290648-csqcl" event={"ID":"860638d1-933d-4f90-bd50-1b47995cf168","Type":"ContainerDied","Data":"574f4941705920a35bc38b7b2a95c4ba8a175825fd03f36552d8b25c49140410"} Jul 23 12:23:25.381177 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:25.381159 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-290648-csqcl" Jul 23 12:23:25.381399 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:25.381186 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="574f4941705920a35bc38b7b2a95c4ba8a175825fd03f36552d8b25c49140410" Jul 23 12:23:32.506275 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:32.506237 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-290648-c5686d489-4c5pj"] Jul 23 12:23:32.506755 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:32.506528 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-290648-c5686d489-4c5pj" podUID="e49f098b-169d-45f5-82b5-ded09a4eb10c" containerName="maas-api" containerID="cri-o://3587b49a728fea415ea057e085aa027b31949602317425845966e17279509f74" gracePeriod=30 Jul 23 12:23:32.955869 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:32.955843 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-290648-c5686d489-4c5pj" Jul 23 12:23:33.068682 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:33.068654 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e49f098b-169d-45f5-82b5-ded09a4eb10c-openshift-service-ca\") pod \"e49f098b-169d-45f5-82b5-ded09a4eb10c\" (UID: \"e49f098b-169d-45f5-82b5-ded09a4eb10c\") " Jul 23 12:23:33.068855 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:33.068746 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h8tb8\" (UniqueName: \"kubernetes.io/projected/e49f098b-169d-45f5-82b5-ded09a4eb10c-kube-api-access-h8tb8\") pod \"e49f098b-169d-45f5-82b5-ded09a4eb10c\" (UID: \"e49f098b-169d-45f5-82b5-ded09a4eb10c\") " Jul 23 12:23:33.068855 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:33.068790 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e49f098b-169d-45f5-82b5-ded09a4eb10c-maas-api-tls\") pod \"e49f098b-169d-45f5-82b5-ded09a4eb10c\" (UID: \"e49f098b-169d-45f5-82b5-ded09a4eb10c\") " Jul 23 12:23:33.069147 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:33.069120 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e49f098b-169d-45f5-82b5-ded09a4eb10c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e49f098b-169d-45f5-82b5-ded09a4eb10c" (UID: "e49f098b-169d-45f5-82b5-ded09a4eb10c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 12:23:33.071108 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:33.071078 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e49f098b-169d-45f5-82b5-ded09a4eb10c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e49f098b-169d-45f5-82b5-ded09a4eb10c" (UID: "e49f098b-169d-45f5-82b5-ded09a4eb10c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 12:23:33.071221 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:33.071146 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e49f098b-169d-45f5-82b5-ded09a4eb10c-kube-api-access-h8tb8" (OuterVolumeSpecName: "kube-api-access-h8tb8") pod "e49f098b-169d-45f5-82b5-ded09a4eb10c" (UID: "e49f098b-169d-45f5-82b5-ded09a4eb10c"). InnerVolumeSpecName "kube-api-access-h8tb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 12:23:33.170051 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:33.170019 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8tb8\" (UniqueName: \"kubernetes.io/projected/e49f098b-169d-45f5-82b5-ded09a4eb10c-kube-api-access-h8tb8\") on node \"ip-10-0-134-236.ec2.internal\" DevicePath \"\"" Jul 23 12:23:33.170051 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:33.170048 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e49f098b-169d-45f5-82b5-ded09a4eb10c-maas-api-tls\") on node \"ip-10-0-134-236.ec2.internal\" DevicePath \"\"" Jul 23 12:23:33.170051 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:33.170057 2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e49f098b-169d-45f5-82b5-ded09a4eb10c-openshift-service-ca\") on node \"ip-10-0-134-236.ec2.internal\" DevicePath \"\"" Jul 23 12:23:33.421694 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:33.421595 2561 generic.go:356] "Generic (PLEG): container finished" podID="e49f098b-169d-45f5-82b5-ded09a4eb10c" containerID="3587b49a728fea415ea057e085aa027b31949602317425845966e17279509f74" exitCode=0 Jul 23 12:23:33.421694 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:33.421654 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-290648-c5686d489-4c5pj" Jul 23 12:23:33.421921 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:33.421689 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-290648-c5686d489-4c5pj" event={"ID":"e49f098b-169d-45f5-82b5-ded09a4eb10c","Type":"ContainerDied","Data":"3587b49a728fea415ea057e085aa027b31949602317425845966e17279509f74"} Jul 23 12:23:33.421921 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:33.421742 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-290648-c5686d489-4c5pj" event={"ID":"e49f098b-169d-45f5-82b5-ded09a4eb10c","Type":"ContainerDied","Data":"ee816dbf515c862e189be2588b9b1511829fdb991b414fb9ac2d3edecfe20f0b"} Jul 23 12:23:33.421921 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:33.421766 2561 scope.go:117] "RemoveContainer" containerID="3587b49a728fea415ea057e085aa027b31949602317425845966e17279509f74" Jul 23 12:23:33.431625 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:33.431604 2561 scope.go:117] "RemoveContainer" containerID="3587b49a728fea415ea057e085aa027b31949602317425845966e17279509f74" Jul 23 12:23:33.431937 ip-10-0-134-236 kubenswrapper[2561]: E0723 12:23:33.431897 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3587b49a728fea415ea057e085aa027b31949602317425845966e17279509f74\": container with ID starting with 3587b49a728fea415ea057e085aa027b31949602317425845966e17279509f74 not found: ID does not exist" containerID="3587b49a728fea415ea057e085aa027b31949602317425845966e17279509f74" Jul 23 12:23:33.432028 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:33.431945 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3587b49a728fea415ea057e085aa027b31949602317425845966e17279509f74"} err="failed to get container status \"3587b49a728fea415ea057e085aa027b31949602317425845966e17279509f74\": rpc error: code = NotFound desc = could not find container \"3587b49a728fea415ea057e085aa027b31949602317425845966e17279509f74\": container with ID starting with 3587b49a728fea415ea057e085aa027b31949602317425845966e17279509f74 not found: ID does not exist" Jul 23 12:23:33.455342 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:33.455317 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-290648-c5686d489-4c5pj"] Jul 23 12:23:33.459925 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:33.459887 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-290648-c5686d489-4c5pj"] Jul 23 12:23:35.455258 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:35.455222 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e49f098b-169d-45f5-82b5-ded09a4eb10c" path="/var/lib/kubelet/pods/e49f098b-169d-45f5-82b5-ded09a4eb10c/volumes" Jul 23 12:23:35.636835 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:35.636799 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 12:23:36.998650 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:36.998614 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-290648-csqcl"] Jul 23 12:23:37.005112 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:37.005086 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-290648-csqcl"] Jul 23 12:23:37.454626 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:37.454581 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="860638d1-933d-4f90-bd50-1b47995cf168" path="/var/lib/kubelet/pods/860638d1-933d-4f90-bd50-1b47995cf168/volumes" Jul 23 12:23:44.143999 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:44.143966 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1e1abd-t6fqz"] Jul 23 12:23:44.144567 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:44.144316 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="860638d1-933d-4f90-bd50-1b47995cf168" containerName="revoke-keys" Jul 23 12:23:44.144567 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:44.144328 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="860638d1-933d-4f90-bd50-1b47995cf168" containerName="revoke-keys" Jul 23 12:23:44.144567 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:44.144345 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e49f098b-169d-45f5-82b5-ded09a4eb10c" containerName="maas-api" Jul 23 12:23:44.144567 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:44.144352 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e49f098b-169d-45f5-82b5-ded09a4eb10c" containerName="maas-api" Jul 23 12:23:44.144567 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:44.144416 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="860638d1-933d-4f90-bd50-1b47995cf168" containerName="revoke-keys" Jul 23 12:23:44.144567 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:44.144428 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="e49f098b-169d-45f5-82b5-ded09a4eb10c" containerName="maas-api" Jul 23 12:23:44.147718 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:44.147692 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1e1abd-t6fqz" Jul 23 12:23:44.150815 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:44.150790 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q8m5p\"" Jul 23 12:23:44.157306 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:44.156792 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1e1abd-t6fqz"] Jul 23 12:23:44.267247 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:44.267209 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/152a1d68-04bf-428d-bfc0-a6b1bbbf952b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-1e1abd-t6fqz\" (UID: \"152a1d68-04bf-428d-bfc0-a6b1bbbf952b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1e1abd-t6fqz" Jul 23 12:23:44.368638 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:44.368600 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/152a1d68-04bf-428d-bfc0-a6b1bbbf952b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-1e1abd-t6fqz\" (UID: \"152a1d68-04bf-428d-bfc0-a6b1bbbf952b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1e1abd-t6fqz" Jul 23 12:23:44.369271 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:44.369252 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/152a1d68-04bf-428d-bfc0-a6b1bbbf952b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-1e1abd-t6fqz\" (UID: \"152a1d68-04bf-428d-bfc0-a6b1bbbf952b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1e1abd-t6fqz" Jul 23 12:23:44.461808 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:44.461724 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1e1abd-t6fqz" Jul 23 12:23:44.602578 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:44.602554 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1e1abd-t6fqz"] Jul 23 12:23:44.604730 ip-10-0-134-236 kubenswrapper[2561]: W0723 12:23:44.604703 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod152a1d68_04bf_428d_bfc0_a6b1bbbf952b.slice/crio-6de50aa1ef92fbd5dd44ad33c73b0a0ea93d93d677a7471de856b8fc7395a1c3 WatchSource:0}: Error finding container 6de50aa1ef92fbd5dd44ad33c73b0a0ea93d93d677a7471de856b8fc7395a1c3: Status 404 returned error can't find the container with id 6de50aa1ef92fbd5dd44ad33c73b0a0ea93d93d677a7471de856b8fc7395a1c3 Jul 23 12:23:45.479517 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:45.479484 2561 generic.go:356] "Generic (PLEG): container finished" podID="152a1d68-04bf-428d-bfc0-a6b1bbbf952b" containerID="ad5550fe42b9c021472809d2a714c1344d3a1f11b2091d5e93179ea61c5d8977" exitCode=0 Jul 23 12:23:45.479991 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:45.479546 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1e1abd-t6fqz" event={"ID":"152a1d68-04bf-428d-bfc0-a6b1bbbf952b","Type":"ContainerDied","Data":"ad5550fe42b9c021472809d2a714c1344d3a1f11b2091d5e93179ea61c5d8977"} Jul 23 12:23:45.479991 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:45.479599 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1e1abd-t6fqz" event={"ID":"152a1d68-04bf-428d-bfc0-a6b1bbbf952b","Type":"ContainerStarted","Data":"6de50aa1ef92fbd5dd44ad33c73b0a0ea93d93d677a7471de856b8fc7395a1c3"} Jul 23 12:23:46.614854 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:46.614828 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1e1abd-t6fqz" Jul 23 12:23:46.684414 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:46.684385 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/152a1d68-04bf-428d-bfc0-a6b1bbbf952b-maas-api-service-ca\") pod \"152a1d68-04bf-428d-bfc0-a6b1bbbf952b\" (UID: \"152a1d68-04bf-428d-bfc0-a6b1bbbf952b\") " Jul 23 12:23:46.684750 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:46.684706 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/152a1d68-04bf-428d-bfc0-a6b1bbbf952b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "152a1d68-04bf-428d-bfc0-a6b1bbbf952b" (UID: "152a1d68-04bf-428d-bfc0-a6b1bbbf952b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 12:23:46.785767 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:46.785681 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/152a1d68-04bf-428d-bfc0-a6b1bbbf952b-maas-api-service-ca\") on node \"ip-10-0-134-236.ec2.internal\" DevicePath \"\"" Jul 23 12:23:47.491127 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:47.491102 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1e1abd-t6fqz" Jul 23 12:23:47.491293 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:47.491101 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1e1abd-t6fqz" event={"ID":"152a1d68-04bf-428d-bfc0-a6b1bbbf952b","Type":"ContainerDied","Data":"6de50aa1ef92fbd5dd44ad33c73b0a0ea93d93d677a7471de856b8fc7395a1c3"} Jul 23 12:23:47.491293 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:47.491228 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6de50aa1ef92fbd5dd44ad33c73b0a0ea93d93d677a7471de856b8fc7395a1c3" Jul 23 12:23:54.649143 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:54.649104 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-1e1abd-76d9bdf959-qsfnf"] Jul 23 12:23:54.649629 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:54.649360 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1e1abd-76d9bdf959-qsfnf" podUID="8b6b9b16-f931-4154-ac0f-320cafa0c447" containerName="maas-api" containerID="cri-o://e814d56c2e0234f4cf91e031c13989686d5f5aa8a1c97c0e237e71fd52d972e2" gracePeriod=30 Jul 23 12:23:54.905386 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:54.905325 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1e1abd-76d9bdf959-qsfnf" Jul 23 12:23:54.956007 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:54.955980 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b6b9b16-f931-4154-ac0f-320cafa0c447-openshift-service-ca\") pod \"8b6b9b16-f931-4154-ac0f-320cafa0c447\" (UID: \"8b6b9b16-f931-4154-ac0f-320cafa0c447\") " Jul 23 12:23:54.956184 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:54.956028 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-97nfw\" (UniqueName: \"kubernetes.io/projected/8b6b9b16-f931-4154-ac0f-320cafa0c447-kube-api-access-97nfw\") pod \"8b6b9b16-f931-4154-ac0f-320cafa0c447\" (UID: \"8b6b9b16-f931-4154-ac0f-320cafa0c447\") " Jul 23 12:23:54.956184 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:54.956055 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b6b9b16-f931-4154-ac0f-320cafa0c447-maas-api-tls\") pod \"8b6b9b16-f931-4154-ac0f-320cafa0c447\" (UID: \"8b6b9b16-f931-4154-ac0f-320cafa0c447\") " Jul 23 12:23:54.956347 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:54.956325 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b6b9b16-f931-4154-ac0f-320cafa0c447-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8b6b9b16-f931-4154-ac0f-320cafa0c447" (UID: "8b6b9b16-f931-4154-ac0f-320cafa0c447"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 12:23:54.958253 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:54.958228 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b6b9b16-f931-4154-ac0f-320cafa0c447-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8b6b9b16-f931-4154-ac0f-320cafa0c447" (UID: "8b6b9b16-f931-4154-ac0f-320cafa0c447"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 12:23:54.958379 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:54.958364 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b6b9b16-f931-4154-ac0f-320cafa0c447-kube-api-access-97nfw" (OuterVolumeSpecName: "kube-api-access-97nfw") pod "8b6b9b16-f931-4154-ac0f-320cafa0c447" (UID: "8b6b9b16-f931-4154-ac0f-320cafa0c447"). InnerVolumeSpecName "kube-api-access-97nfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 12:23:55.056938 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:55.056882 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97nfw\" (UniqueName: \"kubernetes.io/projected/8b6b9b16-f931-4154-ac0f-320cafa0c447-kube-api-access-97nfw\") on node \"ip-10-0-134-236.ec2.internal\" DevicePath \"\"" Jul 23 12:23:55.056938 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:55.056941 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b6b9b16-f931-4154-ac0f-320cafa0c447-maas-api-tls\") on node \"ip-10-0-134-236.ec2.internal\" DevicePath \"\"" Jul 23 12:23:55.057131 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:55.056954 2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b6b9b16-f931-4154-ac0f-320cafa0c447-openshift-service-ca\") on node \"ip-10-0-134-236.ec2.internal\" DevicePath \"\"" Jul 23 12:23:55.533412 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:55.533374 2561 generic.go:356] "Generic (PLEG): container finished" podID="8b6b9b16-f931-4154-ac0f-320cafa0c447" containerID="e814d56c2e0234f4cf91e031c13989686d5f5aa8a1c97c0e237e71fd52d972e2" exitCode=0 Jul 23 12:23:55.533593 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:55.533423 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1e1abd-76d9bdf959-qsfnf" Jul 23 12:23:55.533593 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:55.533430 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1e1abd-76d9bdf959-qsfnf" event={"ID":"8b6b9b16-f931-4154-ac0f-320cafa0c447","Type":"ContainerDied","Data":"e814d56c2e0234f4cf91e031c13989686d5f5aa8a1c97c0e237e71fd52d972e2"} Jul 23 12:23:55.533593 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:55.533483 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1e1abd-76d9bdf959-qsfnf" event={"ID":"8b6b9b16-f931-4154-ac0f-320cafa0c447","Type":"ContainerDied","Data":"d82bde6a3d18499ccfe10244dc5b285c56261aff735bcfdcd00830a3e4a8f01b"} Jul 23 12:23:55.533593 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:55.533515 2561 scope.go:117] "RemoveContainer" containerID="e814d56c2e0234f4cf91e031c13989686d5f5aa8a1c97c0e237e71fd52d972e2" Jul 23 12:23:55.542683 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:55.542660 2561 scope.go:117] "RemoveContainer" containerID="e814d56c2e0234f4cf91e031c13989686d5f5aa8a1c97c0e237e71fd52d972e2" Jul 23 12:23:55.542958 ip-10-0-134-236 kubenswrapper[2561]: E0723 12:23:55.542938 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e814d56c2e0234f4cf91e031c13989686d5f5aa8a1c97c0e237e71fd52d972e2\": container with ID starting with e814d56c2e0234f4cf91e031c13989686d5f5aa8a1c97c0e237e71fd52d972e2 not found: ID does not exist" containerID="e814d56c2e0234f4cf91e031c13989686d5f5aa8a1c97c0e237e71fd52d972e2" Jul 23 12:23:55.543016 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:55.542967 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e814d56c2e0234f4cf91e031c13989686d5f5aa8a1c97c0e237e71fd52d972e2"} err="failed to get container status \"e814d56c2e0234f4cf91e031c13989686d5f5aa8a1c97c0e237e71fd52d972e2\": rpc error: code = NotFound desc = could not find container \"e814d56c2e0234f4cf91e031c13989686d5f5aa8a1c97c0e237e71fd52d972e2\": container with ID starting with e814d56c2e0234f4cf91e031c13989686d5f5aa8a1c97c0e237e71fd52d972e2 not found: ID does not exist" Jul 23 12:23:55.550317 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:55.550291 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-1e1abd-76d9bdf959-qsfnf"] Jul 23 12:23:55.554125 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:55.554101 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-1e1abd-76d9bdf959-qsfnf"] Jul 23 12:23:57.457104 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:57.455580 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b6b9b16-f931-4154-ac0f-320cafa0c447" path="/var/lib/kubelet/pods/8b6b9b16-f931-4154-ac0f-320cafa0c447/volumes" Jul 23 12:23:57.642888 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:57.642854 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-t7jsv"] Jul 23 12:23:59.237030 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:59.236997 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1e1abd-t6fqz"] Jul 23 12:23:59.239636 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:59.239609 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1e1abd-t6fqz"] Jul 23 12:23:59.454707 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:23:59.454673 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="152a1d68-04bf-428d-bfc0-a6b1bbbf952b" path="/var/lib/kubelet/pods/152a1d68-04bf-428d-bfc0-a6b1bbbf952b/volumes" Jul 23 12:24:13.685042 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:13.685009 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7c7646c6cc-xnlnm_d2b3b67b-5cd8-4428-88f1-486ae20e1ead/copy-manifests/0.log" Jul 23 12:24:13.703851 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:13.703802 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7c7646c6cc-xnlnm_d2b3b67b-5cd8-4428-88f1-486ae20e1ead/manager/0.log" Jul 23 12:24:14.197141 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:14.197110 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29746800-wtftr_35e2b33c-01af-47d6-81b1-2a68d9d244e0/cleanup/0.log" Jul 23 12:24:14.311050 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:14.311023 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29746815-9m4r5_c10604aa-cbbb-40e5-8c7d-8e1c16cab1db/cleanup/0.log" Jul 23 12:24:15.260425 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:15.260391 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nj7wg_ffef83f1-4832-40f6-a8ca-8e62bc8db03b/util/0.log" Jul 23 12:24:15.267326 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:15.267298 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nj7wg_ffef83f1-4832-40f6-a8ca-8e62bc8db03b/pull/0.log" Jul 23 12:24:15.273362 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:15.273341 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nj7wg_ffef83f1-4832-40f6-a8ca-8e62bc8db03b/extract/0.log" Jul 23 12:24:15.393856 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:15.393826 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zhkzz_2a8af5b7-35f7-4208-bf9f-f58d30954ae0/util/0.log" Jul 23 12:24:15.400027 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:15.399998 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zhkzz_2a8af5b7-35f7-4208-bf9f-f58d30954ae0/pull/0.log" Jul 23 12:24:15.405960 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:15.405935 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zhkzz_2a8af5b7-35f7-4208-bf9f-f58d30954ae0/extract/0.log" Jul 23 12:24:15.523545 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:15.523456 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73b2dck_0748219c-2587-496d-a3c1-b218b6748b7c/util/0.log" Jul 23 12:24:15.533187 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:15.533154 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73b2dck_0748219c-2587-496d-a3c1-b218b6748b7c/pull/0.log" Jul 23 12:24:15.539204 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:15.539179 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73b2dck_0748219c-2587-496d-a3c1-b218b6748b7c/extract/0.log" Jul 23 12:24:15.662528 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:15.662495 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xz224_b4233887-c309-4a1f-9759-363bc6092fd9/util/0.log" Jul 23 12:24:15.669254 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:15.669224 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xz224_b4233887-c309-4a1f-9759-363bc6092fd9/pull/0.log" Jul 23 12:24:15.675793 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:15.675757 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xz224_b4233887-c309-4a1f-9759-363bc6092fd9/extract/0.log" Jul 23 12:24:16.542606 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:16.542572 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-t7jsv_5fbe789f-5522-405d-a998-b7f4181d255a/limitador/0.log" Jul 23 12:24:17.191777 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:17.191747 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-d67vc_ceaf1c1b-b54f-48f9-bcaf-58a505669805/discovery/0.log" Jul 23 12:24:17.936780 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:17.936752 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-78fdc88b76-5w9ql_c5179eda-de1f-4d89-9e22-afb9d63d098b/router/0.log" Jul 23 12:24:18.685193 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:18.685163 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fr9r68_6aecbb33-e47a-41f3-b284-9f323c93ff41/main/0.log" Jul 23 12:24:28.293407 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:28.293374 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-8df69_1e281f98-6114-4c25-9b31-79c65f9100e2/global-pull-secret-syncer/0.log" Jul 23 12:24:28.522077 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:28.522037 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-lp5f4_620c5e1f-97de-42e6-b752-50d951728a5d/konnectivity-agent/0.log" Jul 23 12:24:28.544772 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:28.544683 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-236.ec2.internal_f6f6dd6dc338193833341a6aab50de06/haproxy/0.log" Jul 23 12:24:33.259984 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:33.259948 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nj7wg_ffef83f1-4832-40f6-a8ca-8e62bc8db03b/extract/0.log" Jul 23 12:24:33.284414 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:33.284383 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nj7wg_ffef83f1-4832-40f6-a8ca-8e62bc8db03b/util/0.log" Jul 23 12:24:33.308186 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:33.308158 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759nj7wg_ffef83f1-4832-40f6-a8ca-8e62bc8db03b/pull/0.log" Jul 23 12:24:33.340014 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:33.339975 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zhkzz_2a8af5b7-35f7-4208-bf9f-f58d30954ae0/extract/0.log" Jul 23 12:24:33.362708 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:33.362620 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zhkzz_2a8af5b7-35f7-4208-bf9f-f58d30954ae0/util/0.log" Jul 23 12:24:33.389602 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:33.389572 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0zhkzz_2a8af5b7-35f7-4208-bf9f-f58d30954ae0/pull/0.log" Jul 23 12:24:33.426741 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:33.426717 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73b2dck_0748219c-2587-496d-a3c1-b218b6748b7c/extract/0.log" Jul 23 12:24:33.458060 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:33.458030 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73b2dck_0748219c-2587-496d-a3c1-b218b6748b7c/util/0.log" Jul 23 12:24:33.482889 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:33.482854 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73b2dck_0748219c-2587-496d-a3c1-b218b6748b7c/pull/0.log" Jul 23 12:24:33.516146 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:33.516118 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xz224_b4233887-c309-4a1f-9759-363bc6092fd9/extract/0.log" Jul 23 12:24:33.548871 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:33.548835 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xz224_b4233887-c309-4a1f-9759-363bc6092fd9/util/0.log" Jul 23 12:24:33.572836 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:33.572808 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1xz224_b4233887-c309-4a1f-9759-363bc6092fd9/pull/0.log" Jul 23 12:24:34.493225 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:34.493195 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-t7jsv_5fbe789f-5522-405d-a998-b7f4181d255a/limitador/0.log" Jul 23 12:24:36.600190 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:36.600151 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-j4sbd_f249368d-c49d-4bfe-906f-d1d12104dbd0/cluster-monitoring-operator/0.log" Jul 23 12:24:37.100999 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:37.100967 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ptljk_bd1684db-f985-4971-b821-eb4a6b9a1b50/node-exporter/0.log" Jul 23 12:24:37.128514 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:37.128476 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ptljk_bd1684db-f985-4971-b821-eb4a6b9a1b50/kube-rbac-proxy/0.log" Jul 23 12:24:37.150989 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:37.150957 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ptljk_bd1684db-f985-4971-b821-eb4a6b9a1b50/init-textfile/0.log" Jul 23 12:24:38.742637 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:38.742598 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5jgf7/perf-node-gather-daemonset-dlwn8"] Jul 23 12:24:38.743065 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:38.742933 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b6b9b16-f931-4154-ac0f-320cafa0c447" containerName="maas-api" Jul 23 12:24:38.743065 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:38.742945 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b6b9b16-f931-4154-ac0f-320cafa0c447" containerName="maas-api" Jul 23 12:24:38.743065 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:38.742954 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="152a1d68-04bf-428d-bfc0-a6b1bbbf952b" containerName="revoke-keys" Jul 23 12:24:38.743065 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:38.742959 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="152a1d68-04bf-428d-bfc0-a6b1bbbf952b" containerName="revoke-keys" Jul 23 12:24:38.743065 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:38.743027 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b6b9b16-f931-4154-ac0f-320cafa0c447" containerName="maas-api" Jul 23 12:24:38.743065 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:38.743036 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="152a1d68-04bf-428d-bfc0-a6b1bbbf952b" containerName="revoke-keys" Jul 23 12:24:38.747339 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:38.747316 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5jgf7/perf-node-gather-daemonset-dlwn8" Jul 23 12:24:38.749889 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:38.749861 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5jgf7\"/\"openshift-service-ca.crt\"" Jul 23 12:24:38.749889 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:38.749863 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5jgf7\"/\"kube-root-ca.crt\"" Jul 23 12:24:38.750137 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:38.749945 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5jgf7\"/\"default-dockercfg-st8ws\"" Jul 23 12:24:38.757755 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:38.757730 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5jgf7/perf-node-gather-daemonset-dlwn8"] Jul 23 12:24:38.841133 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:38.841092 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c34d1831-91b5-4927-a3b8-3668842356e2-sys\") pod \"perf-node-gather-daemonset-dlwn8\" (UID: \"c34d1831-91b5-4927-a3b8-3668842356e2\") " pod="openshift-must-gather-5jgf7/perf-node-gather-daemonset-dlwn8" Jul 23 12:24:38.841311 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:38.841140 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c34d1831-91b5-4927-a3b8-3668842356e2-lib-modules\") pod \"perf-node-gather-daemonset-dlwn8\" (UID: \"c34d1831-91b5-4927-a3b8-3668842356e2\") " pod="openshift-must-gather-5jgf7/perf-node-gather-daemonset-dlwn8" Jul 23 12:24:38.841311 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:38.841197 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c34d1831-91b5-4927-a3b8-3668842356e2-proc\") pod \"perf-node-gather-daemonset-dlwn8\" (UID: \"c34d1831-91b5-4927-a3b8-3668842356e2\") " pod="openshift-must-gather-5jgf7/perf-node-gather-daemonset-dlwn8" Jul 23 12:24:38.841311 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:38.841247 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c34d1831-91b5-4927-a3b8-3668842356e2-podres\") pod \"perf-node-gather-daemonset-dlwn8\" (UID: \"c34d1831-91b5-4927-a3b8-3668842356e2\") " pod="openshift-must-gather-5jgf7/perf-node-gather-daemonset-dlwn8" Jul 23 12:24:38.841311 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:38.841264 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tzq7\" (UniqueName: \"kubernetes.io/projected/c34d1831-91b5-4927-a3b8-3668842356e2-kube-api-access-8tzq7\") pod \"perf-node-gather-daemonset-dlwn8\" (UID: \"c34d1831-91b5-4927-a3b8-3668842356e2\") " pod="openshift-must-gather-5jgf7/perf-node-gather-daemonset-dlwn8" Jul 23 12:24:38.942198 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:38.942158 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c34d1831-91b5-4927-a3b8-3668842356e2-sys\") pod \"perf-node-gather-daemonset-dlwn8\" (UID: \"c34d1831-91b5-4927-a3b8-3668842356e2\") " pod="openshift-must-gather-5jgf7/perf-node-gather-daemonset-dlwn8" Jul 23 12:24:38.942198 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:38.942201 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c34d1831-91b5-4927-a3b8-3668842356e2-lib-modules\") pod \"perf-node-gather-daemonset-dlwn8\" (UID: \"c34d1831-91b5-4927-a3b8-3668842356e2\") " pod="openshift-must-gather-5jgf7/perf-node-gather-daemonset-dlwn8" Jul 23 12:24:38.942459 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:38.942228 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c34d1831-91b5-4927-a3b8-3668842356e2-proc\") pod \"perf-node-gather-daemonset-dlwn8\" (UID: \"c34d1831-91b5-4927-a3b8-3668842356e2\") " pod="openshift-must-gather-5jgf7/perf-node-gather-daemonset-dlwn8" Jul 23 12:24:38.942459 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:38.942275 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c34d1831-91b5-4927-a3b8-3668842356e2-podres\") pod \"perf-node-gather-daemonset-dlwn8\" (UID: \"c34d1831-91b5-4927-a3b8-3668842356e2\") " pod="openshift-must-gather-5jgf7/perf-node-gather-daemonset-dlwn8" Jul 23 12:24:38.942459 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:38.942298 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tzq7\" (UniqueName: \"kubernetes.io/projected/c34d1831-91b5-4927-a3b8-3668842356e2-kube-api-access-8tzq7\") pod \"perf-node-gather-daemonset-dlwn8\" (UID: \"c34d1831-91b5-4927-a3b8-3668842356e2\") " pod="openshift-must-gather-5jgf7/perf-node-gather-daemonset-dlwn8" Jul 23 12:24:38.942459 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:38.942303 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c34d1831-91b5-4927-a3b8-3668842356e2-sys\") pod \"perf-node-gather-daemonset-dlwn8\" (UID: \"c34d1831-91b5-4927-a3b8-3668842356e2\") " pod="openshift-must-gather-5jgf7/perf-node-gather-daemonset-dlwn8" Jul 23 12:24:38.942459 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:38.942363 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c34d1831-91b5-4927-a3b8-3668842356e2-lib-modules\") pod \"perf-node-gather-daemonset-dlwn8\" (UID: \"c34d1831-91b5-4927-a3b8-3668842356e2\") " pod="openshift-must-gather-5jgf7/perf-node-gather-daemonset-dlwn8" Jul 23 12:24:38.942459 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:38.942383 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c34d1831-91b5-4927-a3b8-3668842356e2-proc\") pod \"perf-node-gather-daemonset-dlwn8\" (UID: \"c34d1831-91b5-4927-a3b8-3668842356e2\") " pod="openshift-must-gather-5jgf7/perf-node-gather-daemonset-dlwn8" Jul 23 12:24:38.942459 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:38.942454 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c34d1831-91b5-4927-a3b8-3668842356e2-podres\") pod \"perf-node-gather-daemonset-dlwn8\" (UID: \"c34d1831-91b5-4927-a3b8-3668842356e2\") " pod="openshift-must-gather-5jgf7/perf-node-gather-daemonset-dlwn8" Jul 23 12:24:38.950649 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:38.950622 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tzq7\" (UniqueName: \"kubernetes.io/projected/c34d1831-91b5-4927-a3b8-3668842356e2-kube-api-access-8tzq7\") pod \"perf-node-gather-daemonset-dlwn8\" (UID: \"c34d1831-91b5-4927-a3b8-3668842356e2\") " pod="openshift-must-gather-5jgf7/perf-node-gather-daemonset-dlwn8" Jul 23 12:24:39.059112 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:39.059020 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5jgf7/perf-node-gather-daemonset-dlwn8" Jul 23 12:24:39.196117 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:39.196037 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5jgf7/perf-node-gather-daemonset-dlwn8"] Jul 23 12:24:39.198146 ip-10-0-134-236 kubenswrapper[2561]: W0723 12:24:39.198119 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc34d1831_91b5_4927_a3b8_3668842356e2.slice/crio-cdf0de5419aba494fd025d8323c3b138d0ca0aa636dbd748e444c867d04ee4f4 WatchSource:0}: Error finding container cdf0de5419aba494fd025d8323c3b138d0ca0aa636dbd748e444c867d04ee4f4: Status 404 returned error can't find the container with id cdf0de5419aba494fd025d8323c3b138d0ca0aa636dbd748e444c867d04ee4f4 Jul 23 12:24:39.583668 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:39.583638 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-7slpj_65247e1f-fc10-4b40-af67-5e646f352263/console-operator/1.log" Jul 23 12:24:39.592505 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:39.592479 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-7slpj_65247e1f-fc10-4b40-af67-5e646f352263/console-operator/2.log" Jul 23 12:24:39.758222 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:39.758184 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5jgf7/perf-node-gather-daemonset-dlwn8" event={"ID":"c34d1831-91b5-4927-a3b8-3668842356e2","Type":"ContainerStarted","Data":"e22d96242c0e757f4e64143a805ec761d85d60d21312f6f06840e36d01dca6b9"} Jul 23 12:24:39.758712 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:39.758245 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5jgf7/perf-node-gather-daemonset-dlwn8" event={"ID":"c34d1831-91b5-4927-a3b8-3668842356e2","Type":"ContainerStarted","Data":"cdf0de5419aba494fd025d8323c3b138d0ca0aa636dbd748e444c867d04ee4f4"} Jul 23 12:24:39.758712 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:39.758316 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-5jgf7/perf-node-gather-daemonset-dlwn8" Jul 23 12:24:39.775858 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:39.775801 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5jgf7/perf-node-gather-daemonset-dlwn8" podStartSLOduration=1.7757843979999999 podStartE2EDuration="1.775784398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 12:24:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 12:24:39.773804821 +0000 UTC m=+2946.845494814" watchObservedRunningTime="2026-07-23 12:24:39.775784398 +0000 UTC m=+2946.847474389" Jul 23 12:24:40.735936 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:40.735891 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78cf4c6f6-8gtcv_52ccdb9c-03da-48cf-94f4-a916e42975ac/volume-data-source-validator/0.log" Jul 23 12:24:41.845721 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:41.845690 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fkvjh_74038c52-71a3-4c88-b607-7f0f9ec11001/dns/0.log" Jul 23 12:24:41.867226 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:41.867193 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fkvjh_74038c52-71a3-4c88-b607-7f0f9ec11001/kube-rbac-proxy/0.log" Jul 23 12:24:41.987758 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:41.987727 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-pkwm5_a17ee9c0-c387-4e9d-a6a4-481501f5b97b/dns-node-resolver/0.log" Jul 23 12:24:42.595538 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:42.595508 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6cbrx_aa374cc8-936e-41ca-a905-c0449df6825d/node-ca/0.log" Jul 23 12:24:43.851634 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:43.851595 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-d67vc_ceaf1c1b-b54f-48f9-bcaf-58a505669805/discovery/0.log" Jul 23 12:24:44.076520 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:44.076493 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-78fdc88b76-5w9ql_c5179eda-de1f-4d89-9e22-afb9d63d098b/router/0.log" Jul 23 12:24:44.813646 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:44.813614 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-fww69_a46e320d-ae78-4a40-9314-075f03b0b6ce/serve-healthcheck-canary/0.log" Jul 23 12:24:45.291491 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:45.291375 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-5txgl_21319854-f159-442b-bf88-4498cec86096/insights-operator/0.log" Jul 23 12:24:45.295119 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:45.295083 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-5txgl_21319854-f159-442b-bf88-4498cec86096/insights-operator/1.log" Jul 23 12:24:45.315018 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:45.314991 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4rfbr_1ae0a33f-4691-4a97-aea8-70267e2a8057/kube-rbac-proxy/0.log" Jul 23 12:24:45.337638 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:45.337614 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4rfbr_1ae0a33f-4691-4a97-aea8-70267e2a8057/exporter/0.log" Jul 23 12:24:45.358170 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:45.358136 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4rfbr_1ae0a33f-4691-4a97-aea8-70267e2a8057/extractor/0.log" Jul 23 12:24:45.774843 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:45.774816 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-5jgf7/perf-node-gather-daemonset-dlwn8" Jul 23 12:24:48.564675 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:48.564645 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7c7646c6cc-xnlnm_d2b3b67b-5cd8-4428-88f1-486ae20e1ead/manager/0.log" Jul 23 12:24:48.581259 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:48.581234 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7c7646c6cc-xnlnm_d2b3b67b-5cd8-4428-88f1-486ae20e1ead/copy-manifests/0.log" Jul 23 12:24:49.910158 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:49.910131 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-544c65c659-j4mgd_9c120a07-26e4-46b8-a3d3-d031c4a0c7d3/manager/0.log" Jul 23 12:24:50.476863 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:50.476769 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-d67vc_ceaf1c1b-b54f-48f9-bcaf-58a505669805/discovery/0.log" Jul 23 12:24:50.618562 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:50.618529 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-78fdc88b76-5w9ql_c5179eda-de1f-4d89-9e22-afb9d63d098b/router/0.log" Jul 23 12:24:52.028095 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:52.028059 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-d67vc_ceaf1c1b-b54f-48f9-bcaf-58a505669805/discovery/0.log" Jul 23 12:24:52.195817 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:52.195781 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-78fdc88b76-5w9ql_c5179eda-de1f-4d89-9e22-afb9d63d098b/router/0.log" Jul 23 12:24:55.052153 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:55.052121 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-zq4l7_57491661-d002-4cf8-8437-77a7637d4a24/migrator/0.log" Jul 23 12:24:55.083620 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:55.083588 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-zq4l7_57491661-d002-4cf8-8437-77a7637d4a24/graceful-termination/0.log" Jul 23 12:24:55.452093 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:55.452057 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-48j4r_b64d0632-5844-419e-9f53-ac57a0495a8d/kube-storage-version-migrator-operator/1.log" Jul 23 12:24:55.453982 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:55.453942 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-48j4r_b64d0632-5844-419e-9f53-ac57a0495a8d/kube-storage-version-migrator-operator/0.log" Jul 23 12:24:56.503198 ip-10-0-134-236 kubenswrapper[2561]: I0723 12:24:56.503161 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-8d65t_792c59f0-f23f-4694-9cde-37b9431760f4/kube-multus/0.log"