Jul 09 16:13:24.853631 ip-10-0-135-114 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 09 16:13:24.853643 ip-10-0-135-114 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 09 16:13:24.853650 ip-10-0-135-114 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 09 16:13:24.853907 ip-10-0-135-114 systemd[1]: Failed to start Kubernetes Kubelet. Jul 09 16:13:34.991279 ip-10-0-135-114 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 09 16:13:34.991302 ip-10-0-135-114 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 559577674fd9419c91cbce52fe43298d -- Jul 09 16:15:14.675234 ip-10-0-135-114 systemd[1]: Starting Kubernetes Kubelet... Jul 09 16:15:15.274904 ip-10-0-135-114 kubenswrapper[2569]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 09 16:15:15.274904 ip-10-0-135-114 kubenswrapper[2569]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 09 16:15:15.274904 ip-10-0-135-114 kubenswrapper[2569]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 09 16:15:15.274904 ip-10-0-135-114 kubenswrapper[2569]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 09 16:15:15.274904 ip-10-0-135-114 kubenswrapper[2569]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 09 16:15:15.274904 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.127798 2569 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 09 16:15:15.274904 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132008 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 16:15:15.274904 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132022 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 16:15:15.274904 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132026 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 16:15:15.274904 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132029 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 16:15:15.274904 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132031 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 16:15:15.274904 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132034 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 16:15:15.274904 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132037 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 16:15:15.274904 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132040 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 16:15:15.212718 ip-10-0-135-114 systemd[1]: Started Kubernetes Kubelet. Jul 09 16:15:15.277519 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132042 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 16:15:15.277519 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132045 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 16:15:15.277519 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132048 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 16:15:15.277519 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132050 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 16:15:15.277519 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132053 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 16:15:15.277519 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132056 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 16:15:15.277519 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132062 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 16:15:15.277519 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132065 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 16:15:15.277519 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132068 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 16:15:15.277519 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132071 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 16:15:15.277519 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132075 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 16:15:15.277519 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132078 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 16:15:15.277519 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132081 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 16:15:15.277519 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132084 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 16:15:15.277519 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132087 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 16:15:15.277519 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132091 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 16:15:15.277519 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132093 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 16:15:15.277519 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132096 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 16:15:15.277519 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132099 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 16:15:15.277519 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132101 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 16:15:15.279571 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132104 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 16:15:15.279571 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132106 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 16:15:15.279571 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132109 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 16:15:15.279571 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132112 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 16:15:15.279571 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132114 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 16:15:15.279571 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132117 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 16:15:15.279571 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132119 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 16:15:15.279571 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132122 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 16:15:15.279571 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132124 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 16:15:15.279571 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132127 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 16:15:15.279571 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132132 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 16:15:15.279571 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132136 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 16:15:15.279571 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132139 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 16:15:15.279571 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132142 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 16:15:15.279571 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132145 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 16:15:15.279571 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132148 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 16:15:15.279571 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132150 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 16:15:15.279571 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132153 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 16:15:15.279571 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132155 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 16:15:15.279571 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132158 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 16:15:15.300405 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132160 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 16:15:15.300405 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132163 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 16:15:15.300405 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132165 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 16:15:15.300405 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132169 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 16:15:15.300405 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132171 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 16:15:15.300405 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132174 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 16:15:15.300405 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132176 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 16:15:15.300405 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132179 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 16:15:15.300405 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132182 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 16:15:15.300405 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132186 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 16:15:15.300405 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132190 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 16:15:15.300405 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132193 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 16:15:15.300405 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132196 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 16:15:15.300405 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132199 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 16:15:15.300405 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132202 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 16:15:15.300405 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132205 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 16:15:15.300405 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132208 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 16:15:15.300405 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132211 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 16:15:15.300405 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132213 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 16:15:15.300405 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132216 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 16:15:15.311298 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132218 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 16:15:15.311298 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132222 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 16:15:15.311298 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132224 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 16:15:15.311298 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132227 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 16:15:15.311298 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132230 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 16:15:15.311298 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132232 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 16:15:15.311298 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132235 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 16:15:15.311298 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132238 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 16:15:15.311298 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132241 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 09 16:15:15.311298 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132244 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 16:15:15.311298 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132246 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 16:15:15.311298 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132250 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 16:15:15.311298 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132253 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 16:15:15.311298 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132256 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 16:15:15.311298 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132261 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 16:15:15.311298 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132264 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 16:15:15.311298 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132267 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 16:15:15.311298 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132270 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 16:15:15.311298 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132272 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 16:15:15.313461 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132275 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 16:15:15.313461 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132278 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 16:15:15.313461 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132281 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 16:15:15.313461 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132284 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 16:15:15.313461 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132287 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 16:15:15.313461 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132289 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 16:15:15.313461 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132293 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 16:15:15.313461 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132296 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 16:15:15.313461 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132298 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 16:15:15.313461 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132301 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 16:15:15.313461 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132303 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 16:15:15.313461 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.132306 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 16:15:15.313461 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133056 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 16:15:15.313461 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133063 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 16:15:15.313461 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133066 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 16:15:15.313461 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133069 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 16:15:15.313461 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133072 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 16:15:15.313461 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133075 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 16:15:15.313461 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133078 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 16:15:15.313461 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133080 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 16:15:15.320674 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133083 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 16:15:15.320674 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133086 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 16:15:15.320674 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133089 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 16:15:15.320674 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133092 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 16:15:15.320674 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133095 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 16:15:15.320674 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133097 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 16:15:15.320674 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133100 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 16:15:15.320674 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133102 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 16:15:15.320674 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133105 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 16:15:15.320674 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133108 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 16:15:15.320674 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133111 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 16:15:15.320674 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133114 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 16:15:15.320674 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133116 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 16:15:15.320674 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133119 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 16:15:15.320674 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133121 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 16:15:15.320674 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133124 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 16:15:15.320674 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133126 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 16:15:15.320674 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133129 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 16:15:15.320674 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133132 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 16:15:15.321749 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133137 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 16:15:15.321749 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133141 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 16:15:15.321749 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133144 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 16:15:15.321749 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133147 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 16:15:15.321749 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133150 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 16:15:15.321749 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133153 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 16:15:15.321749 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133155 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 16:15:15.321749 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133158 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 16:15:15.321749 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133161 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 16:15:15.321749 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133167 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 16:15:15.321749 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133169 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 16:15:15.321749 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133172 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 16:15:15.321749 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133175 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 16:15:15.321749 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133178 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 16:15:15.321749 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133181 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 16:15:15.321749 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133184 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 16:15:15.321749 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133187 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 16:15:15.321749 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133190 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 16:15:15.321749 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133193 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 16:15:15.321749 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133196 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 16:15:15.322641 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133199 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 16:15:15.322641 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133202 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 16:15:15.322641 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133205 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 16:15:15.322641 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133207 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 16:15:15.322641 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133210 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 16:15:15.322641 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133212 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 16:15:15.322641 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133215 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 16:15:15.322641 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133217 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 16:15:15.322641 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133220 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 16:15:15.322641 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133223 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 16:15:15.322641 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133225 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 16:15:15.322641 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133228 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 16:15:15.322641 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133230 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 16:15:15.322641 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133233 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 16:15:15.322641 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133236 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 16:15:15.322641 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133238 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 16:15:15.322641 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133242 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 16:15:15.322641 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133244 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 16:15:15.322641 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133248 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 16:15:15.322641 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133251 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 16:15:15.323740 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133253 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 16:15:15.323740 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133257 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 16:15:15.323740 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133260 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 16:15:15.323740 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133263 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 16:15:15.323740 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133267 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 16:15:15.323740 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133270 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 09 16:15:15.323740 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133272 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 16:15:15.323740 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133277 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 16:15:15.323740 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133280 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 16:15:15.323740 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133283 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 16:15:15.323740 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133287 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 16:15:15.323740 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133290 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 16:15:15.323740 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133293 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 16:15:15.323740 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133296 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 16:15:15.323740 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133299 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 16:15:15.323740 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133302 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 16:15:15.323740 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133304 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 16:15:15.323740 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133307 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 16:15:15.323740 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133310 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 16:15:15.324638 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133312 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 16:15:15.324638 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133315 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 16:15:15.324638 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133317 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 16:15:15.324638 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133320 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 16:15:15.324638 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133322 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 16:15:15.324638 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133325 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 16:15:15.324638 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133327 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 16:15:15.324638 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133330 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 16:15:15.324638 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133332 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 16:15:15.324638 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133335 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 16:15:15.324638 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133337 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 16:15:15.324638 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133340 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 16:15:15.324638 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133342 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 16:15:15.324638 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133730 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 16:15:15.324638 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133737 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 16:15:15.324638 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133740 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 16:15:15.324638 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133742 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 16:15:15.324638 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133745 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 16:15:15.324638 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133748 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 16:15:15.324638 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133750 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 16:15:15.325740 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133757 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 16:15:15.325740 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133760 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 16:15:15.325740 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133763 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 16:15:15.325740 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133766 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 16:15:15.325740 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133769 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 16:15:15.325740 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133771 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 16:15:15.325740 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133774 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 16:15:15.325740 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133777 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 16:15:15.325740 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133779 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 16:15:15.325740 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133782 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 16:15:15.325740 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133785 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 16:15:15.325740 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133787 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 16:15:15.325740 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133790 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 16:15:15.325740 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133793 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 16:15:15.325740 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133795 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 16:15:15.325740 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133798 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 16:15:15.325740 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133801 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 16:15:15.325740 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133804 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 09 16:15:15.325740 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133808 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 16:15:15.326639 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133811 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 16:15:15.326639 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133815 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 16:15:15.326639 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133818 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 16:15:15.326639 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133821 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 16:15:15.326639 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133824 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 16:15:15.326639 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133827 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 16:15:15.326639 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133830 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 16:15:15.326639 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133834 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 16:15:15.326639 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133837 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 16:15:15.326639 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133840 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 16:15:15.326639 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133842 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 16:15:15.326639 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133845 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 16:15:15.326639 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133847 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 16:15:15.326639 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133851 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 16:15:15.326639 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133854 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 16:15:15.326639 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133856 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 16:15:15.326639 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133859 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 16:15:15.326639 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133862 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 16:15:15.326639 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133865 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 16:15:15.326639 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133868 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 16:15:15.327748 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133886 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 16:15:15.327748 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133889 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 16:15:15.327748 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133891 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 16:15:15.327748 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133894 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 16:15:15.327748 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133897 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 16:15:15.327748 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133899 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 16:15:15.327748 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133902 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 16:15:15.327748 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133904 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 16:15:15.327748 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133907 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 16:15:15.327748 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133910 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 16:15:15.327748 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133912 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 16:15:15.327748 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133915 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 16:15:15.327748 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133917 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 16:15:15.327748 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133920 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 16:15:15.327748 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133923 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 16:15:15.327748 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133925 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 16:15:15.327748 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133928 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 16:15:15.327748 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133930 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 16:15:15.327748 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133933 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 16:15:15.327748 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133937 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 16:15:15.328588 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133940 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 16:15:15.328588 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133943 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 16:15:15.328588 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133945 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 16:15:15.328588 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133948 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 16:15:15.328588 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133950 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 16:15:15.328588 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133954 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 16:15:15.328588 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133956 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 16:15:15.328588 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133959 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 16:15:15.328588 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133962 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 16:15:15.328588 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133965 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 16:15:15.328588 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133967 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 16:15:15.328588 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133970 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 16:15:15.328588 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133972 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 16:15:15.328588 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133975 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 16:15:15.328588 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133978 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 16:15:15.328588 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133980 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 16:15:15.328588 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133984 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 16:15:15.328588 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133987 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 16:15:15.328588 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133989 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 16:15:15.328588 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133991 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 16:15:15.329490 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133994 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 16:15:15.329490 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133997 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 16:15:15.329490 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.133999 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 16:15:15.329490 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.134002 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 16:15:15.329490 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.134004 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 16:15:15.329490 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.134007 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 16:15:15.329490 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.134010 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 16:15:15.329490 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.134012 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 16:15:15.329490 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.134015 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 16:15:15.329490 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.134017 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 16:15:15.329490 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.134020 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 16:15:15.329490 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.134025 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 16:15:15.329490 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.134028 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 16:15:15.329490 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134107 2569 flags.go:64] FLAG: --address="0.0.0.0" Jul 09 16:15:15.329490 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134115 2569 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 09 16:15:15.329490 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134124 2569 flags.go:64] FLAG: --anonymous-auth="true" Jul 09 16:15:15.329490 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134130 2569 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 09 16:15:15.329490 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134138 2569 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 09 16:15:15.329490 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134143 2569 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 09 16:15:15.329490 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134150 2569 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 09 16:15:15.329490 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134154 2569 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 09 16:15:15.330402 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134158 2569 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 09 16:15:15.330402 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134162 2569 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 09 16:15:15.330402 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134166 2569 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 09 16:15:15.330402 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134169 2569 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 09 16:15:15.330402 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134172 2569 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 09 16:15:15.330402 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134175 2569 flags.go:64] FLAG: --cgroup-root="" Jul 09 16:15:15.330402 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134178 2569 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 09 16:15:15.330402 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134181 2569 flags.go:64] FLAG: --client-ca-file="" Jul 09 16:15:15.330402 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134184 2569 flags.go:64] FLAG: --cloud-provider="external" Jul 09 16:15:15.330402 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134187 2569 flags.go:64] FLAG: --cluster-dns="[]" Jul 09 16:15:15.330402 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134192 2569 flags.go:64] FLAG: --cluster-domain="" Jul 09 16:15:15.330402 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134195 2569 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 09 16:15:15.330402 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134199 2569 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 09 16:15:15.330402 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134202 2569 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 09 16:15:15.330402 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134205 2569 flags.go:64] FLAG: --container-log-max-files="5" Jul 09 16:15:15.330402 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134209 2569 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 09 16:15:15.330402 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134212 2569 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 09 16:15:15.330402 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134215 2569 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 09 16:15:15.330402 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134219 2569 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 09 16:15:15.330402 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134222 2569 flags.go:64] FLAG: --contention-profiling="false" Jul 09 16:15:15.330402 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134225 2569 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 09 16:15:15.330402 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134228 2569 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 09 16:15:15.330402 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134232 2569 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 09 16:15:15.330402 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134235 2569 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 09 16:15:15.331518 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134240 2569 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 09 16:15:15.331518 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134243 2569 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 09 16:15:15.331518 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134246 2569 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 09 16:15:15.331518 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134249 2569 flags.go:64] FLAG: --enable-load-reader="false" Jul 09 16:15:15.331518 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134255 2569 flags.go:64] FLAG: --enable-server="true" Jul 09 16:15:15.331518 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134258 2569 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 09 16:15:15.331518 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134263 2569 flags.go:64] FLAG: --event-burst="100" Jul 09 16:15:15.331518 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134266 2569 flags.go:64] FLAG: --event-qps="50" Jul 09 16:15:15.331518 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134269 2569 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 09 16:15:15.331518 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134272 2569 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 09 16:15:15.331518 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134276 2569 flags.go:64] FLAG: --eviction-hard="" Jul 09 16:15:15.331518 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134279 2569 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 09 16:15:15.331518 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134283 2569 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 09 16:15:15.331518 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134286 2569 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 09 16:15:15.331518 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134289 2569 flags.go:64] FLAG: --eviction-soft="" Jul 09 16:15:15.331518 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134292 2569 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 09 16:15:15.331518 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134295 2569 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 09 16:15:15.331518 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134298 2569 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 09 16:15:15.331518 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134301 2569 flags.go:64] FLAG: --experimental-mounter-path="" Jul 09 16:15:15.331518 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134304 2569 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 09 16:15:15.331518 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134307 2569 flags.go:64] FLAG: --fail-swap-on="true" Jul 09 16:15:15.331518 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134310 2569 flags.go:64] FLAG: --feature-gates="" Jul 09 16:15:15.331518 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134313 2569 flags.go:64] FLAG: --file-check-frequency="20s" Jul 09 16:15:15.331518 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134317 2569 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 09 16:15:15.331518 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134320 2569 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 09 16:15:15.332633 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134323 2569 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 09 16:15:15.332633 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134326 2569 flags.go:64] FLAG: --healthz-port="10248" Jul 09 16:15:15.332633 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134329 2569 flags.go:64] FLAG: --help="false" Jul 09 16:15:15.332633 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134332 2569 flags.go:64] FLAG: --hostname-override="ip-10-0-135-114.ec2.internal" Jul 09 16:15:15.332633 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134335 2569 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 09 16:15:15.332633 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134339 2569 flags.go:64] FLAG: --http-check-frequency="20s" Jul 09 16:15:15.332633 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134342 2569 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 09 16:15:15.332633 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134346 2569 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 09 16:15:15.332633 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134349 2569 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 09 16:15:15.332633 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134352 2569 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 09 16:15:15.332633 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134355 2569 flags.go:64] FLAG: --image-service-endpoint="" Jul 09 16:15:15.332633 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134359 2569 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 09 16:15:15.332633 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134362 2569 flags.go:64] FLAG: --kube-api-burst="100" Jul 09 16:15:15.332633 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134365 2569 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 09 16:15:15.332633 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134369 2569 flags.go:64] FLAG: --kube-api-qps="50" Jul 09 16:15:15.332633 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134372 2569 flags.go:64] FLAG: --kube-reserved="" Jul 09 16:15:15.332633 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134375 2569 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 09 16:15:15.332633 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134378 2569 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 09 16:15:15.332633 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134381 2569 flags.go:64] FLAG: --kubelet-cgroups="" Jul 09 16:15:15.332633 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134384 2569 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 09 16:15:15.332633 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134387 2569 flags.go:64] FLAG: --lock-file="" Jul 09 16:15:15.332633 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134390 2569 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 09 16:15:15.332633 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134393 2569 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 09 16:15:15.332633 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134396 2569 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 09 16:15:15.333731 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134402 2569 flags.go:64] FLAG: --log-json-split-stream="false" Jul 09 16:15:15.333731 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134404 2569 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 09 16:15:15.333731 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134407 2569 flags.go:64] FLAG: --log-text-split-stream="false" Jul 09 16:15:15.333731 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134410 2569 flags.go:64] FLAG: --logging-format="text" Jul 09 16:15:15.333731 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134413 2569 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 09 16:15:15.333731 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134416 2569 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 09 16:15:15.333731 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134419 2569 flags.go:64] FLAG: --manifest-url="" Jul 09 16:15:15.333731 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134422 2569 flags.go:64] FLAG: --manifest-url-header="" Jul 09 16:15:15.333731 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134426 2569 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 09 16:15:15.333731 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134429 2569 flags.go:64] FLAG: --max-open-files="1000000" Jul 09 16:15:15.333731 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134433 2569 flags.go:64] FLAG: --max-pods="110" Jul 09 16:15:15.333731 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134437 2569 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 09 16:15:15.333731 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134440 2569 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 09 16:15:15.333731 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134444 2569 flags.go:64] FLAG: --memory-manager-policy="None" Jul 09 16:15:15.333731 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134447 2569 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 09 16:15:15.333731 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134450 2569 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 09 16:15:15.333731 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134453 2569 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 09 16:15:15.333731 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134456 2569 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 09 16:15:15.333731 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134464 2569 flags.go:64] FLAG: --node-status-max-images="50" Jul 09 16:15:15.333731 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134468 2569 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 09 16:15:15.333731 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134471 2569 flags.go:64] FLAG: --oom-score-adj="-999" Jul 09 16:15:15.333731 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134475 2569 flags.go:64] FLAG: --pod-cidr="" Jul 09 16:15:15.333731 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134478 2569 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 09 16:15:15.334827 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134484 2569 flags.go:64] FLAG: --pod-manifest-path="" Jul 09 16:15:15.334827 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134487 2569 flags.go:64] FLAG: --pod-max-pids="-1" Jul 09 16:15:15.334827 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134490 2569 flags.go:64] FLAG: --pods-per-core="0" Jul 09 16:15:15.334827 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134493 2569 flags.go:64] FLAG: --port="10250" Jul 09 16:15:15.334827 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134497 2569 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 09 16:15:15.334827 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134500 2569 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-07b0c7f5de8164743" Jul 09 16:15:15.334827 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134503 2569 flags.go:64] FLAG: --qos-reserved="" Jul 09 16:15:15.334827 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134506 2569 flags.go:64] FLAG: --read-only-port="10255" Jul 09 16:15:15.334827 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134509 2569 flags.go:64] FLAG: --register-node="true" Jul 09 16:15:15.334827 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134512 2569 flags.go:64] FLAG: --register-with-taints="" Jul 09 16:15:15.334827 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134516 2569 flags.go:64] FLAG: --registry-burst="10" Jul 09 16:15:15.334827 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134519 2569 flags.go:64] FLAG: --registry-qps="5" Jul 09 16:15:15.334827 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134522 2569 flags.go:64] FLAG: --reserved-cpus="" Jul 09 16:15:15.334827 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134524 2569 flags.go:64] FLAG: --reserved-memory="" Jul 09 16:15:15.334827 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134528 2569 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 09 16:15:15.334827 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134531 2569 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 09 16:15:15.334827 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134534 2569 flags.go:64] FLAG: --rotate-certificates="false" Jul 09 16:15:15.334827 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134537 2569 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 09 16:15:15.334827 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134540 2569 flags.go:64] FLAG: --runonce="false" Jul 09 16:15:15.334827 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134543 2569 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 09 16:15:15.334827 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134546 2569 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 09 16:15:15.334827 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134549 2569 flags.go:64] FLAG: --seccomp-default="false" Jul 09 16:15:15.334827 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134554 2569 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 09 16:15:15.334827 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134556 2569 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 09 16:15:15.334827 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134560 2569 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 09 16:15:15.334827 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134563 2569 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 09 16:15:15.336148 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134566 2569 flags.go:64] FLAG: --storage-driver-password="root" Jul 09 16:15:15.336148 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134569 2569 flags.go:64] FLAG: --storage-driver-secure="false" Jul 09 16:15:15.336148 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134572 2569 flags.go:64] FLAG: --storage-driver-table="stats" Jul 09 16:15:15.336148 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134576 2569 flags.go:64] FLAG: --storage-driver-user="root" Jul 09 16:15:15.336148 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134579 2569 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 09 16:15:15.336148 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134582 2569 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 09 16:15:15.336148 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134585 2569 flags.go:64] FLAG: --system-cgroups="" Jul 09 16:15:15.336148 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134588 2569 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 09 16:15:15.336148 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134593 2569 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 09 16:15:15.336148 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134596 2569 flags.go:64] FLAG: --tls-cert-file="" Jul 09 16:15:15.336148 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134599 2569 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 09 16:15:15.336148 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134604 2569 flags.go:64] FLAG: --tls-min-version="" Jul 09 16:15:15.336148 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134606 2569 flags.go:64] FLAG: --tls-private-key-file="" Jul 09 16:15:15.336148 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134609 2569 flags.go:64] FLAG: --topology-manager-policy="none" Jul 09 16:15:15.336148 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134612 2569 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 09 16:15:15.336148 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134615 2569 flags.go:64] FLAG: --topology-manager-scope="container" Jul 09 16:15:15.336148 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134618 2569 flags.go:64] FLAG: --v="2" Jul 09 16:15:15.336148 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134622 2569 flags.go:64] FLAG: --version="false" Jul 09 16:15:15.336148 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134626 2569 flags.go:64] FLAG: --vmodule="" Jul 09 16:15:15.336148 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134630 2569 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 09 16:15:15.336148 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.134633 2569 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 09 16:15:15.336148 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.134779 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 16:15:15.336148 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.134784 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 16:15:15.336148 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.134787 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 16:15:15.337252 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.134790 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 16:15:15.337252 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.134793 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 16:15:15.337252 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.134796 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 16:15:15.337252 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.134799 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 16:15:15.337252 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.134803 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 16:15:15.337252 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.134806 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 16:15:15.337252 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.134808 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 09 16:15:15.337252 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.134811 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 16:15:15.337252 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.134813 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 16:15:15.337252 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.134816 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 16:15:15.337252 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.134818 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 16:15:15.337252 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.134822 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 16:15:15.337252 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.134825 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 16:15:15.337252 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.134828 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 16:15:15.337252 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.134831 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 16:15:15.337252 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.134833 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 16:15:15.337252 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.134836 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 16:15:15.337252 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.134839 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 16:15:15.337252 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.134841 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 16:15:15.337252 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.134844 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 16:15:15.338155 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.134847 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 16:15:15.338155 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.134850 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 16:15:15.338155 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.134853 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 16:15:15.338155 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.134855 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 16:15:15.338155 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.134858 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 16:15:15.338155 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.134862 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 16:15:15.338155 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.134866 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 16:15:15.338155 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.134882 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 16:15:15.338155 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.134888 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 16:15:15.338155 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.134891 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 16:15:15.338155 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.134894 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 16:15:15.338155 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.134896 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 16:15:15.338155 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.134900 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 16:15:15.338155 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.134902 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 16:15:15.338155 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.134905 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 16:15:15.338155 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.134908 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 16:15:15.338155 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.134912 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 16:15:15.338155 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.134915 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 16:15:15.338155 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.134917 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 16:15:15.339217 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.134920 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 16:15:15.339217 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.134923 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 16:15:15.339217 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.134925 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 16:15:15.339217 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.134928 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 16:15:15.339217 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.134932 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 16:15:15.339217 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.134935 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 16:15:15.339217 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.134938 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 16:15:15.339217 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.134940 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 16:15:15.339217 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.134943 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 16:15:15.339217 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.134946 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 16:15:15.339217 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.134949 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 16:15:15.339217 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.134951 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 16:15:15.339217 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.134954 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 16:15:15.339217 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.134957 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 16:15:15.339217 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.134961 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 16:15:15.339217 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.134964 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 16:15:15.339217 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.134966 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 16:15:15.339217 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.134969 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 16:15:15.339217 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.134973 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 16:15:15.340100 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.134976 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 16:15:15.340100 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.134979 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 16:15:15.340100 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.134982 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 16:15:15.340100 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.134984 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 16:15:15.340100 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.134988 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 16:15:15.340100 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.134990 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 16:15:15.340100 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.134993 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 16:15:15.340100 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.134996 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 16:15:15.340100 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.134999 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 16:15:15.340100 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.135002 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 16:15:15.340100 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.135005 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 16:15:15.340100 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.135009 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 16:15:15.340100 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.135011 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 16:15:15.340100 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.135014 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 16:15:15.340100 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.135017 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 16:15:15.340100 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.135019 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 16:15:15.340100 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.135022 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 16:15:15.340100 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.135026 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 16:15:15.340100 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.135028 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 16:15:15.340100 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.135031 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 16:15:15.343320 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.135033 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 16:15:15.343320 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.135036 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 16:15:15.343320 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.135039 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 16:15:15.343320 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.135041 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 16:15:15.343320 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.135044 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 16:15:15.343320 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.135046 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 16:15:15.343320 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.135050 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 16:15:15.343320 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.135053 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 16:15:15.343320 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.135057 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 16:15:15.343320 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.135059 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 16:15:15.343320 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.135062 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 16:15:15.343320 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.135065 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 16:15:15.343320 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.135067 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 16:15:15.343320 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.135070 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 16:15:15.343320 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.135072 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 16:15:15.343320 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.135075 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 16:15:15.343320 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.135078 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 16:15:15.343320 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.135081 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 16:15:15.549939 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.135897 2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 09 16:15:15.549939 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.145405 2569 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 09 16:15:15.549939 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.145426 2569 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 09 16:15:15.549939 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.145447 2569 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 09 16:15:15.549939 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.145454 2569 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 09 16:15:15.549939 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145515 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 16:15:15.549939 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145522 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 16:15:15.549939 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145525 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 16:15:15.549939 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145530 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 16:15:15.549939 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145534 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 16:15:15.549939 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145538 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 16:15:15.549939 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145541 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 16:15:15.549939 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145544 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 16:15:15.549939 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145547 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 16:15:15.557408 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145550 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 16:15:15.557408 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145553 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 16:15:15.557408 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145556 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 16:15:15.557408 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145559 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 16:15:15.557408 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145562 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 16:15:15.557408 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145564 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 16:15:15.557408 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145567 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 16:15:15.557408 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145571 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 16:15:15.557408 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145574 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 16:15:15.557408 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145576 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 16:15:15.557408 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145579 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 16:15:15.557408 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145582 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 16:15:15.557408 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145585 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 16:15:15.557408 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145587 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 16:15:15.557408 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145590 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 16:15:15.557408 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145599 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 16:15:15.557408 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145602 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 16:15:15.557408 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145605 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 16:15:15.557408 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145608 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 16:15:15.557408 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145610 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 16:15:15.558514 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145613 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 16:15:15.558514 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145617 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 16:15:15.558514 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145619 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 16:15:15.558514 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145622 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 16:15:15.558514 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145626 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 16:15:15.558514 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145630 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 16:15:15.558514 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145635 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 16:15:15.558514 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145638 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 16:15:15.558514 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145641 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 16:15:15.558514 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145645 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 16:15:15.558514 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145648 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 16:15:15.558514 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145650 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 16:15:15.558514 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145654 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 16:15:15.558514 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145657 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 16:15:15.558514 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145660 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 16:15:15.558514 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145662 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 16:15:15.558514 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145665 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 16:15:15.558514 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145668 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 16:15:15.558514 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145671 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 16:15:15.559398 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145673 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 16:15:15.559398 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145677 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 16:15:15.559398 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145680 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 16:15:15.559398 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145682 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 16:15:15.559398 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145685 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 16:15:15.559398 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145687 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 16:15:15.559398 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145690 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 16:15:15.559398 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145693 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 16:15:15.559398 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145697 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 16:15:15.559398 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145703 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 16:15:15.559398 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145706 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 16:15:15.559398 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145708 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 16:15:15.559398 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145711 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 16:15:15.559398 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145713 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 16:15:15.559398 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145716 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 16:15:15.559398 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145719 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 16:15:15.559398 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145721 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 16:15:15.559398 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145724 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 16:15:15.559398 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145726 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 16:15:15.559398 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145729 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 16:15:15.560266 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145732 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 16:15:15.560266 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145734 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 16:15:15.560266 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145737 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 16:15:15.560266 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145739 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 09 16:15:15.560266 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145742 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 16:15:15.560266 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145745 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 16:15:15.560266 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145747 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 16:15:15.560266 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145750 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 16:15:15.560266 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145752 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 16:15:15.560266 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145755 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 16:15:15.560266 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145758 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 16:15:15.560266 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145761 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 16:15:15.560266 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145764 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 16:15:15.560266 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145767 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 16:15:15.560266 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145769 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 16:15:15.560266 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145772 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 16:15:15.560266 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145774 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 16:15:15.560266 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145777 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 16:15:15.560266 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145780 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 16:15:15.560266 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145782 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 16:15:15.561153 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145786 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 16:15:15.561153 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145790 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 16:15:15.561153 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145793 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 16:15:15.561153 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145796 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 16:15:15.561153 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145798 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 16:15:15.561153 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145801 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 16:15:15.561153 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145804 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 16:15:15.561153 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145807 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 16:15:15.561153 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145809 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 16:15:15.561153 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145812 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 16:15:15.561153 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145815 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 16:15:15.561153 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.145820 2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 09 16:15:15.561153 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145985 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 16:15:15.561153 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145991 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 16:15:15.561823 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145995 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 16:15:15.561823 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.145998 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 16:15:15.561823 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146001 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 16:15:15.561823 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146005 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 16:15:15.561823 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146008 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 16:15:15.561823 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146011 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 16:15:15.561823 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146013 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 16:15:15.561823 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146016 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 16:15:15.561823 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146020 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 16:15:15.561823 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146024 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 16:15:15.561823 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146028 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 16:15:15.561823 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146031 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 16:15:15.561823 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146033 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 16:15:15.561823 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146036 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 16:15:15.561823 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146039 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 16:15:15.561823 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146041 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 16:15:15.561823 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146044 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 16:15:15.561823 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146047 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 16:15:15.561823 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146049 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 16:15:15.561823 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146055 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 16:15:15.562703 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146058 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 16:15:15.562703 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146060 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 16:15:15.562703 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146063 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 16:15:15.562703 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146066 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 16:15:15.562703 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146068 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 16:15:15.562703 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146071 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 16:15:15.562703 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146073 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 16:15:15.562703 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146076 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 16:15:15.562703 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146078 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 16:15:15.562703 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146081 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 16:15:15.562703 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146084 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 16:15:15.562703 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146088 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 16:15:15.562703 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146091 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 16:15:15.562703 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146094 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 16:15:15.562703 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146097 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 16:15:15.562703 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146101 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 16:15:15.562703 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146103 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 16:15:15.562703 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146106 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 16:15:15.562703 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146109 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 16:15:15.563600 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146112 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 16:15:15.563600 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146115 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 16:15:15.563600 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146117 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 16:15:15.563600 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146120 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 16:15:15.563600 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146123 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 16:15:15.563600 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146125 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 16:15:15.563600 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146128 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 16:15:15.563600 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146130 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 16:15:15.563600 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146133 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 16:15:15.563600 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146136 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 16:15:15.563600 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146138 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 16:15:15.563600 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146141 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 16:15:15.563600 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146145 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 16:15:15.563600 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146148 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 16:15:15.563600 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146151 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 16:15:15.563600 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146153 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 16:15:15.563600 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146156 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 16:15:15.563600 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146159 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 16:15:15.563600 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146161 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 16:15:15.563600 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146164 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 16:15:15.564490 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146166 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 16:15:15.564490 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146169 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 16:15:15.564490 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146172 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 16:15:15.564490 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146174 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 16:15:15.564490 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146177 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 16:15:15.564490 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146180 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 16:15:15.564490 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146182 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 16:15:15.564490 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146185 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 16:15:15.564490 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146187 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 16:15:15.564490 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146190 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 16:15:15.564490 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146192 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 16:15:15.564490 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146195 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 16:15:15.564490 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146197 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 16:15:15.564490 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146200 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 16:15:15.564490 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146203 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 16:15:15.564490 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146205 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 16:15:15.564490 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146207 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 16:15:15.564490 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146213 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 16:15:15.564490 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146216 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 16:15:15.564490 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146218 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 16:15:15.565399 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146221 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 16:15:15.565399 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146223 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 16:15:15.565399 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146226 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 16:15:15.565399 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146229 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 16:15:15.565399 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146235 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 16:15:15.565399 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146237 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 16:15:15.565399 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146240 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 16:15:15.565399 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146243 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 16:15:15.565399 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146245 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 16:15:15.565399 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146250 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 16:15:15.565399 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146253 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 09 16:15:15.565399 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146256 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 16:15:15.565399 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146258 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 16:15:15.565399 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146261 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 16:15:15.565399 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146264 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 16:15:15.565399 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146266 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 16:15:15.565399 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146269 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 16:15:15.565399 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:15.146271 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 16:15:15.566309 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.146276 2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 09 16:15:15.566309 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.146426 2569 server.go:962] "Client rotation is on, will bootstrap in background" Jul 09 16:15:15.566309 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.151527 2569 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 09 16:15:15.566309 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.153257 2569 server.go:1019] "Starting client certificate rotation" Jul 09 16:15:15.566309 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.153362 2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 09 16:15:15.566309 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.153410 2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 09 16:15:15.566309 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.157848 2569 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 09 16:15:15.566309 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.160818 2569 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 09 16:15:15.566309 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.173928 2569 log.go:25] "Validated CRI v1 runtime API" Jul 09 16:15:15.566309 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.176508 2569 log.go:25] "Validated CRI v1 image API" Jul 09 16:15:15.566309 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.177510 2569 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 09 16:15:15.566309 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.180308 2569 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 c7fe5a5a-702a-4008-907d-4b4b657845b2:/dev/nvme0n1p4 e51a45cb-ea92-456c-a1ea-45068025d4b6:/dev/nvme0n1p3] Jul 09 16:15:15.567018 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.180338 2569 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 09 16:15:15.567018 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.182021 2569 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 09 16:15:15.567451 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.187143 2569 manager.go:216] Machine: {Timestamp:2026-07-09 16:15:15.185762471 +0000 UTC m=+0.378868843 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099931 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2176329d655563c059363f15ebe0d5 SystemUUID:ec217632-9d65-5563-c059-363f15ebe0d5 BootID:55957767-4fd9-419c-91cb-ce52fe43298d Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:49:bc:15:12:07 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:49:bc:15:12:07 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ca:cb:49:4d:ab:1f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 09 16:15:15.567451 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.187264 2569 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 09 16:15:15.567451 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.187356 2569 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 09 16:15:15.567451 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.190399 2569 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 09 16:15:15.567451 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.190433 2569 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-114.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 09 16:15:15.567451 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.191420 2569 topology_manager.go:138] "Creating topology manager with none policy" Jul 09 16:15:15.567451 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.191435 2569 container_manager_linux.go:309] "Creating device plugin manager" Jul 09 16:15:15.567451 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.191450 2569 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 09 16:15:15.567451 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.191469 2569 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 09 16:15:15.567451 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.193481 2569 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 09 16:15:15.567451 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.198221 2569 state_mem.go:36] "Initialized new in-memory state store" Jul 09 16:15:15.567451 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.198379 2569 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 09 16:15:15.567451 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.198414 2569 kubelet.go:486] "Attempting to sync node with API server" Jul 09 16:15:15.567451 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.198428 2569 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 09 16:15:15.567451 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.198446 2569 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 09 16:15:15.567451 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.198458 2569 kubelet.go:398] "Adding apiserver pod source" Jul 09 16:15:15.567451 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.198469 2569 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 09 16:15:15.567451 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.199199 2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 09 16:15:15.568339 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.199236 2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 09 16:15:15.568339 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.202990 2569 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 09 16:15:15.568339 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.205769 2569 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 09 16:15:15.568339 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.205798 2569 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 09 16:15:15.568339 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.205980 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 09 16:15:15.568339 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.205999 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 09 16:15:15.568339 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.206009 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 09 16:15:15.568339 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.206019 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 09 16:15:15.568339 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.206031 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 09 16:15:15.568339 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.206039 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 09 16:15:15.568339 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.206592 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 09 16:15:15.568339 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.206602 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 09 16:15:15.568339 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.206609 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 09 16:15:15.568339 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.206615 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 09 16:15:15.568339 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.206627 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 09 16:15:15.568339 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.206635 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 09 16:15:15.568339 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.208345 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 09 16:15:15.568339 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.208353 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 09 16:15:15.568339 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.211681 2569 server.go:1268] "Started kubelet" Jul 09 16:15:15.568339 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.211756 2569 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 09 16:15:15.568339 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.211765 2569 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 09 16:15:15.568339 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.211862 2569 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 09 16:15:15.568339 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.213388 2569 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 09 16:15:15.568339 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.214678 2569 server.go:310] "Adding debug handlers to kubelet server" Jul 09 16:15:15.568339 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:15.215287 2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-114.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 09 16:15:15.568339 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:15.215394 2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 09 16:15:15.568339 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.215404 2569 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-114.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 09 16:15:15.568339 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:15.219350 2569 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 09 16:15:15.568339 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:15.220519 2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-114.ec2.internal.18c0ab3957b48eaa default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-114.ec2.internal,UID:ip-10-0-135-114.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-114.ec2.internal,},FirstTimestamp:2026-07-09 16:15:15.211652778 +0000 UTC m=+0.404759150,LastTimestamp:2026-07-09 16:15:15.211652778 +0000 UTC m=+0.404759150,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-114.ec2.internal,}" Jul 09 16:15:15.568339 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.221818 2569 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 09 16:15:15.568339 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.221818 2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 09 16:15:15.568339 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:15.221937 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-114.ec2.internal\" not found" Jul 09 16:15:15.568339 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.222630 2569 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 09 16:15:15.568339 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.222643 2569 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 09 16:15:15.568339 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.222657 2569 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 09 16:15:15.568339 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.222772 2569 factory.go:55] Registering systemd factory Jul 09 16:15:15.568339 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.222790 2569 factory.go:223] Registration of the systemd container factory successfully Jul 09 16:15:15.568339 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.222777 2569 reconstruct.go:97] "Volume reconstruction finished" Jul 09 16:15:15.568339 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.222836 2569 reconciler.go:29] "Reconciler: start to sync state" Jul 09 16:15:15.568339 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.222984 2569 factory.go:153] Registering CRI-O factory Jul 09 16:15:15.568339 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.222998 2569 factory.go:223] Registration of the crio container factory successfully Jul 09 16:15:15.570135 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.223043 2569 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 09 16:15:15.570135 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.223067 2569 factory.go:103] Registering Raw factory Jul 09 16:15:15.570135 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.223083 2569 manager.go:1196] Started watching for new ooms in manager Jul 09 16:15:15.570135 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.223468 2569 manager.go:349] Starting recovery of all containers Jul 09 16:15:15.570135 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:15.227698 2569 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-114.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 09 16:15:15.570135 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:15.227819 2569 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 09 16:15:15.570135 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.236483 2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hk26q" Jul 09 16:15:15.570135 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.236838 2569 manager.go:354] Recovery completed Jul 09 16:15:15.570135 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.241604 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 16:15:15.570135 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.243712 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-114.ec2.internal" event="NodeHasSufficientMemory" Jul 09 16:15:15.570135 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.243742 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-114.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 16:15:15.570135 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.243754 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-114.ec2.internal" event="NodeHasSufficientPID" Jul 09 16:15:15.570135 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.243788 2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hk26q" Jul 09 16:15:15.570135 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.244335 2569 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 09 16:15:15.570135 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.244345 2569 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 09 16:15:15.570135 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.244348 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 16:15:15.570135 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.244363 2569 state_mem.go:36] "Initialized new in-memory state store" Jul 09 16:15:15.570135 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.245201 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-114.ec2.internal" event="NodeHasSufficientMemory" Jul 09 16:15:15.570135 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.245224 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-114.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 16:15:15.570135 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.245234 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-114.ec2.internal" event="NodeHasSufficientPID" Jul 09 16:15:15.570135 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:15.245691 2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-114.ec2.internal.18c0ab39599df65a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-114.ec2.internal,UID:ip-10-0-135-114.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-114.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-114.ec2.internal,},FirstTimestamp:2026-07-09 16:15:15.243726426 +0000 UTC m=+0.436832798,LastTimestamp:2026-07-09 16:15:15.243726426 +0000 UTC m=+0.436832798,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-114.ec2.internal,}" Jul 09 16:15:15.570135 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.246416 2569 policy_none.go:49] "None policy: Start" Jul 09 16:15:15.570135 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.246439 2569 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 09 16:15:15.570135 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.246452 2569 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 09 16:15:15.570135 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.248652 2569 policy_none.go:47] "Start" Jul 09 16:15:15.570135 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.295752 2569 manager.go:337] "Starting Device Plugin manager" Jul 09 16:15:15.570135 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:15.295947 2569 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 09 16:15:15.570135 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.295962 2569 server.go:87] "Starting device plugin registration server" Jul 09 16:15:15.570135 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.296277 2569 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 09 16:15:15.570135 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.296292 2569 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 09 16:15:15.570135 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.296398 2569 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 09 16:15:15.570135 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.296496 2569 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 09 16:15:15.570135 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.296507 2569 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 09 16:15:15.570135 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:15.297169 2569 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 09 16:15:15.570135 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:15.297222 2569 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-114.ec2.internal\" not found" Jul 09 16:15:15.570135 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.381286 2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 09 16:15:15.570135 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.382737 2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 09 16:15:15.570135 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.382782 2569 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 09 16:15:15.570135 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.382824 2569 kubelet.go:2448] "Starting kubelet main sync loop" Jul 09 16:15:15.571693 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:15.382888 2569 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 09 16:15:15.571693 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.385298 2569 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 16:15:15.571693 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.397290 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 16:15:15.571693 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.398569 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-114.ec2.internal" event="NodeHasSufficientMemory" Jul 09 16:15:15.571693 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.398593 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-114.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 16:15:15.571693 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.398603 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-114.ec2.internal" event="NodeHasSufficientPID" Jul 09 16:15:15.571693 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.398624 2569 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-114.ec2.internal" Jul 09 16:15:15.571693 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.406588 2569 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-114.ec2.internal" Jul 09 16:15:15.571693 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:15.406611 2569 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-114.ec2.internal\": node \"ip-10-0-135-114.ec2.internal\" not found" Jul 09 16:15:15.571693 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:15.435140 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-114.ec2.internal\" not found" Jul 09 16:15:15.571693 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.483130 2569 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-114.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-114.ec2.internal"] Jul 09 16:15:15.571693 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.483211 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 16:15:15.571693 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.484180 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-114.ec2.internal" event="NodeHasSufficientMemory" Jul 09 16:15:15.571693 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.484205 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-114.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 16:15:15.571693 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.484217 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-114.ec2.internal" event="NodeHasSufficientPID" Jul 09 16:15:15.571693 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.486940 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 16:15:15.571693 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.487045 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-114.ec2.internal" Jul 09 16:15:15.571693 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.487073 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 16:15:15.571693 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.489321 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-114.ec2.internal" event="NodeHasSufficientMemory" Jul 09 16:15:15.571693 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.489331 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-114.ec2.internal" event="NodeHasSufficientMemory" Jul 09 16:15:15.571693 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.489353 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-114.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 16:15:15.571693 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.489364 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-114.ec2.internal" event="NodeHasSufficientPID" Jul 09 16:15:15.571693 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.489355 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-114.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 16:15:15.571693 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.489397 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-114.ec2.internal" event="NodeHasSufficientPID" Jul 09 16:15:15.571693 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.491799 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-114.ec2.internal" Jul 09 16:15:15.571693 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.491830 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 16:15:15.571693 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.492812 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-114.ec2.internal" event="NodeHasSufficientMemory" Jul 09 16:15:15.571693 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.492838 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-114.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 16:15:15.571693 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.492848 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-114.ec2.internal" event="NodeHasSufficientPID" Jul 09 16:15:15.571693 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:15.519906 2569 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-114.ec2.internal\" not found" node="ip-10-0-135-114.ec2.internal" Jul 09 16:15:15.571693 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.524620 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/68e180d1aa487674ae66c09f4bc0df95-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-114.ec2.internal\" (UID: \"68e180d1aa487674ae66c09f4bc0df95\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-114.ec2.internal" Jul 09 16:15:15.571693 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.524653 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/68e180d1aa487674ae66c09f4bc0df95-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-114.ec2.internal\" (UID: \"68e180d1aa487674ae66c09f4bc0df95\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-114.ec2.internal" Jul 09 16:15:15.571693 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.524714 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ef080aa212d8ff70e6e7c4e03505fc8a-config\") pod \"kube-apiserver-proxy-ip-10-0-135-114.ec2.internal\" (UID: \"ef080aa212d8ff70e6e7c4e03505fc8a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-114.ec2.internal" Jul 09 16:15:15.573034 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:15.524656 2569 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-114.ec2.internal\" not found" node="ip-10-0-135-114.ec2.internal" Jul 09 16:15:15.573034 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:15.535849 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-114.ec2.internal\" not found" Jul 09 16:15:15.756790 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.625326 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/68e180d1aa487674ae66c09f4bc0df95-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-114.ec2.internal\" (UID: \"68e180d1aa487674ae66c09f4bc0df95\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-114.ec2.internal" Jul 09 16:15:15.756790 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.625357 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/68e180d1aa487674ae66c09f4bc0df95-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-114.ec2.internal\" (UID: \"68e180d1aa487674ae66c09f4bc0df95\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-114.ec2.internal" Jul 09 16:15:15.756790 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.625374 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ef080aa212d8ff70e6e7c4e03505fc8a-config\") pod \"kube-apiserver-proxy-ip-10-0-135-114.ec2.internal\" (UID: \"ef080aa212d8ff70e6e7c4e03505fc8a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-114.ec2.internal" Jul 09 16:15:15.756790 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.625435 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/68e180d1aa487674ae66c09f4bc0df95-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-114.ec2.internal\" (UID: \"68e180d1aa487674ae66c09f4bc0df95\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-114.ec2.internal" Jul 09 16:15:15.756790 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.625476 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/68e180d1aa487674ae66c09f4bc0df95-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-114.ec2.internal\" (UID: \"68e180d1aa487674ae66c09f4bc0df95\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-114.ec2.internal" Jul 09 16:15:15.756790 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.625497 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ef080aa212d8ff70e6e7c4e03505fc8a-config\") pod \"kube-apiserver-proxy-ip-10-0-135-114.ec2.internal\" (UID: \"ef080aa212d8ff70e6e7c4e03505fc8a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-114.ec2.internal" Jul 09 16:15:15.756790 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:15.636419 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-114.ec2.internal\" not found" Jul 09 16:15:15.756790 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:15.736743 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-114.ec2.internal\" not found" Jul 09 16:15:15.976703 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.825384 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-114.ec2.internal" Jul 09 16:15:15.976703 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:15.827746 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-114.ec2.internal" Jul 09 16:15:15.976703 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:15.837003 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-114.ec2.internal\" not found" Jul 09 16:15:15.976703 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:15.937507 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-114.ec2.internal\" not found" Jul 09 16:15:16.177230 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:16.038263 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-114.ec2.internal\" not found" Jul 09 16:15:16.177230 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:16.139291 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-114.ec2.internal\" not found" Jul 09 16:15:16.177230 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:16.152423 2569 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 09 16:15:16.177230 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:16.152585 2569 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 09 16:15:16.404721 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:16.216760 2569 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 16:15:16.404721 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:16.239736 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-114.ec2.internal\" not found" Jul 09 16:15:16.404721 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:16.248503 2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-08 16:10:15 +0000 UTC" deadline="2028-01-13 08:49:23.244945635 +0000 UTC" Jul 09 16:15:16.404721 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:16.248542 2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13264h34m6.996408702s" Jul 09 16:15:16.404721 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:16.340501 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-114.ec2.internal\" not found" Jul 09 16:15:16.529107 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:16.441681 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-114.ec2.internal\" not found" Jul 09 16:15:16.529107 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:16.492337 2569 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 16:15:16.529107 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:16.522242 2569 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-114.ec2.internal" Jul 09 16:15:16.538126 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:16.537989 2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 09 16:15:16.732340 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:16.539649 2569 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-114.ec2.internal" Jul 09 16:15:16.732340 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:16.549835 2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 09 16:15:16.732340 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:16.630304 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef080aa212d8ff70e6e7c4e03505fc8a.slice/crio-dab292dc388124a5d88da4a16cfe21ce0a08e12547ca36f972d27aa3b72ed793 WatchSource:0}: Error finding container dab292dc388124a5d88da4a16cfe21ce0a08e12547ca36f972d27aa3b72ed793: Status 404 returned error can't find the container with id dab292dc388124a5d88da4a16cfe21ce0a08e12547ca36f972d27aa3b72ed793 Jul 09 16:15:16.732340 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:16.630639 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68e180d1aa487674ae66c09f4bc0df95.slice/crio-ab6c74cc84dd37dfd4303c61be5c7d67ffa82a4abdc53d2e5fb2dd4ff794cae7 WatchSource:0}: Error finding container ab6c74cc84dd37dfd4303c61be5c7d67ffa82a4abdc53d2e5fb2dd4ff794cae7: Status 404 returned error can't find the container with id ab6c74cc84dd37dfd4303c61be5c7d67ffa82a4abdc53d2e5fb2dd4ff794cae7 Jul 09 16:15:16.732340 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:16.636194 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 16:15:16.732340 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:16.672594 2569 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 16:15:17.281240 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.201225 2569 apiserver.go:52] "Watching apiserver" Jul 09 16:15:17.281240 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.211121 2569 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 09 16:15:17.281240 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.211518 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-hgqgw","openshift-ovn-kubernetes/ovnkube-node-49sqd","kube-system/kube-apiserver-proxy-ip-10-0-135-114.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ckqxw","openshift-cluster-node-tuning-operator/tuned-q4flk","openshift-dns/node-resolver-xx9mg","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-114.ec2.internal","openshift-multus/multus-additional-cni-plugins-6q72v","openshift-multus/network-metrics-daemon-b7sxk","openshift-network-diagnostics/network-check-target-kn7sp","kube-system/global-pull-secret-syncer-f6wzq","kube-system/konnectivity-agent-zj5qc","openshift-image-registry/node-ca-jj6qs","openshift-multus/multus-xrbgl"] Jul 09 16:15:17.281240 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.214259 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xx9mg" Jul 09 16:15:17.281240 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.216665 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xrbgl" Jul 09 16:15:17.281240 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.217183 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 09 16:15:17.281240 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.217189 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 09 16:15:17.281240 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.217235 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-f2vcp\"" Jul 09 16:15:17.281240 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.219210 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jj6qs" Jul 09 16:15:17.281240 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.219727 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 09 16:15:17.281240 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.219805 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 09 16:15:17.281240 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.219813 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-x5g2b\"" Jul 09 16:15:17.281240 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.219984 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 09 16:15:17.281240 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.220000 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 09 16:15:17.281240 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.221862 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6q72v" Jul 09 16:15:17.281240 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.221934 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 09 16:15:17.281240 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.222374 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 09 16:15:17.281240 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.222390 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 09 16:15:17.281240 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.222395 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-tvqd9\"" Jul 09 16:15:17.281240 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.222608 2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 09 16:15:17.281240 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.224382 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hgqgw" Jul 09 16:15:17.281240 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.225066 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 09 16:15:17.281240 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.225254 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-dgcfw\"" Jul 09 16:15:17.281240 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.225254 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 09 16:15:17.281240 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.227064 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-q4flk" Jul 09 16:15:17.281240 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.227270 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 09 16:15:17.281240 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.227558 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 09 16:15:17.281240 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.227707 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-q9n2b\"" Jul 09 16:15:17.281240 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.227795 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 09 16:15:17.281240 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.229559 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ckqxw" Jul 09 16:15:17.281240 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.231097 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 09 16:15:17.281240 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.231242 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 09 16:15:17.281240 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.232062 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b7sxk" Jul 09 16:15:17.281240 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:17.232155 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b7sxk" podUID="6f45ad4f-0e3f-432e-8328-715ec5f97a69" Jul 09 16:15:17.282839 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.234129 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 09 16:15:17.282839 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.234134 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-2d6p4\"" Jul 09 16:15:17.282839 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.234400 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 09 16:15:17.282839 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.234659 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kn7sp" Jul 09 16:15:17.282839 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:17.234774 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kn7sp" podUID="473a10a3-ceca-4c00-915e-a4a21820ce3e" Jul 09 16:15:17.282839 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.236269 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bed43223-f252-48e2-b014-b917007e39dd-os-release\") pod \"multus-xrbgl\" (UID: \"bed43223-f252-48e2-b014-b917007e39dd\") " pod="openshift-multus/multus-xrbgl" Jul 09 16:15:17.282839 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.236298 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bed43223-f252-48e2-b014-b917007e39dd-host-run-k8s-cni-cncf-io\") pod \"multus-xrbgl\" (UID: \"bed43223-f252-48e2-b014-b917007e39dd\") " pod="openshift-multus/multus-xrbgl" Jul 09 16:15:17.282839 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.236327 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c17a000a-a79a-4dd8-af71-1fc47077bea7-serviceca\") pod \"node-ca-jj6qs\" (UID: \"c17a000a-a79a-4dd8-af71-1fc47077bea7\") " pod="openshift-image-registry/node-ca-jj6qs" Jul 09 16:15:17.282839 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.236396 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1a7bbb58-c6a9-4763-a12b-c2ae62a5beaf-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6q72v\" (UID: \"1a7bbb58-c6a9-4763-a12b-c2ae62a5beaf\") " pod="openshift-multus/multus-additional-cni-plugins-6q72v" Jul 09 16:15:17.282839 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.236456 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8stpf\" (UniqueName: \"kubernetes.io/projected/29d74a8d-7acf-4663-ba5f-40e4884064bd-kube-api-access-8stpf\") pod \"iptables-alerter-hgqgw\" (UID: \"29d74a8d-7acf-4663-ba5f-40e4884064bd\") " pod="openshift-network-operator/iptables-alerter-hgqgw" Jul 09 16:15:17.282839 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.236497 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88be5834-3fd7-45da-83d5-4cb1d0f84c75-tmp-dir\") pod \"node-resolver-xx9mg\" (UID: \"88be5834-3fd7-45da-83d5-4cb1d0f84c75\") " pod="openshift-dns/node-resolver-xx9mg" Jul 09 16:15:17.282839 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.236518 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bed43223-f252-48e2-b014-b917007e39dd-system-cni-dir\") pod \"multus-xrbgl\" (UID: \"bed43223-f252-48e2-b014-b917007e39dd\") " pod="openshift-multus/multus-xrbgl" Jul 09 16:15:17.282839 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.236558 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bed43223-f252-48e2-b014-b917007e39dd-multus-cni-dir\") pod \"multus-xrbgl\" (UID: \"bed43223-f252-48e2-b014-b917007e39dd\") " pod="openshift-multus/multus-xrbgl" Jul 09 16:15:17.282839 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.236589 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpwq5\" (UniqueName: \"kubernetes.io/projected/1a7bbb58-c6a9-4763-a12b-c2ae62a5beaf-kube-api-access-fpwq5\") pod \"multus-additional-cni-plugins-6q72v\" (UID: \"1a7bbb58-c6a9-4763-a12b-c2ae62a5beaf\") " pod="openshift-multus/multus-additional-cni-plugins-6q72v" Jul 09 16:15:17.282839 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.236626 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/29d74a8d-7acf-4663-ba5f-40e4884064bd-host-slash\") pod \"iptables-alerter-hgqgw\" (UID: \"29d74a8d-7acf-4663-ba5f-40e4884064bd\") " pod="openshift-network-operator/iptables-alerter-hgqgw" Jul 09 16:15:17.282839 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.236652 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bed43223-f252-48e2-b014-b917007e39dd-multus-socket-dir-parent\") pod \"multus-xrbgl\" (UID: \"bed43223-f252-48e2-b014-b917007e39dd\") " pod="openshift-multus/multus-xrbgl" Jul 09 16:15:17.282839 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.236655 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 09 16:15:17.282839 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.236691 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztpfp\" (UniqueName: \"kubernetes.io/projected/c17a000a-a79a-4dd8-af71-1fc47077bea7-kube-api-access-ztpfp\") pod \"node-ca-jj6qs\" (UID: \"c17a000a-a79a-4dd8-af71-1fc47077bea7\") " pod="openshift-image-registry/node-ca-jj6qs" Jul 09 16:15:17.282839 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.236726 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a7bbb58-c6a9-4763-a12b-c2ae62a5beaf-system-cni-dir\") pod \"multus-additional-cni-plugins-6q72v\" (UID: \"1a7bbb58-c6a9-4763-a12b-c2ae62a5beaf\") " pod="openshift-multus/multus-additional-cni-plugins-6q72v" Jul 09 16:15:17.284017 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.236743 2569 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 09 16:15:17.284017 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.236786 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1a7bbb58-c6a9-4763-a12b-c2ae62a5beaf-cni-binary-copy\") pod \"multus-additional-cni-plugins-6q72v\" (UID: \"1a7bbb58-c6a9-4763-a12b-c2ae62a5beaf\") " pod="openshift-multus/multus-additional-cni-plugins-6q72v" Jul 09 16:15:17.284017 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.236812 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bed43223-f252-48e2-b014-b917007e39dd-cnibin\") pod \"multus-xrbgl\" (UID: \"bed43223-f252-48e2-b014-b917007e39dd\") " pod="openshift-multus/multus-xrbgl" Jul 09 16:15:17.284017 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.236867 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bed43223-f252-48e2-b014-b917007e39dd-host-run-netns\") pod \"multus-xrbgl\" (UID: \"bed43223-f252-48e2-b014-b917007e39dd\") " pod="openshift-multus/multus-xrbgl" Jul 09 16:15:17.284017 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.236900 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bed43223-f252-48e2-b014-b917007e39dd-multus-conf-dir\") pod \"multus-xrbgl\" (UID: \"bed43223-f252-48e2-b014-b917007e39dd\") " pod="openshift-multus/multus-xrbgl" Jul 09 16:15:17.284017 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.236943 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bed43223-f252-48e2-b014-b917007e39dd-host-run-multus-certs\") pod \"multus-xrbgl\" (UID: \"bed43223-f252-48e2-b014-b917007e39dd\") " pod="openshift-multus/multus-xrbgl" Jul 09 16:15:17.284017 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.236969 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/88be5834-3fd7-45da-83d5-4cb1d0f84c75-hosts-file\") pod \"node-resolver-xx9mg\" (UID: \"88be5834-3fd7-45da-83d5-4cb1d0f84c75\") " pod="openshift-dns/node-resolver-xx9mg" Jul 09 16:15:17.284017 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.236997 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bed43223-f252-48e2-b014-b917007e39dd-cni-binary-copy\") pod \"multus-xrbgl\" (UID: \"bed43223-f252-48e2-b014-b917007e39dd\") " pod="openshift-multus/multus-xrbgl" Jul 09 16:15:17.284017 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.237057 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-m299r\"" Jul 09 16:15:17.284017 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.237044 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bed43223-f252-48e2-b014-b917007e39dd-host-var-lib-cni-multus\") pod \"multus-xrbgl\" (UID: \"bed43223-f252-48e2-b014-b917007e39dd\") " pod="openshift-multus/multus-xrbgl" Jul 09 16:15:17.284017 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.237150 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bed43223-f252-48e2-b014-b917007e39dd-hostroot\") pod \"multus-xrbgl\" (UID: \"bed43223-f252-48e2-b014-b917007e39dd\") " pod="openshift-multus/multus-xrbgl" Jul 09 16:15:17.284017 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.237184 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n9s2\" (UniqueName: \"kubernetes.io/projected/bed43223-f252-48e2-b014-b917007e39dd-kube-api-access-2n9s2\") pod \"multus-xrbgl\" (UID: \"bed43223-f252-48e2-b014-b917007e39dd\") " pod="openshift-multus/multus-xrbgl" Jul 09 16:15:17.284017 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.237209 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1a7bbb58-c6a9-4763-a12b-c2ae62a5beaf-os-release\") pod \"multus-additional-cni-plugins-6q72v\" (UID: \"1a7bbb58-c6a9-4763-a12b-c2ae62a5beaf\") " pod="openshift-multus/multus-additional-cni-plugins-6q72v" Jul 09 16:15:17.284017 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.237217 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6wzq" Jul 09 16:15:17.284017 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.237236 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5qdf\" (UniqueName: \"kubernetes.io/projected/88be5834-3fd7-45da-83d5-4cb1d0f84c75-kube-api-access-w5qdf\") pod \"node-resolver-xx9mg\" (UID: \"88be5834-3fd7-45da-83d5-4cb1d0f84c75\") " pod="openshift-dns/node-resolver-xx9mg" Jul 09 16:15:17.284017 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.237271 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bed43223-f252-48e2-b014-b917007e39dd-host-var-lib-cni-bin\") pod \"multus-xrbgl\" (UID: \"bed43223-f252-48e2-b014-b917007e39dd\") " pod="openshift-multus/multus-xrbgl" Jul 09 16:15:17.284017 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:17.237274 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6wzq" podUID="3bd017f3-bffb-4016-bdac-48c6fe9fa352" Jul 09 16:15:17.284017 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.237313 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bed43223-f252-48e2-b014-b917007e39dd-host-var-lib-kubelet\") pod \"multus-xrbgl\" (UID: \"bed43223-f252-48e2-b014-b917007e39dd\") " pod="openshift-multus/multus-xrbgl" Jul 09 16:15:17.284977 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.237346 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1a7bbb58-c6a9-4763-a12b-c2ae62a5beaf-cnibin\") pod \"multus-additional-cni-plugins-6q72v\" (UID: \"1a7bbb58-c6a9-4763-a12b-c2ae62a5beaf\") " pod="openshift-multus/multus-additional-cni-plugins-6q72v" Jul 09 16:15:17.284977 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.237377 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1a7bbb58-c6a9-4763-a12b-c2ae62a5beaf-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6q72v\" (UID: \"1a7bbb58-c6a9-4763-a12b-c2ae62a5beaf\") " pod="openshift-multus/multus-additional-cni-plugins-6q72v" Jul 09 16:15:17.284977 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.237400 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bed43223-f252-48e2-b014-b917007e39dd-multus-daemon-config\") pod \"multus-xrbgl\" (UID: \"bed43223-f252-48e2-b014-b917007e39dd\") " pod="openshift-multus/multus-xrbgl" Jul 09 16:15:17.284977 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.237420 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bed43223-f252-48e2-b014-b917007e39dd-etc-kubernetes\") pod \"multus-xrbgl\" (UID: \"bed43223-f252-48e2-b014-b917007e39dd\") " pod="openshift-multus/multus-xrbgl" Jul 09 16:15:17.284977 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.237461 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c17a000a-a79a-4dd8-af71-1fc47077bea7-host\") pod \"node-ca-jj6qs\" (UID: \"c17a000a-a79a-4dd8-af71-1fc47077bea7\") " pod="openshift-image-registry/node-ca-jj6qs" Jul 09 16:15:17.284977 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.237520 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1a7bbb58-c6a9-4763-a12b-c2ae62a5beaf-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6q72v\" (UID: \"1a7bbb58-c6a9-4763-a12b-c2ae62a5beaf\") " pod="openshift-multus/multus-additional-cni-plugins-6q72v" Jul 09 16:15:17.284977 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.237549 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/29d74a8d-7acf-4663-ba5f-40e4884064bd-iptables-alerter-script\") pod \"iptables-alerter-hgqgw\" (UID: \"29d74a8d-7acf-4663-ba5f-40e4884064bd\") " pod="openshift-network-operator/iptables-alerter-hgqgw" Jul 09 16:15:17.284977 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.240313 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zj5qc" Jul 09 16:15:17.284977 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.244000 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 09 16:15:17.284977 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.244018 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 09 16:15:17.284977 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.244073 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-tjl8g\"" Jul 09 16:15:17.284977 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.245917 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-49sqd" Jul 09 16:15:17.284977 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.249161 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 09 16:15:17.284977 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.253857 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 09 16:15:17.284977 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.255291 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 09 16:15:17.284977 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.255336 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-lj2km\"" Jul 09 16:15:17.284977 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.255498 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 09 16:15:17.284977 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.255603 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 09 16:15:17.284977 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.255655 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 09 16:15:17.288611 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.287671 2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-nkzml" Jul 09 16:15:17.301561 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.298156 2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-nkzml" Jul 09 16:15:17.370734 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.324276 2569 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 09 16:15:17.370734 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.338133 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/98ab9c4f-0fd3-485f-8d0e-af1257f14d51-sys\") pod \"tuned-q4flk\" (UID: \"98ab9c4f-0fd3-485f-8d0e-af1257f14d51\") " pod="openshift-cluster-node-tuning-operator/tuned-q4flk" Jul 09 16:15:17.370734 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.338186 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c17a000a-a79a-4dd8-af71-1fc47077bea7-serviceca\") pod \"node-ca-jj6qs\" (UID: \"c17a000a-a79a-4dd8-af71-1fc47077bea7\") " pod="openshift-image-registry/node-ca-jj6qs" Jul 09 16:15:17.370734 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.338207 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1a7bbb58-c6a9-4763-a12b-c2ae62a5beaf-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6q72v\" (UID: \"1a7bbb58-c6a9-4763-a12b-c2ae62a5beaf\") " pod="openshift-multus/multus-additional-cni-plugins-6q72v" Jul 09 16:15:17.370734 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.338232 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88be5834-3fd7-45da-83d5-4cb1d0f84c75-tmp-dir\") pod \"node-resolver-xx9mg\" (UID: \"88be5834-3fd7-45da-83d5-4cb1d0f84c75\") " pod="openshift-dns/node-resolver-xx9mg" Jul 09 16:15:17.370734 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.338255 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/09e70b88-01bd-4114-93c0-fb224d649e1d-device-dir\") pod \"aws-ebs-csi-driver-node-ckqxw\" (UID: \"09e70b88-01bd-4114-93c0-fb224d649e1d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ckqxw" Jul 09 16:15:17.370734 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.338282 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bed43223-f252-48e2-b014-b917007e39dd-system-cni-dir\") pod \"multus-xrbgl\" (UID: \"bed43223-f252-48e2-b014-b917007e39dd\") " pod="openshift-multus/multus-xrbgl" Jul 09 16:15:17.370734 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.338304 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bed43223-f252-48e2-b014-b917007e39dd-multus-cni-dir\") pod \"multus-xrbgl\" (UID: \"bed43223-f252-48e2-b014-b917007e39dd\") " pod="openshift-multus/multus-xrbgl" Jul 09 16:15:17.370734 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.338325 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpwq5\" (UniqueName: \"kubernetes.io/projected/1a7bbb58-c6a9-4763-a12b-c2ae62a5beaf-kube-api-access-fpwq5\") pod \"multus-additional-cni-plugins-6q72v\" (UID: \"1a7bbb58-c6a9-4763-a12b-c2ae62a5beaf\") " pod="openshift-multus/multus-additional-cni-plugins-6q72v" Jul 09 16:15:17.370734 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.338347 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/98ab9c4f-0fd3-485f-8d0e-af1257f14d51-etc-modprobe-d\") pod \"tuned-q4flk\" (UID: \"98ab9c4f-0fd3-485f-8d0e-af1257f14d51\") " pod="openshift-cluster-node-tuning-operator/tuned-q4flk" Jul 09 16:15:17.370734 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.338368 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/98ab9c4f-0fd3-485f-8d0e-af1257f14d51-etc-systemd\") pod \"tuned-q4flk\" (UID: \"98ab9c4f-0fd3-485f-8d0e-af1257f14d51\") " pod="openshift-cluster-node-tuning-operator/tuned-q4flk" Jul 09 16:15:17.370734 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.338414 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/09e70b88-01bd-4114-93c0-fb224d649e1d-registration-dir\") pod \"aws-ebs-csi-driver-node-ckqxw\" (UID: \"09e70b88-01bd-4114-93c0-fb224d649e1d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ckqxw" Jul 09 16:15:17.370734 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.338434 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/17313664-7a0f-472b-a597-39f8c5412bc3-host-kubelet\") pod \"ovnkube-node-49sqd\" (UID: \"17313664-7a0f-472b-a597-39f8c5412bc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-49sqd" Jul 09 16:15:17.370734 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.338457 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bed43223-f252-48e2-b014-b917007e39dd-multus-socket-dir-parent\") pod \"multus-xrbgl\" (UID: \"bed43223-f252-48e2-b014-b917007e39dd\") " pod="openshift-multus/multus-xrbgl" Jul 09 16:15:17.370734 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.338478 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1a7bbb58-c6a9-4763-a12b-c2ae62a5beaf-cni-binary-copy\") pod \"multus-additional-cni-plugins-6q72v\" (UID: \"1a7bbb58-c6a9-4763-a12b-c2ae62a5beaf\") " pod="openshift-multus/multus-additional-cni-plugins-6q72v" Jul 09 16:15:17.370734 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.338498 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3bd017f3-bffb-4016-bdac-48c6fe9fa352-original-pull-secret\") pod \"global-pull-secret-syncer-f6wzq\" (UID: \"3bd017f3-bffb-4016-bdac-48c6fe9fa352\") " pod="kube-system/global-pull-secret-syncer-f6wzq" Jul 09 16:15:17.370734 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.338518 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/17313664-7a0f-472b-a597-39f8c5412bc3-var-lib-openvswitch\") pod \"ovnkube-node-49sqd\" (UID: \"17313664-7a0f-472b-a597-39f8c5412bc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-49sqd" Jul 09 16:15:17.372617 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.338539 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/17313664-7a0f-472b-a597-39f8c5412bc3-run-openvswitch\") pod \"ovnkube-node-49sqd\" (UID: \"17313664-7a0f-472b-a597-39f8c5412bc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-49sqd" Jul 09 16:15:17.372617 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.338563 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/17313664-7a0f-472b-a597-39f8c5412bc3-host-run-ovn-kubernetes\") pod \"ovnkube-node-49sqd\" (UID: \"17313664-7a0f-472b-a597-39f8c5412bc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-49sqd" Jul 09 16:15:17.372617 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.338586 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bed43223-f252-48e2-b014-b917007e39dd-cnibin\") pod \"multus-xrbgl\" (UID: \"bed43223-f252-48e2-b014-b917007e39dd\") " pod="openshift-multus/multus-xrbgl" Jul 09 16:15:17.372617 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.338609 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bed43223-f252-48e2-b014-b917007e39dd-host-run-netns\") pod \"multus-xrbgl\" (UID: \"bed43223-f252-48e2-b014-b917007e39dd\") " pod="openshift-multus/multus-xrbgl" Jul 09 16:15:17.372617 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.338642 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hz52\" (UniqueName: \"kubernetes.io/projected/6f45ad4f-0e3f-432e-8328-715ec5f97a69-kube-api-access-6hz52\") pod \"network-metrics-daemon-b7sxk\" (UID: \"6f45ad4f-0e3f-432e-8328-715ec5f97a69\") " pod="openshift-multus/network-metrics-daemon-b7sxk" Jul 09 16:15:17.372617 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.338651 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88be5834-3fd7-45da-83d5-4cb1d0f84c75-tmp-dir\") pod \"node-resolver-xx9mg\" (UID: \"88be5834-3fd7-45da-83d5-4cb1d0f84c75\") " pod="openshift-dns/node-resolver-xx9mg" Jul 09 16:15:17.372617 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.338665 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/09e70b88-01bd-4114-93c0-fb224d649e1d-sys-fs\") pod \"aws-ebs-csi-driver-node-ckqxw\" (UID: \"09e70b88-01bd-4114-93c0-fb224d649e1d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ckqxw" Jul 09 16:15:17.372617 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.338689 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/17313664-7a0f-472b-a597-39f8c5412bc3-env-overrides\") pod \"ovnkube-node-49sqd\" (UID: \"17313664-7a0f-472b-a597-39f8c5412bc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-49sqd" Jul 09 16:15:17.372617 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.338726 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bed43223-f252-48e2-b014-b917007e39dd-cni-binary-copy\") pod \"multus-xrbgl\" (UID: \"bed43223-f252-48e2-b014-b917007e39dd\") " pod="openshift-multus/multus-xrbgl" Jul 09 16:15:17.372617 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.338750 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bed43223-f252-48e2-b014-b917007e39dd-host-var-lib-cni-multus\") pod \"multus-xrbgl\" (UID: \"bed43223-f252-48e2-b014-b917007e39dd\") " pod="openshift-multus/multus-xrbgl" Jul 09 16:15:17.372617 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.338771 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bed43223-f252-48e2-b014-b917007e39dd-hostroot\") pod \"multus-xrbgl\" (UID: \"bed43223-f252-48e2-b014-b917007e39dd\") " pod="openshift-multus/multus-xrbgl" Jul 09 16:15:17.372617 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.338794 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n9s2\" (UniqueName: \"kubernetes.io/projected/bed43223-f252-48e2-b014-b917007e39dd-kube-api-access-2n9s2\") pod \"multus-xrbgl\" (UID: \"bed43223-f252-48e2-b014-b917007e39dd\") " pod="openshift-multus/multus-xrbgl" Jul 09 16:15:17.372617 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.338815 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1a7bbb58-c6a9-4763-a12b-c2ae62a5beaf-os-release\") pod \"multus-additional-cni-plugins-6q72v\" (UID: \"1a7bbb58-c6a9-4763-a12b-c2ae62a5beaf\") " pod="openshift-multus/multus-additional-cni-plugins-6q72v" Jul 09 16:15:17.372617 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.338847 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5qdf\" (UniqueName: \"kubernetes.io/projected/88be5834-3fd7-45da-83d5-4cb1d0f84c75-kube-api-access-w5qdf\") pod \"node-resolver-xx9mg\" (UID: \"88be5834-3fd7-45da-83d5-4cb1d0f84c75\") " pod="openshift-dns/node-resolver-xx9mg" Jul 09 16:15:17.372617 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.338854 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bed43223-f252-48e2-b014-b917007e39dd-multus-socket-dir-parent\") pod \"multus-xrbgl\" (UID: \"bed43223-f252-48e2-b014-b917007e39dd\") " pod="openshift-multus/multus-xrbgl" Jul 09 16:15:17.372617 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.338870 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f45ad4f-0e3f-432e-8328-715ec5f97a69-metrics-certs\") pod \"network-metrics-daemon-b7sxk\" (UID: \"6f45ad4f-0e3f-432e-8328-715ec5f97a69\") " pod="openshift-multus/network-metrics-daemon-b7sxk" Jul 09 16:15:17.372617 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.338899 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/98ab9c4f-0fd3-485f-8d0e-af1257f14d51-etc-tuned\") pod \"tuned-q4flk\" (UID: \"98ab9c4f-0fd3-485f-8d0e-af1257f14d51\") " pod="openshift-cluster-node-tuning-operator/tuned-q4flk" Jul 09 16:15:17.373580 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.338926 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bed43223-f252-48e2-b014-b917007e39dd-host-var-lib-cni-bin\") pod \"multus-xrbgl\" (UID: \"bed43223-f252-48e2-b014-b917007e39dd\") " pod="openshift-multus/multus-xrbgl" Jul 09 16:15:17.373580 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.338963 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bed43223-f252-48e2-b014-b917007e39dd-host-var-lib-kubelet\") pod \"multus-xrbgl\" (UID: \"bed43223-f252-48e2-b014-b917007e39dd\") " pod="openshift-multus/multus-xrbgl" Jul 09 16:15:17.373580 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.338970 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bed43223-f252-48e2-b014-b917007e39dd-system-cni-dir\") pod \"multus-xrbgl\" (UID: \"bed43223-f252-48e2-b014-b917007e39dd\") " pod="openshift-multus/multus-xrbgl" Jul 09 16:15:17.373580 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.338983 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1a7bbb58-c6a9-4763-a12b-c2ae62a5beaf-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6q72v\" (UID: \"1a7bbb58-c6a9-4763-a12b-c2ae62a5beaf\") " pod="openshift-multus/multus-additional-cni-plugins-6q72v" Jul 09 16:15:17.373580 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.339069 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1a7bbb58-c6a9-4763-a12b-c2ae62a5beaf-cnibin\") pod \"multus-additional-cni-plugins-6q72v\" (UID: \"1a7bbb58-c6a9-4763-a12b-c2ae62a5beaf\") " pod="openshift-multus/multus-additional-cni-plugins-6q72v" Jul 09 16:15:17.373580 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.339115 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bed43223-f252-48e2-b014-b917007e39dd-multus-cni-dir\") pod \"multus-xrbgl\" (UID: \"bed43223-f252-48e2-b014-b917007e39dd\") " pod="openshift-multus/multus-xrbgl" Jul 09 16:15:17.373580 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.339448 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1a7bbb58-c6a9-4763-a12b-c2ae62a5beaf-cni-binary-copy\") pod \"multus-additional-cni-plugins-6q72v\" (UID: \"1a7bbb58-c6a9-4763-a12b-c2ae62a5beaf\") " pod="openshift-multus/multus-additional-cni-plugins-6q72v" Jul 09 16:15:17.373580 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.339466 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c17a000a-a79a-4dd8-af71-1fc47077bea7-serviceca\") pod \"node-ca-jj6qs\" (UID: \"c17a000a-a79a-4dd8-af71-1fc47077bea7\") " pod="openshift-image-registry/node-ca-jj6qs" Jul 09 16:15:17.373580 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.338988 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1a7bbb58-c6a9-4763-a12b-c2ae62a5beaf-cnibin\") pod \"multus-additional-cni-plugins-6q72v\" (UID: \"1a7bbb58-c6a9-4763-a12b-c2ae62a5beaf\") " pod="openshift-multus/multus-additional-cni-plugins-6q72v" Jul 09 16:15:17.373580 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.339544 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bed43223-f252-48e2-b014-b917007e39dd-host-var-lib-cni-multus\") pod \"multus-xrbgl\" (UID: \"bed43223-f252-48e2-b014-b917007e39dd\") " pod="openshift-multus/multus-xrbgl" Jul 09 16:15:17.373580 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.339548 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bed43223-f252-48e2-b014-b917007e39dd-cnibin\") pod \"multus-xrbgl\" (UID: \"bed43223-f252-48e2-b014-b917007e39dd\") " pod="openshift-multus/multus-xrbgl" Jul 09 16:15:17.373580 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.339730 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bed43223-f252-48e2-b014-b917007e39dd-host-var-lib-cni-bin\") pod \"multus-xrbgl\" (UID: \"bed43223-f252-48e2-b014-b917007e39dd\") " pod="openshift-multus/multus-xrbgl" Jul 09 16:15:17.373580 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.339789 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bed43223-f252-48e2-b014-b917007e39dd-hostroot\") pod \"multus-xrbgl\" (UID: \"bed43223-f252-48e2-b014-b917007e39dd\") " pod="openshift-multus/multus-xrbgl" Jul 09 16:15:17.373580 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.339825 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/98ab9c4f-0fd3-485f-8d0e-af1257f14d51-run\") pod \"tuned-q4flk\" (UID: \"98ab9c4f-0fd3-485f-8d0e-af1257f14d51\") " pod="openshift-cluster-node-tuning-operator/tuned-q4flk" Jul 09 16:15:17.373580 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.339834 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bed43223-f252-48e2-b014-b917007e39dd-host-run-netns\") pod \"multus-xrbgl\" (UID: \"bed43223-f252-48e2-b014-b917007e39dd\") " pod="openshift-multus/multus-xrbgl" Jul 09 16:15:17.373580 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.339851 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/98ab9c4f-0fd3-485f-8d0e-af1257f14d51-lib-modules\") pod \"tuned-q4flk\" (UID: \"98ab9c4f-0fd3-485f-8d0e-af1257f14d51\") " pod="openshift-cluster-node-tuning-operator/tuned-q4flk" Jul 09 16:15:17.373580 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.339879 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srj2j\" (UniqueName: \"kubernetes.io/projected/98ab9c4f-0fd3-485f-8d0e-af1257f14d51-kube-api-access-srj2j\") pod \"tuned-q4flk\" (UID: \"98ab9c4f-0fd3-485f-8d0e-af1257f14d51\") " pod="openshift-cluster-node-tuning-operator/tuned-q4flk" Jul 09 16:15:17.373580 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.339908 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p7td\" (UniqueName: \"kubernetes.io/projected/09e70b88-01bd-4114-93c0-fb224d649e1d-kube-api-access-7p7td\") pod \"aws-ebs-csi-driver-node-ckqxw\" (UID: \"09e70b88-01bd-4114-93c0-fb224d649e1d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ckqxw" Jul 09 16:15:17.375448 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.339915 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bed43223-f252-48e2-b014-b917007e39dd-host-var-lib-kubelet\") pod \"multus-xrbgl\" (UID: \"bed43223-f252-48e2-b014-b917007e39dd\") " pod="openshift-multus/multus-xrbgl" Jul 09 16:15:17.375448 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.339880 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bed43223-f252-48e2-b014-b917007e39dd-cni-binary-copy\") pod \"multus-xrbgl\" (UID: \"bed43223-f252-48e2-b014-b917007e39dd\") " pod="openshift-multus/multus-xrbgl" Jul 09 16:15:17.375448 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.339936 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r99g2\" (UniqueName: \"kubernetes.io/projected/473a10a3-ceca-4c00-915e-a4a21820ce3e-kube-api-access-r99g2\") pod \"network-check-target-kn7sp\" (UID: \"473a10a3-ceca-4c00-915e-a4a21820ce3e\") " pod="openshift-network-diagnostics/network-check-target-kn7sp" Jul 09 16:15:17.375448 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.339938 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1a7bbb58-c6a9-4763-a12b-c2ae62a5beaf-os-release\") pod \"multus-additional-cni-plugins-6q72v\" (UID: \"1a7bbb58-c6a9-4763-a12b-c2ae62a5beaf\") " pod="openshift-multus/multus-additional-cni-plugins-6q72v" Jul 09 16:15:17.375448 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.339998 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bed43223-f252-48e2-b014-b917007e39dd-multus-daemon-config\") pod \"multus-xrbgl\" (UID: \"bed43223-f252-48e2-b014-b917007e39dd\") " pod="openshift-multus/multus-xrbgl" Jul 09 16:15:17.375448 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.340041 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bed43223-f252-48e2-b014-b917007e39dd-etc-kubernetes\") pod \"multus-xrbgl\" (UID: \"bed43223-f252-48e2-b014-b917007e39dd\") " pod="openshift-multus/multus-xrbgl" Jul 09 16:15:17.375448 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.340127 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c17a000a-a79a-4dd8-af71-1fc47077bea7-host\") pod \"node-ca-jj6qs\" (UID: \"c17a000a-a79a-4dd8-af71-1fc47077bea7\") " pod="openshift-image-registry/node-ca-jj6qs" Jul 09 16:15:17.375448 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.340155 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/29d74a8d-7acf-4663-ba5f-40e4884064bd-iptables-alerter-script\") pod \"iptables-alerter-hgqgw\" (UID: \"29d74a8d-7acf-4663-ba5f-40e4884064bd\") " pod="openshift-network-operator/iptables-alerter-hgqgw" Jul 09 16:15:17.375448 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.340181 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e6d9b732-f9d0-48e1-b3a2-b9641eda1c0b-konnectivity-ca\") pod \"konnectivity-agent-zj5qc\" (UID: \"e6d9b732-f9d0-48e1-b3a2-b9641eda1c0b\") " pod="kube-system/konnectivity-agent-zj5qc" Jul 09 16:15:17.375448 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.340184 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bed43223-f252-48e2-b014-b917007e39dd-etc-kubernetes\") pod \"multus-xrbgl\" (UID: \"bed43223-f252-48e2-b014-b917007e39dd\") " pod="openshift-multus/multus-xrbgl" Jul 09 16:15:17.375448 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.340216 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/98ab9c4f-0fd3-485f-8d0e-af1257f14d51-tmp\") pod \"tuned-q4flk\" (UID: \"98ab9c4f-0fd3-485f-8d0e-af1257f14d51\") " pod="openshift-cluster-node-tuning-operator/tuned-q4flk" Jul 09 16:15:17.375448 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.340232 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c17a000a-a79a-4dd8-af71-1fc47077bea7-host\") pod \"node-ca-jj6qs\" (UID: \"c17a000a-a79a-4dd8-af71-1fc47077bea7\") " pod="openshift-image-registry/node-ca-jj6qs" Jul 09 16:15:17.375448 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.340238 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/17313664-7a0f-472b-a597-39f8c5412bc3-run-ovn\") pod \"ovnkube-node-49sqd\" (UID: \"17313664-7a0f-472b-a597-39f8c5412bc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-49sqd" Jul 09 16:15:17.375448 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.340277 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3bd017f3-bffb-4016-bdac-48c6fe9fa352-kubelet-config\") pod \"global-pull-secret-syncer-f6wzq\" (UID: \"3bd017f3-bffb-4016-bdac-48c6fe9fa352\") " pod="kube-system/global-pull-secret-syncer-f6wzq" Jul 09 16:15:17.375448 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.340347 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/17313664-7a0f-472b-a597-39f8c5412bc3-log-socket\") pod \"ovnkube-node-49sqd\" (UID: \"17313664-7a0f-472b-a597-39f8c5412bc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-49sqd" Jul 09 16:15:17.375448 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.340379 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8stpf\" (UniqueName: \"kubernetes.io/projected/29d74a8d-7acf-4663-ba5f-40e4884064bd-kube-api-access-8stpf\") pod \"iptables-alerter-hgqgw\" (UID: \"29d74a8d-7acf-4663-ba5f-40e4884064bd\") " pod="openshift-network-operator/iptables-alerter-hgqgw" Jul 09 16:15:17.375448 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.340416 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/98ab9c4f-0fd3-485f-8d0e-af1257f14d51-etc-sysctl-conf\") pod \"tuned-q4flk\" (UID: \"98ab9c4f-0fd3-485f-8d0e-af1257f14d51\") " pod="openshift-cluster-node-tuning-operator/tuned-q4flk" Jul 09 16:15:17.376413 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.340438 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/17313664-7a0f-472b-a597-39f8c5412bc3-host-run-netns\") pod \"ovnkube-node-49sqd\" (UID: \"17313664-7a0f-472b-a597-39f8c5412bc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-49sqd" Jul 09 16:15:17.376413 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.340461 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/17313664-7a0f-472b-a597-39f8c5412bc3-node-log\") pod \"ovnkube-node-49sqd\" (UID: \"17313664-7a0f-472b-a597-39f8c5412bc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-49sqd" Jul 09 16:15:17.376413 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.340501 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/29d74a8d-7acf-4663-ba5f-40e4884064bd-host-slash\") pod \"iptables-alerter-hgqgw\" (UID: \"29d74a8d-7acf-4663-ba5f-40e4884064bd\") " pod="openshift-network-operator/iptables-alerter-hgqgw" Jul 09 16:15:17.376413 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.340542 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3bd017f3-bffb-4016-bdac-48c6fe9fa352-dbus\") pod \"global-pull-secret-syncer-f6wzq\" (UID: \"3bd017f3-bffb-4016-bdac-48c6fe9fa352\") " pod="kube-system/global-pull-secret-syncer-f6wzq" Jul 09 16:15:17.376413 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.340568 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/17313664-7a0f-472b-a597-39f8c5412bc3-ovnkube-config\") pod \"ovnkube-node-49sqd\" (UID: \"17313664-7a0f-472b-a597-39f8c5412bc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-49sqd" Jul 09 16:15:17.376413 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.340597 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztpfp\" (UniqueName: \"kubernetes.io/projected/c17a000a-a79a-4dd8-af71-1fc47077bea7-kube-api-access-ztpfp\") pod \"node-ca-jj6qs\" (UID: \"c17a000a-a79a-4dd8-af71-1fc47077bea7\") " pod="openshift-image-registry/node-ca-jj6qs" Jul 09 16:15:17.376413 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.340631 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a7bbb58-c6a9-4763-a12b-c2ae62a5beaf-system-cni-dir\") pod \"multus-additional-cni-plugins-6q72v\" (UID: \"1a7bbb58-c6a9-4763-a12b-c2ae62a5beaf\") " pod="openshift-multus/multus-additional-cni-plugins-6q72v" Jul 09 16:15:17.376413 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.340566 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bed43223-f252-48e2-b014-b917007e39dd-multus-daemon-config\") pod \"multus-xrbgl\" (UID: \"bed43223-f252-48e2-b014-b917007e39dd\") " pod="openshift-multus/multus-xrbgl" Jul 09 16:15:17.376413 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.340718 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/98ab9c4f-0fd3-485f-8d0e-af1257f14d51-host\") pod \"tuned-q4flk\" (UID: \"98ab9c4f-0fd3-485f-8d0e-af1257f14d51\") " pod="openshift-cluster-node-tuning-operator/tuned-q4flk" Jul 09 16:15:17.376413 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.340742 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/17313664-7a0f-472b-a597-39f8c5412bc3-host-slash\") pod \"ovnkube-node-49sqd\" (UID: \"17313664-7a0f-472b-a597-39f8c5412bc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-49sqd" Jul 09 16:15:17.376413 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.340763 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/17313664-7a0f-472b-a597-39f8c5412bc3-run-systemd\") pod \"ovnkube-node-49sqd\" (UID: \"17313664-7a0f-472b-a597-39f8c5412bc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-49sqd" Jul 09 16:15:17.376413 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.340788 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/17313664-7a0f-472b-a597-39f8c5412bc3-etc-openvswitch\") pod \"ovnkube-node-49sqd\" (UID: \"17313664-7a0f-472b-a597-39f8c5412bc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-49sqd" Jul 09 16:15:17.376413 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.340812 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/17313664-7a0f-472b-a597-39f8c5412bc3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-49sqd\" (UID: \"17313664-7a0f-472b-a597-39f8c5412bc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-49sqd" Jul 09 16:15:17.376413 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.340839 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/17313664-7a0f-472b-a597-39f8c5412bc3-ovn-node-metrics-cert\") pod \"ovnkube-node-49sqd\" (UID: \"17313664-7a0f-472b-a597-39f8c5412bc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-49sqd" Jul 09 16:15:17.376413 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.340874 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bed43223-f252-48e2-b014-b917007e39dd-multus-conf-dir\") pod \"multus-xrbgl\" (UID: \"bed43223-f252-48e2-b014-b917007e39dd\") " pod="openshift-multus/multus-xrbgl" Jul 09 16:15:17.376413 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.340962 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/29d74a8d-7acf-4663-ba5f-40e4884064bd-iptables-alerter-script\") pod \"iptables-alerter-hgqgw\" (UID: \"29d74a8d-7acf-4663-ba5f-40e4884064bd\") " pod="openshift-network-operator/iptables-alerter-hgqgw" Jul 09 16:15:17.376413 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.341007 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a7bbb58-c6a9-4763-a12b-c2ae62a5beaf-system-cni-dir\") pod \"multus-additional-cni-plugins-6q72v\" (UID: \"1a7bbb58-c6a9-4763-a12b-c2ae62a5beaf\") " pod="openshift-multus/multus-additional-cni-plugins-6q72v" Jul 09 16:15:17.378713 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.341094 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bed43223-f252-48e2-b014-b917007e39dd-host-run-multus-certs\") pod \"multus-xrbgl\" (UID: \"bed43223-f252-48e2-b014-b917007e39dd\") " pod="openshift-multus/multus-xrbgl" Jul 09 16:15:17.378713 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.341131 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bed43223-f252-48e2-b014-b917007e39dd-multus-conf-dir\") pod \"multus-xrbgl\" (UID: \"bed43223-f252-48e2-b014-b917007e39dd\") " pod="openshift-multus/multus-xrbgl" Jul 09 16:15:17.378713 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.341171 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/29d74a8d-7acf-4663-ba5f-40e4884064bd-host-slash\") pod \"iptables-alerter-hgqgw\" (UID: \"29d74a8d-7acf-4663-ba5f-40e4884064bd\") " pod="openshift-network-operator/iptables-alerter-hgqgw" Jul 09 16:15:17.378713 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.341295 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/88be5834-3fd7-45da-83d5-4cb1d0f84c75-hosts-file\") pod \"node-resolver-xx9mg\" (UID: \"88be5834-3fd7-45da-83d5-4cb1d0f84c75\") " pod="openshift-dns/node-resolver-xx9mg" Jul 09 16:15:17.378713 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.341339 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bed43223-f252-48e2-b014-b917007e39dd-host-run-multus-certs\") pod \"multus-xrbgl\" (UID: \"bed43223-f252-48e2-b014-b917007e39dd\") " pod="openshift-multus/multus-xrbgl" Jul 09 16:15:17.378713 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.341387 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/98ab9c4f-0fd3-485f-8d0e-af1257f14d51-etc-sysconfig\") pod \"tuned-q4flk\" (UID: \"98ab9c4f-0fd3-485f-8d0e-af1257f14d51\") " pod="openshift-cluster-node-tuning-operator/tuned-q4flk" Jul 09 16:15:17.378713 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.341443 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/88be5834-3fd7-45da-83d5-4cb1d0f84c75-hosts-file\") pod \"node-resolver-xx9mg\" (UID: \"88be5834-3fd7-45da-83d5-4cb1d0f84c75\") " pod="openshift-dns/node-resolver-xx9mg" Jul 09 16:15:17.378713 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.341478 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/17313664-7a0f-472b-a597-39f8c5412bc3-systemd-units\") pod \"ovnkube-node-49sqd\" (UID: \"17313664-7a0f-472b-a597-39f8c5412bc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-49sqd" Jul 09 16:15:17.378713 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.341502 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7rmg\" (UniqueName: \"kubernetes.io/projected/17313664-7a0f-472b-a597-39f8c5412bc3-kube-api-access-m7rmg\") pod \"ovnkube-node-49sqd\" (UID: \"17313664-7a0f-472b-a597-39f8c5412bc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-49sqd" Jul 09 16:15:17.378713 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.341538 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e6d9b732-f9d0-48e1-b3a2-b9641eda1c0b-agent-certs\") pod \"konnectivity-agent-zj5qc\" (UID: \"e6d9b732-f9d0-48e1-b3a2-b9641eda1c0b\") " pod="kube-system/konnectivity-agent-zj5qc" Jul 09 16:15:17.378713 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.341575 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98ab9c4f-0fd3-485f-8d0e-af1257f14d51-etc-kubernetes\") pod \"tuned-q4flk\" (UID: \"98ab9c4f-0fd3-485f-8d0e-af1257f14d51\") " pod="openshift-cluster-node-tuning-operator/tuned-q4flk" Jul 09 16:15:17.378713 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.341613 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/98ab9c4f-0fd3-485f-8d0e-af1257f14d51-etc-sysctl-d\") pod \"tuned-q4flk\" (UID: \"98ab9c4f-0fd3-485f-8d0e-af1257f14d51\") " pod="openshift-cluster-node-tuning-operator/tuned-q4flk" Jul 09 16:15:17.378713 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.341641 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/98ab9c4f-0fd3-485f-8d0e-af1257f14d51-var-lib-kubelet\") pod \"tuned-q4flk\" (UID: \"98ab9c4f-0fd3-485f-8d0e-af1257f14d51\") " pod="openshift-cluster-node-tuning-operator/tuned-q4flk" Jul 09 16:15:17.378713 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.341672 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/09e70b88-01bd-4114-93c0-fb224d649e1d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ckqxw\" (UID: \"09e70b88-01bd-4114-93c0-fb224d649e1d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ckqxw" Jul 09 16:15:17.378713 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.341697 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/17313664-7a0f-472b-a597-39f8c5412bc3-host-cni-netd\") pod \"ovnkube-node-49sqd\" (UID: \"17313664-7a0f-472b-a597-39f8c5412bc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-49sqd" Jul 09 16:15:17.378713 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.341725 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/17313664-7a0f-472b-a597-39f8c5412bc3-ovnkube-script-lib\") pod \"ovnkube-node-49sqd\" (UID: \"17313664-7a0f-472b-a597-39f8c5412bc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-49sqd" Jul 09 16:15:17.378713 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.341763 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1a7bbb58-c6a9-4763-a12b-c2ae62a5beaf-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6q72v\" (UID: \"1a7bbb58-c6a9-4763-a12b-c2ae62a5beaf\") " pod="openshift-multus/multus-additional-cni-plugins-6q72v" Jul 09 16:15:17.379668 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.341799 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1a7bbb58-c6a9-4763-a12b-c2ae62a5beaf-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6q72v\" (UID: \"1a7bbb58-c6a9-4763-a12b-c2ae62a5beaf\") " pod="openshift-multus/multus-additional-cni-plugins-6q72v" Jul 09 16:15:17.379668 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.341828 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/09e70b88-01bd-4114-93c0-fb224d649e1d-socket-dir\") pod \"aws-ebs-csi-driver-node-ckqxw\" (UID: \"09e70b88-01bd-4114-93c0-fb224d649e1d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ckqxw" Jul 09 16:15:17.379668 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.341863 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/09e70b88-01bd-4114-93c0-fb224d649e1d-etc-selinux\") pod \"aws-ebs-csi-driver-node-ckqxw\" (UID: \"09e70b88-01bd-4114-93c0-fb224d649e1d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ckqxw" Jul 09 16:15:17.379668 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.341899 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/17313664-7a0f-472b-a597-39f8c5412bc3-host-cni-bin\") pod \"ovnkube-node-49sqd\" (UID: \"17313664-7a0f-472b-a597-39f8c5412bc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-49sqd" Jul 09 16:15:17.379668 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.341936 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bed43223-f252-48e2-b014-b917007e39dd-os-release\") pod \"multus-xrbgl\" (UID: \"bed43223-f252-48e2-b014-b917007e39dd\") " pod="openshift-multus/multus-xrbgl" Jul 09 16:15:17.379668 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.341961 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bed43223-f252-48e2-b014-b917007e39dd-host-run-k8s-cni-cncf-io\") pod \"multus-xrbgl\" (UID: \"bed43223-f252-48e2-b014-b917007e39dd\") " pod="openshift-multus/multus-xrbgl" Jul 09 16:15:17.379668 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.342003 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1a7bbb58-c6a9-4763-a12b-c2ae62a5beaf-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6q72v\" (UID: \"1a7bbb58-c6a9-4763-a12b-c2ae62a5beaf\") " pod="openshift-multus/multus-additional-cni-plugins-6q72v" Jul 09 16:15:17.379668 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.342029 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bed43223-f252-48e2-b014-b917007e39dd-host-run-k8s-cni-cncf-io\") pod \"multus-xrbgl\" (UID: \"bed43223-f252-48e2-b014-b917007e39dd\") " pod="openshift-multus/multus-xrbgl" Jul 09 16:15:17.379668 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.342087 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bed43223-f252-48e2-b014-b917007e39dd-os-release\") pod \"multus-xrbgl\" (UID: \"bed43223-f252-48e2-b014-b917007e39dd\") " pod="openshift-multus/multus-xrbgl" Jul 09 16:15:17.379668 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.342295 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1a7bbb58-c6a9-4763-a12b-c2ae62a5beaf-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6q72v\" (UID: \"1a7bbb58-c6a9-4763-a12b-c2ae62a5beaf\") " pod="openshift-multus/multus-additional-cni-plugins-6q72v" Jul 09 16:15:17.379668 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.349232 2569 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 09 16:15:17.379668 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.354010 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztpfp\" (UniqueName: \"kubernetes.io/projected/c17a000a-a79a-4dd8-af71-1fc47077bea7-kube-api-access-ztpfp\") pod \"node-ca-jj6qs\" (UID: \"c17a000a-a79a-4dd8-af71-1fc47077bea7\") " pod="openshift-image-registry/node-ca-jj6qs" Jul 09 16:15:17.379668 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.364094 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8stpf\" (UniqueName: \"kubernetes.io/projected/29d74a8d-7acf-4663-ba5f-40e4884064bd-kube-api-access-8stpf\") pod \"iptables-alerter-hgqgw\" (UID: \"29d74a8d-7acf-4663-ba5f-40e4884064bd\") " pod="openshift-network-operator/iptables-alerter-hgqgw" Jul 09 16:15:17.379668 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.364150 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpwq5\" (UniqueName: \"kubernetes.io/projected/1a7bbb58-c6a9-4763-a12b-c2ae62a5beaf-kube-api-access-fpwq5\") pod \"multus-additional-cni-plugins-6q72v\" (UID: \"1a7bbb58-c6a9-4763-a12b-c2ae62a5beaf\") " pod="openshift-multus/multus-additional-cni-plugins-6q72v" Jul 09 16:15:17.379668 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.364514 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n9s2\" (UniqueName: \"kubernetes.io/projected/bed43223-f252-48e2-b014-b917007e39dd-kube-api-access-2n9s2\") pod \"multus-xrbgl\" (UID: \"bed43223-f252-48e2-b014-b917007e39dd\") " pod="openshift-multus/multus-xrbgl" Jul 09 16:15:17.379668 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.366110 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5qdf\" (UniqueName: \"kubernetes.io/projected/88be5834-3fd7-45da-83d5-4cb1d0f84c75-kube-api-access-w5qdf\") pod \"node-resolver-xx9mg\" (UID: \"88be5834-3fd7-45da-83d5-4cb1d0f84c75\") " pod="openshift-dns/node-resolver-xx9mg" Jul 09 16:15:17.387103 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.385741 2569 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 16:15:17.387830 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.387251 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hgqgw" Jul 09 16:15:17.455775 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.392955 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-114.ec2.internal" event={"ID":"ef080aa212d8ff70e6e7c4e03505fc8a","Type":"ContainerStarted","Data":"dab292dc388124a5d88da4a16cfe21ce0a08e12547ca36f972d27aa3b72ed793"} Jul 09 16:15:17.455775 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.395222 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-114.ec2.internal" event={"ID":"68e180d1aa487674ae66c09f4bc0df95","Type":"ContainerStarted","Data":"ab6c74cc84dd37dfd4303c61be5c7d67ffa82a4abdc53d2e5fb2dd4ff794cae7"} Jul 09 16:15:17.455775 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:17.404616 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29d74a8d_7acf_4663_ba5f_40e4884064bd.slice/crio-ebe1596b38ae84146906609595d0ffb52bacde6577aaad69337c4f2f125662ed WatchSource:0}: Error finding container ebe1596b38ae84146906609595d0ffb52bacde6577aaad69337c4f2f125662ed: Status 404 returned error can't find the container with id ebe1596b38ae84146906609595d0ffb52bacde6577aaad69337c4f2f125662ed Jul 09 16:15:17.455775 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.442204 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/09e70b88-01bd-4114-93c0-fb224d649e1d-device-dir\") pod \"aws-ebs-csi-driver-node-ckqxw\" (UID: \"09e70b88-01bd-4114-93c0-fb224d649e1d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ckqxw" Jul 09 16:15:17.455775 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.442243 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/98ab9c4f-0fd3-485f-8d0e-af1257f14d51-etc-modprobe-d\") pod \"tuned-q4flk\" (UID: \"98ab9c4f-0fd3-485f-8d0e-af1257f14d51\") " pod="openshift-cluster-node-tuning-operator/tuned-q4flk" Jul 09 16:15:17.455775 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.442268 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/98ab9c4f-0fd3-485f-8d0e-af1257f14d51-etc-systemd\") pod \"tuned-q4flk\" (UID: \"98ab9c4f-0fd3-485f-8d0e-af1257f14d51\") " pod="openshift-cluster-node-tuning-operator/tuned-q4flk" Jul 09 16:15:17.455775 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.442293 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/09e70b88-01bd-4114-93c0-fb224d649e1d-registration-dir\") pod \"aws-ebs-csi-driver-node-ckqxw\" (UID: \"09e70b88-01bd-4114-93c0-fb224d649e1d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ckqxw" Jul 09 16:15:17.455775 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.442324 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/17313664-7a0f-472b-a597-39f8c5412bc3-host-kubelet\") pod \"ovnkube-node-49sqd\" (UID: \"17313664-7a0f-472b-a597-39f8c5412bc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-49sqd" Jul 09 16:15:17.455775 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.442353 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3bd017f3-bffb-4016-bdac-48c6fe9fa352-original-pull-secret\") pod \"global-pull-secret-syncer-f6wzq\" (UID: \"3bd017f3-bffb-4016-bdac-48c6fe9fa352\") " pod="kube-system/global-pull-secret-syncer-f6wzq" Jul 09 16:15:17.455775 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.442378 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/17313664-7a0f-472b-a597-39f8c5412bc3-var-lib-openvswitch\") pod \"ovnkube-node-49sqd\" (UID: \"17313664-7a0f-472b-a597-39f8c5412bc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-49sqd" Jul 09 16:15:17.455775 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.442402 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/17313664-7a0f-472b-a597-39f8c5412bc3-run-openvswitch\") pod \"ovnkube-node-49sqd\" (UID: \"17313664-7a0f-472b-a597-39f8c5412bc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-49sqd" Jul 09 16:15:17.455775 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.442426 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/17313664-7a0f-472b-a597-39f8c5412bc3-host-run-ovn-kubernetes\") pod \"ovnkube-node-49sqd\" (UID: \"17313664-7a0f-472b-a597-39f8c5412bc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-49sqd" Jul 09 16:15:17.455775 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.442453 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hz52\" (UniqueName: \"kubernetes.io/projected/6f45ad4f-0e3f-432e-8328-715ec5f97a69-kube-api-access-6hz52\") pod \"network-metrics-daemon-b7sxk\" (UID: \"6f45ad4f-0e3f-432e-8328-715ec5f97a69\") " pod="openshift-multus/network-metrics-daemon-b7sxk" Jul 09 16:15:17.455775 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.442478 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/09e70b88-01bd-4114-93c0-fb224d649e1d-sys-fs\") pod \"aws-ebs-csi-driver-node-ckqxw\" (UID: \"09e70b88-01bd-4114-93c0-fb224d649e1d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ckqxw" Jul 09 16:15:17.455775 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.442503 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/17313664-7a0f-472b-a597-39f8c5412bc3-env-overrides\") pod \"ovnkube-node-49sqd\" (UID: \"17313664-7a0f-472b-a597-39f8c5412bc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-49sqd" Jul 09 16:15:17.455775 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.442538 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f45ad4f-0e3f-432e-8328-715ec5f97a69-metrics-certs\") pod \"network-metrics-daemon-b7sxk\" (UID: \"6f45ad4f-0e3f-432e-8328-715ec5f97a69\") " pod="openshift-multus/network-metrics-daemon-b7sxk" Jul 09 16:15:17.455775 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.442564 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/98ab9c4f-0fd3-485f-8d0e-af1257f14d51-etc-tuned\") pod \"tuned-q4flk\" (UID: \"98ab9c4f-0fd3-485f-8d0e-af1257f14d51\") " pod="openshift-cluster-node-tuning-operator/tuned-q4flk" Jul 09 16:15:17.457395 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.442591 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/98ab9c4f-0fd3-485f-8d0e-af1257f14d51-run\") pod \"tuned-q4flk\" (UID: \"98ab9c4f-0fd3-485f-8d0e-af1257f14d51\") " pod="openshift-cluster-node-tuning-operator/tuned-q4flk" Jul 09 16:15:17.457395 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.442614 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/98ab9c4f-0fd3-485f-8d0e-af1257f14d51-lib-modules\") pod \"tuned-q4flk\" (UID: \"98ab9c4f-0fd3-485f-8d0e-af1257f14d51\") " pod="openshift-cluster-node-tuning-operator/tuned-q4flk" Jul 09 16:15:17.457395 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.442638 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srj2j\" (UniqueName: \"kubernetes.io/projected/98ab9c4f-0fd3-485f-8d0e-af1257f14d51-kube-api-access-srj2j\") pod \"tuned-q4flk\" (UID: \"98ab9c4f-0fd3-485f-8d0e-af1257f14d51\") " pod="openshift-cluster-node-tuning-operator/tuned-q4flk" Jul 09 16:15:17.457395 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.442665 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p7td\" (UniqueName: \"kubernetes.io/projected/09e70b88-01bd-4114-93c0-fb224d649e1d-kube-api-access-7p7td\") pod \"aws-ebs-csi-driver-node-ckqxw\" (UID: \"09e70b88-01bd-4114-93c0-fb224d649e1d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ckqxw" Jul 09 16:15:17.457395 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.442691 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r99g2\" (UniqueName: \"kubernetes.io/projected/473a10a3-ceca-4c00-915e-a4a21820ce3e-kube-api-access-r99g2\") pod \"network-check-target-kn7sp\" (UID: \"473a10a3-ceca-4c00-915e-a4a21820ce3e\") " pod="openshift-network-diagnostics/network-check-target-kn7sp" Jul 09 16:15:17.457395 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.442749 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e6d9b732-f9d0-48e1-b3a2-b9641eda1c0b-konnectivity-ca\") pod \"konnectivity-agent-zj5qc\" (UID: \"e6d9b732-f9d0-48e1-b3a2-b9641eda1c0b\") " pod="kube-system/konnectivity-agent-zj5qc" Jul 09 16:15:17.457395 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.442772 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/98ab9c4f-0fd3-485f-8d0e-af1257f14d51-tmp\") pod \"tuned-q4flk\" (UID: \"98ab9c4f-0fd3-485f-8d0e-af1257f14d51\") " pod="openshift-cluster-node-tuning-operator/tuned-q4flk" Jul 09 16:15:17.457395 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.442795 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/17313664-7a0f-472b-a597-39f8c5412bc3-run-ovn\") pod \"ovnkube-node-49sqd\" (UID: \"17313664-7a0f-472b-a597-39f8c5412bc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-49sqd" Jul 09 16:15:17.457395 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.442826 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3bd017f3-bffb-4016-bdac-48c6fe9fa352-kubelet-config\") pod \"global-pull-secret-syncer-f6wzq\" (UID: \"3bd017f3-bffb-4016-bdac-48c6fe9fa352\") " pod="kube-system/global-pull-secret-syncer-f6wzq" Jul 09 16:15:17.457395 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.443328 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/17313664-7a0f-472b-a597-39f8c5412bc3-log-socket\") pod \"ovnkube-node-49sqd\" (UID: \"17313664-7a0f-472b-a597-39f8c5412bc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-49sqd" Jul 09 16:15:17.457395 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.443377 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/98ab9c4f-0fd3-485f-8d0e-af1257f14d51-etc-sysctl-conf\") pod \"tuned-q4flk\" (UID: \"98ab9c4f-0fd3-485f-8d0e-af1257f14d51\") " pod="openshift-cluster-node-tuning-operator/tuned-q4flk" Jul 09 16:15:17.457395 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.443404 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/17313664-7a0f-472b-a597-39f8c5412bc3-host-run-netns\") pod \"ovnkube-node-49sqd\" (UID: \"17313664-7a0f-472b-a597-39f8c5412bc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-49sqd" Jul 09 16:15:17.457395 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.443443 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/17313664-7a0f-472b-a597-39f8c5412bc3-node-log\") pod \"ovnkube-node-49sqd\" (UID: \"17313664-7a0f-472b-a597-39f8c5412bc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-49sqd" Jul 09 16:15:17.457395 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.443470 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3bd017f3-bffb-4016-bdac-48c6fe9fa352-dbus\") pod \"global-pull-secret-syncer-f6wzq\" (UID: \"3bd017f3-bffb-4016-bdac-48c6fe9fa352\") " pod="kube-system/global-pull-secret-syncer-f6wzq" Jul 09 16:15:17.457395 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.443495 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/17313664-7a0f-472b-a597-39f8c5412bc3-ovnkube-config\") pod \"ovnkube-node-49sqd\" (UID: \"17313664-7a0f-472b-a597-39f8c5412bc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-49sqd" Jul 09 16:15:17.457395 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.443524 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/98ab9c4f-0fd3-485f-8d0e-af1257f14d51-host\") pod \"tuned-q4flk\" (UID: \"98ab9c4f-0fd3-485f-8d0e-af1257f14d51\") " pod="openshift-cluster-node-tuning-operator/tuned-q4flk" Jul 09 16:15:17.457395 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.443543 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/17313664-7a0f-472b-a597-39f8c5412bc3-env-overrides\") pod \"ovnkube-node-49sqd\" (UID: \"17313664-7a0f-472b-a597-39f8c5412bc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-49sqd" Jul 09 16:15:17.459905 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.443550 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/17313664-7a0f-472b-a597-39f8c5412bc3-host-slash\") pod \"ovnkube-node-49sqd\" (UID: \"17313664-7a0f-472b-a597-39f8c5412bc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-49sqd" Jul 09 16:15:17.459905 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.443598 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/17313664-7a0f-472b-a597-39f8c5412bc3-host-slash\") pod \"ovnkube-node-49sqd\" (UID: \"17313664-7a0f-472b-a597-39f8c5412bc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-49sqd" Jul 09 16:15:17.459905 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.443615 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/17313664-7a0f-472b-a597-39f8c5412bc3-run-systemd\") pod \"ovnkube-node-49sqd\" (UID: \"17313664-7a0f-472b-a597-39f8c5412bc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-49sqd" Jul 09 16:15:17.459905 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.443651 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/17313664-7a0f-472b-a597-39f8c5412bc3-etc-openvswitch\") pod \"ovnkube-node-49sqd\" (UID: \"17313664-7a0f-472b-a597-39f8c5412bc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-49sqd" Jul 09 16:15:17.459905 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.443685 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/17313664-7a0f-472b-a597-39f8c5412bc3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-49sqd\" (UID: \"17313664-7a0f-472b-a597-39f8c5412bc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-49sqd" Jul 09 16:15:17.459905 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:17.443709 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 16:15:17.459905 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.443714 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/17313664-7a0f-472b-a597-39f8c5412bc3-ovn-node-metrics-cert\") pod \"ovnkube-node-49sqd\" (UID: \"17313664-7a0f-472b-a597-39f8c5412bc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-49sqd" Jul 09 16:15:17.459905 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.443760 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/98ab9c4f-0fd3-485f-8d0e-af1257f14d51-etc-sysconfig\") pod \"tuned-q4flk\" (UID: \"98ab9c4f-0fd3-485f-8d0e-af1257f14d51\") " pod="openshift-cluster-node-tuning-operator/tuned-q4flk" Jul 09 16:15:17.459905 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:17.443788 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f45ad4f-0e3f-432e-8328-715ec5f97a69-metrics-certs podName:6f45ad4f-0e3f-432e-8328-715ec5f97a69 nodeName:}" failed. No retries permitted until 2026-07-09 16:15:17.943754723 +0000 UTC m=+3.136861104 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6f45ad4f-0e3f-432e-8328-715ec5f97a69-metrics-certs") pod "network-metrics-daemon-b7sxk" (UID: "6f45ad4f-0e3f-432e-8328-715ec5f97a69") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 16:15:17.459905 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.443819 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/17313664-7a0f-472b-a597-39f8c5412bc3-systemd-units\") pod \"ovnkube-node-49sqd\" (UID: \"17313664-7a0f-472b-a597-39f8c5412bc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-49sqd" Jul 09 16:15:17.459905 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.443838 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/98ab9c4f-0fd3-485f-8d0e-af1257f14d51-etc-sysconfig\") pod \"tuned-q4flk\" (UID: \"98ab9c4f-0fd3-485f-8d0e-af1257f14d51\") " pod="openshift-cluster-node-tuning-operator/tuned-q4flk" Jul 09 16:15:17.459905 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.443849 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7rmg\" (UniqueName: \"kubernetes.io/projected/17313664-7a0f-472b-a597-39f8c5412bc3-kube-api-access-m7rmg\") pod \"ovnkube-node-49sqd\" (UID: \"17313664-7a0f-472b-a597-39f8c5412bc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-49sqd" Jul 09 16:15:17.459905 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.443876 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e6d9b732-f9d0-48e1-b3a2-b9641eda1c0b-agent-certs\") pod \"konnectivity-agent-zj5qc\" (UID: \"e6d9b732-f9d0-48e1-b3a2-b9641eda1c0b\") " pod="kube-system/konnectivity-agent-zj5qc" Jul 09 16:15:17.459905 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.443896 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/09e70b88-01bd-4114-93c0-fb224d649e1d-device-dir\") pod \"aws-ebs-csi-driver-node-ckqxw\" (UID: \"09e70b88-01bd-4114-93c0-fb224d649e1d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ckqxw" Jul 09 16:15:17.459905 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.443903 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98ab9c4f-0fd3-485f-8d0e-af1257f14d51-etc-kubernetes\") pod \"tuned-q4flk\" (UID: \"98ab9c4f-0fd3-485f-8d0e-af1257f14d51\") " pod="openshift-cluster-node-tuning-operator/tuned-q4flk" Jul 09 16:15:17.459905 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.443931 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/98ab9c4f-0fd3-485f-8d0e-af1257f14d51-etc-sysctl-d\") pod \"tuned-q4flk\" (UID: \"98ab9c4f-0fd3-485f-8d0e-af1257f14d51\") " pod="openshift-cluster-node-tuning-operator/tuned-q4flk" Jul 09 16:15:17.459905 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.444024 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/17313664-7a0f-472b-a597-39f8c5412bc3-host-run-ovn-kubernetes\") pod \"ovnkube-node-49sqd\" (UID: \"17313664-7a0f-472b-a597-39f8c5412bc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-49sqd" Jul 09 16:15:17.460787 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.444046 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/98ab9c4f-0fd3-485f-8d0e-af1257f14d51-etc-modprobe-d\") pod \"tuned-q4flk\" (UID: \"98ab9c4f-0fd3-485f-8d0e-af1257f14d51\") " pod="openshift-cluster-node-tuning-operator/tuned-q4flk" Jul 09 16:15:17.460787 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.444085 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/98ab9c4f-0fd3-485f-8d0e-af1257f14d51-etc-systemd\") pod \"tuned-q4flk\" (UID: \"98ab9c4f-0fd3-485f-8d0e-af1257f14d51\") " pod="openshift-cluster-node-tuning-operator/tuned-q4flk" Jul 09 16:15:17.460787 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.444140 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/09e70b88-01bd-4114-93c0-fb224d649e1d-registration-dir\") pod \"aws-ebs-csi-driver-node-ckqxw\" (UID: \"09e70b88-01bd-4114-93c0-fb224d649e1d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ckqxw" Jul 09 16:15:17.460787 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.444188 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/17313664-7a0f-472b-a597-39f8c5412bc3-host-kubelet\") pod \"ovnkube-node-49sqd\" (UID: \"17313664-7a0f-472b-a597-39f8c5412bc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-49sqd" Jul 09 16:15:17.460787 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:17.444271 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 16:15:17.460787 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:17.444331 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3bd017f3-bffb-4016-bdac-48c6fe9fa352-original-pull-secret podName:3bd017f3-bffb-4016-bdac-48c6fe9fa352 nodeName:}" failed. No retries permitted until 2026-07-09 16:15:17.944314839 +0000 UTC m=+3.137421199 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3bd017f3-bffb-4016-bdac-48c6fe9fa352-original-pull-secret") pod "global-pull-secret-syncer-f6wzq" (UID: "3bd017f3-bffb-4016-bdac-48c6fe9fa352") : object "kube-system"/"original-pull-secret" not registered Jul 09 16:15:17.460787 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.444334 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/17313664-7a0f-472b-a597-39f8c5412bc3-etc-openvswitch\") pod \"ovnkube-node-49sqd\" (UID: \"17313664-7a0f-472b-a597-39f8c5412bc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-49sqd" Jul 09 16:15:17.460787 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.444365 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/17313664-7a0f-472b-a597-39f8c5412bc3-var-lib-openvswitch\") pod \"ovnkube-node-49sqd\" (UID: \"17313664-7a0f-472b-a597-39f8c5412bc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-49sqd" Jul 09 16:15:17.460787 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.444400 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/09e70b88-01bd-4114-93c0-fb224d649e1d-sys-fs\") pod \"aws-ebs-csi-driver-node-ckqxw\" (UID: \"09e70b88-01bd-4114-93c0-fb224d649e1d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ckqxw" Jul 09 16:15:17.460787 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.444408 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/17313664-7a0f-472b-a597-39f8c5412bc3-run-openvswitch\") pod \"ovnkube-node-49sqd\" (UID: \"17313664-7a0f-472b-a597-39f8c5412bc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-49sqd" Jul 09 16:15:17.460787 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.444444 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/17313664-7a0f-472b-a597-39f8c5412bc3-run-systemd\") pod \"ovnkube-node-49sqd\" (UID: \"17313664-7a0f-472b-a597-39f8c5412bc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-49sqd" Jul 09 16:15:17.460787 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.445040 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/17313664-7a0f-472b-a597-39f8c5412bc3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-49sqd\" (UID: \"17313664-7a0f-472b-a597-39f8c5412bc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-49sqd" Jul 09 16:15:17.460787 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.445139 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/98ab9c4f-0fd3-485f-8d0e-af1257f14d51-lib-modules\") pod \"tuned-q4flk\" (UID: \"98ab9c4f-0fd3-485f-8d0e-af1257f14d51\") " pod="openshift-cluster-node-tuning-operator/tuned-q4flk" Jul 09 16:15:17.460787 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.445210 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/98ab9c4f-0fd3-485f-8d0e-af1257f14d51-run\") pod \"tuned-q4flk\" (UID: \"98ab9c4f-0fd3-485f-8d0e-af1257f14d51\") " pod="openshift-cluster-node-tuning-operator/tuned-q4flk" Jul 09 16:15:17.460787 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.445251 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/17313664-7a0f-472b-a597-39f8c5412bc3-run-ovn\") pod \"ovnkube-node-49sqd\" (UID: \"17313664-7a0f-472b-a597-39f8c5412bc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-49sqd" Jul 09 16:15:17.460787 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.445337 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/98ab9c4f-0fd3-485f-8d0e-af1257f14d51-host\") pod \"tuned-q4flk\" (UID: \"98ab9c4f-0fd3-485f-8d0e-af1257f14d51\") " pod="openshift-cluster-node-tuning-operator/tuned-q4flk" Jul 09 16:15:17.460787 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.445482 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/98ab9c4f-0fd3-485f-8d0e-af1257f14d51-etc-sysctl-conf\") pod \"tuned-q4flk\" (UID: \"98ab9c4f-0fd3-485f-8d0e-af1257f14d51\") " pod="openshift-cluster-node-tuning-operator/tuned-q4flk" Jul 09 16:15:17.464452 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.445529 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/17313664-7a0f-472b-a597-39f8c5412bc3-host-run-netns\") pod \"ovnkube-node-49sqd\" (UID: \"17313664-7a0f-472b-a597-39f8c5412bc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-49sqd" Jul 09 16:15:17.464452 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.445564 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e6d9b732-f9d0-48e1-b3a2-b9641eda1c0b-konnectivity-ca\") pod \"konnectivity-agent-zj5qc\" (UID: \"e6d9b732-f9d0-48e1-b3a2-b9641eda1c0b\") " pod="kube-system/konnectivity-agent-zj5qc" Jul 09 16:15:17.464452 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.445606 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3bd017f3-bffb-4016-bdac-48c6fe9fa352-dbus\") pod \"global-pull-secret-syncer-f6wzq\" (UID: \"3bd017f3-bffb-4016-bdac-48c6fe9fa352\") " pod="kube-system/global-pull-secret-syncer-f6wzq" Jul 09 16:15:17.464452 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.445653 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3bd017f3-bffb-4016-bdac-48c6fe9fa352-kubelet-config\") pod \"global-pull-secret-syncer-f6wzq\" (UID: \"3bd017f3-bffb-4016-bdac-48c6fe9fa352\") " pod="kube-system/global-pull-secret-syncer-f6wzq" Jul 09 16:15:17.464452 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.445697 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/17313664-7a0f-472b-a597-39f8c5412bc3-log-socket\") pod \"ovnkube-node-49sqd\" (UID: \"17313664-7a0f-472b-a597-39f8c5412bc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-49sqd" Jul 09 16:15:17.464452 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.446045 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/17313664-7a0f-472b-a597-39f8c5412bc3-ovnkube-config\") pod \"ovnkube-node-49sqd\" (UID: \"17313664-7a0f-472b-a597-39f8c5412bc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-49sqd" Jul 09 16:15:17.464452 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.446079 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/17313664-7a0f-472b-a597-39f8c5412bc3-systemd-units\") pod \"ovnkube-node-49sqd\" (UID: \"17313664-7a0f-472b-a597-39f8c5412bc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-49sqd" Jul 09 16:15:17.464452 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.446156 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98ab9c4f-0fd3-485f-8d0e-af1257f14d51-etc-kubernetes\") pod \"tuned-q4flk\" (UID: \"98ab9c4f-0fd3-485f-8d0e-af1257f14d51\") " pod="openshift-cluster-node-tuning-operator/tuned-q4flk" Jul 09 16:15:17.464452 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.446190 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/17313664-7a0f-472b-a597-39f8c5412bc3-node-log\") pod \"ovnkube-node-49sqd\" (UID: \"17313664-7a0f-472b-a597-39f8c5412bc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-49sqd" Jul 09 16:15:17.464452 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.446355 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/98ab9c4f-0fd3-485f-8d0e-af1257f14d51-var-lib-kubelet\") pod \"tuned-q4flk\" (UID: \"98ab9c4f-0fd3-485f-8d0e-af1257f14d51\") " pod="openshift-cluster-node-tuning-operator/tuned-q4flk" Jul 09 16:15:17.464452 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.446334 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/98ab9c4f-0fd3-485f-8d0e-af1257f14d51-etc-sysctl-d\") pod \"tuned-q4flk\" (UID: \"98ab9c4f-0fd3-485f-8d0e-af1257f14d51\") " pod="openshift-cluster-node-tuning-operator/tuned-q4flk" Jul 09 16:15:17.464452 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.458331 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/98ab9c4f-0fd3-485f-8d0e-af1257f14d51-var-lib-kubelet\") pod \"tuned-q4flk\" (UID: \"98ab9c4f-0fd3-485f-8d0e-af1257f14d51\") " pod="openshift-cluster-node-tuning-operator/tuned-q4flk" Jul 09 16:15:17.464452 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.458397 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/09e70b88-01bd-4114-93c0-fb224d649e1d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ckqxw\" (UID: \"09e70b88-01bd-4114-93c0-fb224d649e1d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ckqxw" Jul 09 16:15:17.464452 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.458436 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/17313664-7a0f-472b-a597-39f8c5412bc3-host-cni-netd\") pod \"ovnkube-node-49sqd\" (UID: \"17313664-7a0f-472b-a597-39f8c5412bc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-49sqd" Jul 09 16:15:17.464452 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.458469 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/17313664-7a0f-472b-a597-39f8c5412bc3-ovnkube-script-lib\") pod \"ovnkube-node-49sqd\" (UID: \"17313664-7a0f-472b-a597-39f8c5412bc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-49sqd" Jul 09 16:15:17.464452 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.458505 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/09e70b88-01bd-4114-93c0-fb224d649e1d-socket-dir\") pod \"aws-ebs-csi-driver-node-ckqxw\" (UID: \"09e70b88-01bd-4114-93c0-fb224d649e1d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ckqxw" Jul 09 16:15:17.464452 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.458540 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/09e70b88-01bd-4114-93c0-fb224d649e1d-etc-selinux\") pod \"aws-ebs-csi-driver-node-ckqxw\" (UID: \"09e70b88-01bd-4114-93c0-fb224d649e1d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ckqxw" Jul 09 16:15:17.465601 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.458566 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/17313664-7a0f-472b-a597-39f8c5412bc3-host-cni-bin\") pod \"ovnkube-node-49sqd\" (UID: \"17313664-7a0f-472b-a597-39f8c5412bc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-49sqd" Jul 09 16:15:17.465601 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.458603 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/98ab9c4f-0fd3-485f-8d0e-af1257f14d51-sys\") pod \"tuned-q4flk\" (UID: \"98ab9c4f-0fd3-485f-8d0e-af1257f14d51\") " pod="openshift-cluster-node-tuning-operator/tuned-q4flk" Jul 09 16:15:17.465601 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.458712 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/98ab9c4f-0fd3-485f-8d0e-af1257f14d51-sys\") pod \"tuned-q4flk\" (UID: \"98ab9c4f-0fd3-485f-8d0e-af1257f14d51\") " pod="openshift-cluster-node-tuning-operator/tuned-q4flk" Jul 09 16:15:17.465601 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.458787 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/09e70b88-01bd-4114-93c0-fb224d649e1d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ckqxw\" (UID: \"09e70b88-01bd-4114-93c0-fb224d649e1d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ckqxw" Jul 09 16:15:17.465601 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.458818 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e6d9b732-f9d0-48e1-b3a2-b9641eda1c0b-agent-certs\") pod \"konnectivity-agent-zj5qc\" (UID: \"e6d9b732-f9d0-48e1-b3a2-b9641eda1c0b\") " pod="kube-system/konnectivity-agent-zj5qc" Jul 09 16:15:17.465601 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.458839 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/17313664-7a0f-472b-a597-39f8c5412bc3-host-cni-netd\") pod \"ovnkube-node-49sqd\" (UID: \"17313664-7a0f-472b-a597-39f8c5412bc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-49sqd" Jul 09 16:15:17.465601 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.458945 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/98ab9c4f-0fd3-485f-8d0e-af1257f14d51-tmp\") pod \"tuned-q4flk\" (UID: \"98ab9c4f-0fd3-485f-8d0e-af1257f14d51\") " pod="openshift-cluster-node-tuning-operator/tuned-q4flk" Jul 09 16:15:17.465601 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.459069 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/98ab9c4f-0fd3-485f-8d0e-af1257f14d51-etc-tuned\") pod \"tuned-q4flk\" (UID: \"98ab9c4f-0fd3-485f-8d0e-af1257f14d51\") " pod="openshift-cluster-node-tuning-operator/tuned-q4flk" Jul 09 16:15:17.465601 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.459214 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/09e70b88-01bd-4114-93c0-fb224d649e1d-etc-selinux\") pod \"aws-ebs-csi-driver-node-ckqxw\" (UID: \"09e70b88-01bd-4114-93c0-fb224d649e1d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ckqxw" Jul 09 16:15:17.465601 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.459232 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/17313664-7a0f-472b-a597-39f8c5412bc3-ovn-node-metrics-cert\") pod \"ovnkube-node-49sqd\" (UID: \"17313664-7a0f-472b-a597-39f8c5412bc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-49sqd" Jul 09 16:15:17.465601 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:17.459325 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 16:15:17.465601 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.459342 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/09e70b88-01bd-4114-93c0-fb224d649e1d-socket-dir\") pod \"aws-ebs-csi-driver-node-ckqxw\" (UID: \"09e70b88-01bd-4114-93c0-fb224d649e1d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ckqxw" Jul 09 16:15:17.465601 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:17.459349 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 16:15:17.465601 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:17.459371 2569 projected.go:196] Error preparing data for projected volume kube-api-access-r99g2 for pod openshift-network-diagnostics/network-check-target-kn7sp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 16:15:17.465601 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.459409 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/17313664-7a0f-472b-a597-39f8c5412bc3-host-cni-bin\") pod \"ovnkube-node-49sqd\" (UID: \"17313664-7a0f-472b-a597-39f8c5412bc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-49sqd" Jul 09 16:15:17.465601 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.459437 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/17313664-7a0f-472b-a597-39f8c5412bc3-ovnkube-script-lib\") pod \"ovnkube-node-49sqd\" (UID: \"17313664-7a0f-472b-a597-39f8c5412bc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-49sqd" Jul 09 16:15:17.465601 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:17.459450 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/473a10a3-ceca-4c00-915e-a4a21820ce3e-kube-api-access-r99g2 podName:473a10a3-ceca-4c00-915e-a4a21820ce3e nodeName:}" failed. No retries permitted until 2026-07-09 16:15:17.959418277 +0000 UTC m=+3.152524660 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-r99g2" (UniqueName: "kubernetes.io/projected/473a10a3-ceca-4c00-915e-a4a21820ce3e-kube-api-access-r99g2") pod "network-check-target-kn7sp" (UID: "473a10a3-ceca-4c00-915e-a4a21820ce3e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 16:15:17.468669 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.463229 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hz52\" (UniqueName: \"kubernetes.io/projected/6f45ad4f-0e3f-432e-8328-715ec5f97a69-kube-api-access-6hz52\") pod \"network-metrics-daemon-b7sxk\" (UID: \"6f45ad4f-0e3f-432e-8328-715ec5f97a69\") " pod="openshift-multus/network-metrics-daemon-b7sxk" Jul 09 16:15:17.468669 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.465579 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-srj2j\" (UniqueName: \"kubernetes.io/projected/98ab9c4f-0fd3-485f-8d0e-af1257f14d51-kube-api-access-srj2j\") pod \"tuned-q4flk\" (UID: \"98ab9c4f-0fd3-485f-8d0e-af1257f14d51\") " pod="openshift-cluster-node-tuning-operator/tuned-q4flk" Jul 09 16:15:17.468669 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.466199 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p7td\" (UniqueName: \"kubernetes.io/projected/09e70b88-01bd-4114-93c0-fb224d649e1d-kube-api-access-7p7td\") pod \"aws-ebs-csi-driver-node-ckqxw\" (UID: \"09e70b88-01bd-4114-93c0-fb224d649e1d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ckqxw" Jul 09 16:15:17.468669 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.466244 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7rmg\" (UniqueName: \"kubernetes.io/projected/17313664-7a0f-472b-a597-39f8c5412bc3-kube-api-access-m7rmg\") pod \"ovnkube-node-49sqd\" (UID: \"17313664-7a0f-472b-a597-39f8c5412bc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-49sqd" Jul 09 16:15:17.468669 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.467717 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-49sqd" Jul 09 16:15:17.546122 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:17.481750 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17313664_7a0f_472b_a597_39f8c5412bc3.slice/crio-da2e22702af707224053c5252473eb1c80720a594bc5f5fc10c5975befe4bafd WatchSource:0}: Error finding container da2e22702af707224053c5252473eb1c80720a594bc5f5fc10c5975befe4bafd: Status 404 returned error can't find the container with id da2e22702af707224053c5252473eb1c80720a594bc5f5fc10c5975befe4bafd Jul 09 16:15:17.546122 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.528563 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xx9mg" Jul 09 16:15:17.546122 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.537010 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xrbgl" Jul 09 16:15:17.550505 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.546353 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jj6qs" Jul 09 16:15:17.558112 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.553694 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6q72v" Jul 09 16:15:17.724945 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.724846 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-q4flk" Jul 09 16:15:17.736846 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.736813 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ckqxw" Jul 09 16:15:17.747181 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.747155 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zj5qc" Jul 09 16:15:17.751439 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:17.751402 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09e70b88_01bd_4114_93c0_fb224d649e1d.slice/crio-6e2be30eb16f6ebbaa7ad0ac851771b73fc188f5317db3f249991e8fb9caa11b WatchSource:0}: Error finding container 6e2be30eb16f6ebbaa7ad0ac851771b73fc188f5317db3f249991e8fb9caa11b: Status 404 returned error can't find the container with id 6e2be30eb16f6ebbaa7ad0ac851771b73fc188f5317db3f249991e8fb9caa11b Jul 09 16:15:17.759135 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:17.759105 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6d9b732_f9d0_48e1_b3a2_b9641eda1c0b.slice/crio-e83da9cff6655147125f90fb28884efd6409bf8e048cffabc9e355f8d0facd77 WatchSource:0}: Error finding container e83da9cff6655147125f90fb28884efd6409bf8e048cffabc9e355f8d0facd77: Status 404 returned error can't find the container with id e83da9cff6655147125f90fb28884efd6409bf8e048cffabc9e355f8d0facd77 Jul 09 16:15:17.962558 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.962519 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3bd017f3-bffb-4016-bdac-48c6fe9fa352-original-pull-secret\") pod \"global-pull-secret-syncer-f6wzq\" (UID: \"3bd017f3-bffb-4016-bdac-48c6fe9fa352\") " pod="kube-system/global-pull-secret-syncer-f6wzq" Jul 09 16:15:17.962724 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.962582 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f45ad4f-0e3f-432e-8328-715ec5f97a69-metrics-certs\") pod \"network-metrics-daemon-b7sxk\" (UID: \"6f45ad4f-0e3f-432e-8328-715ec5f97a69\") " pod="openshift-multus/network-metrics-daemon-b7sxk" Jul 09 16:15:17.962724 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:17.962618 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r99g2\" (UniqueName: \"kubernetes.io/projected/473a10a3-ceca-4c00-915e-a4a21820ce3e-kube-api-access-r99g2\") pod \"network-check-target-kn7sp\" (UID: \"473a10a3-ceca-4c00-915e-a4a21820ce3e\") " pod="openshift-network-diagnostics/network-check-target-kn7sp" Jul 09 16:15:17.962843 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:17.962790 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 16:15:17.962843 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:17.962815 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 16:15:17.962843 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:17.962829 2569 projected.go:196] Error preparing data for projected volume kube-api-access-r99g2 for pod openshift-network-diagnostics/network-check-target-kn7sp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 16:15:17.962995 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:17.962895 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/473a10a3-ceca-4c00-915e-a4a21820ce3e-kube-api-access-r99g2 podName:473a10a3-ceca-4c00-915e-a4a21820ce3e nodeName:}" failed. No retries permitted until 2026-07-09 16:15:18.962876127 +0000 UTC m=+4.155982503 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-r99g2" (UniqueName: "kubernetes.io/projected/473a10a3-ceca-4c00-915e-a4a21820ce3e-kube-api-access-r99g2") pod "network-check-target-kn7sp" (UID: "473a10a3-ceca-4c00-915e-a4a21820ce3e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 16:15:17.963315 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:17.963294 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 16:15:17.963422 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:17.963343 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3bd017f3-bffb-4016-bdac-48c6fe9fa352-original-pull-secret podName:3bd017f3-bffb-4016-bdac-48c6fe9fa352 nodeName:}" failed. No retries permitted until 2026-07-09 16:15:18.963330936 +0000 UTC m=+4.156437296 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3bd017f3-bffb-4016-bdac-48c6fe9fa352-original-pull-secret") pod "global-pull-secret-syncer-f6wzq" (UID: "3bd017f3-bffb-4016-bdac-48c6fe9fa352") : object "kube-system"/"original-pull-secret" not registered Jul 09 16:15:17.963422 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:17.963406 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 16:15:17.963543 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:17.963438 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f45ad4f-0e3f-432e-8328-715ec5f97a69-metrics-certs podName:6f45ad4f-0e3f-432e-8328-715ec5f97a69 nodeName:}" failed. No retries permitted until 2026-07-09 16:15:18.963429376 +0000 UTC m=+4.156535736 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6f45ad4f-0e3f-432e-8328-715ec5f97a69-metrics-certs") pod "network-metrics-daemon-b7sxk" (UID: "6f45ad4f-0e3f-432e-8328-715ec5f97a69") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 16:15:18.301661 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:18.301614 2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 16:10:17 +0000 UTC" deadline="2027-12-15 16:28:09.725059298 +0000 UTC" Jul 09 16:15:18.301661 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:18.301657 2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12576h12m51.423407133s" Jul 09 16:15:18.383395 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:18.383059 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b7sxk" Jul 09 16:15:18.383395 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:18.383207 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b7sxk" podUID="6f45ad4f-0e3f-432e-8328-715ec5f97a69" Jul 09 16:15:18.398544 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:18.398501 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ckqxw" event={"ID":"09e70b88-01bd-4114-93c0-fb224d649e1d","Type":"ContainerStarted","Data":"6e2be30eb16f6ebbaa7ad0ac851771b73fc188f5317db3f249991e8fb9caa11b"} Jul 09 16:15:18.399996 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:18.399953 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-q4flk" event={"ID":"98ab9c4f-0fd3-485f-8d0e-af1257f14d51","Type":"ContainerStarted","Data":"c1d84697fdc60ef96d6e472a407351e32651be96610ac5ee6dbdebba872e5edc"} Jul 09 16:15:18.401660 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:18.401631 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xx9mg" event={"ID":"88be5834-3fd7-45da-83d5-4cb1d0f84c75","Type":"ContainerStarted","Data":"11416f832b0347140f5b7c7c15a332fcd2122f53a5266916896906016253694b"} Jul 09 16:15:18.403424 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:18.403397 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-49sqd" event={"ID":"17313664-7a0f-472b-a597-39f8c5412bc3","Type":"ContainerStarted","Data":"da2e22702af707224053c5252473eb1c80720a594bc5f5fc10c5975befe4bafd"} Jul 09 16:15:18.405172 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:18.405141 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zj5qc" event={"ID":"e6d9b732-f9d0-48e1-b3a2-b9641eda1c0b","Type":"ContainerStarted","Data":"e83da9cff6655147125f90fb28884efd6409bf8e048cffabc9e355f8d0facd77"} Jul 09 16:15:18.406540 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:18.406503 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6q72v" event={"ID":"1a7bbb58-c6a9-4763-a12b-c2ae62a5beaf","Type":"ContainerStarted","Data":"6125158c227f46fa8faab59738b77cb551c9e589d884fe059a6bcdb7f64e6c55"} Jul 09 16:15:18.408223 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:18.408194 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jj6qs" event={"ID":"c17a000a-a79a-4dd8-af71-1fc47077bea7","Type":"ContainerStarted","Data":"ac584318ac9b4c87efcc139785f290f278f3d5094bfc937885ad3b611948b9bb"} Jul 09 16:15:18.409566 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:18.409539 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xrbgl" event={"ID":"bed43223-f252-48e2-b014-b917007e39dd","Type":"ContainerStarted","Data":"fc5f40b226997c2c7ceb64569f99a239aa67ba4c6590a8469e793044e0bb6c5a"} Jul 09 16:15:18.414783 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:18.414740 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hgqgw" event={"ID":"29d74a8d-7acf-4663-ba5f-40e4884064bd","Type":"ContainerStarted","Data":"ebe1596b38ae84146906609595d0ffb52bacde6577aaad69337c4f2f125662ed"} Jul 09 16:15:18.973781 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:18.973264 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f45ad4f-0e3f-432e-8328-715ec5f97a69-metrics-certs\") pod \"network-metrics-daemon-b7sxk\" (UID: \"6f45ad4f-0e3f-432e-8328-715ec5f97a69\") " pod="openshift-multus/network-metrics-daemon-b7sxk" Jul 09 16:15:18.973781 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:18.973327 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r99g2\" (UniqueName: \"kubernetes.io/projected/473a10a3-ceca-4c00-915e-a4a21820ce3e-kube-api-access-r99g2\") pod \"network-check-target-kn7sp\" (UID: \"473a10a3-ceca-4c00-915e-a4a21820ce3e\") " pod="openshift-network-diagnostics/network-check-target-kn7sp" Jul 09 16:15:18.973781 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:18.973409 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3bd017f3-bffb-4016-bdac-48c6fe9fa352-original-pull-secret\") pod \"global-pull-secret-syncer-f6wzq\" (UID: \"3bd017f3-bffb-4016-bdac-48c6fe9fa352\") " pod="kube-system/global-pull-secret-syncer-f6wzq" Jul 09 16:15:18.973781 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:18.973413 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 16:15:18.973781 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:18.973498 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f45ad4f-0e3f-432e-8328-715ec5f97a69-metrics-certs podName:6f45ad4f-0e3f-432e-8328-715ec5f97a69 nodeName:}" failed. No retries permitted until 2026-07-09 16:15:20.9734734 +0000 UTC m=+6.166579770 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6f45ad4f-0e3f-432e-8328-715ec5f97a69-metrics-certs") pod "network-metrics-daemon-b7sxk" (UID: "6f45ad4f-0e3f-432e-8328-715ec5f97a69") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 16:15:18.973781 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:18.973539 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 16:15:18.973781 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:18.973594 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3bd017f3-bffb-4016-bdac-48c6fe9fa352-original-pull-secret podName:3bd017f3-bffb-4016-bdac-48c6fe9fa352 nodeName:}" failed. No retries permitted until 2026-07-09 16:15:20.973578388 +0000 UTC m=+6.166684763 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3bd017f3-bffb-4016-bdac-48c6fe9fa352-original-pull-secret") pod "global-pull-secret-syncer-f6wzq" (UID: "3bd017f3-bffb-4016-bdac-48c6fe9fa352") : object "kube-system"/"original-pull-secret" not registered Jul 09 16:15:18.973781 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:18.973669 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 16:15:18.973781 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:18.973695 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 16:15:18.973781 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:18.973709 2569 projected.go:196] Error preparing data for projected volume kube-api-access-r99g2 for pod openshift-network-diagnostics/network-check-target-kn7sp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 16:15:18.973781 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:18.973751 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/473a10a3-ceca-4c00-915e-a4a21820ce3e-kube-api-access-r99g2 podName:473a10a3-ceca-4c00-915e-a4a21820ce3e nodeName:}" failed. No retries permitted until 2026-07-09 16:15:20.973740778 +0000 UTC m=+6.166847138 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-r99g2" (UniqueName: "kubernetes.io/projected/473a10a3-ceca-4c00-915e-a4a21820ce3e-kube-api-access-r99g2") pod "network-check-target-kn7sp" (UID: "473a10a3-ceca-4c00-915e-a4a21820ce3e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 16:15:19.302109 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:19.301987 2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 16:10:17 +0000 UTC" deadline="2028-03-03 22:21:12.556235408 +0000 UTC" Jul 09 16:15:19.302109 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:19.302055 2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14478h5m53.25418492s" Jul 09 16:15:19.383303 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:19.383270 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kn7sp" Jul 09 16:15:19.383547 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:19.383507 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kn7sp" podUID="473a10a3-ceca-4c00-915e-a4a21820ce3e" Jul 09 16:15:19.383758 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:19.383730 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6wzq" Jul 09 16:15:19.383948 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:19.383871 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6wzq" podUID="3bd017f3-bffb-4016-bdac-48c6fe9fa352" Jul 09 16:15:20.384126 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:20.384028 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b7sxk" Jul 09 16:15:20.384608 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:20.384195 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b7sxk" podUID="6f45ad4f-0e3f-432e-8328-715ec5f97a69" Jul 09 16:15:20.993115 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:20.992026 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3bd017f3-bffb-4016-bdac-48c6fe9fa352-original-pull-secret\") pod \"global-pull-secret-syncer-f6wzq\" (UID: \"3bd017f3-bffb-4016-bdac-48c6fe9fa352\") " pod="kube-system/global-pull-secret-syncer-f6wzq" Jul 09 16:15:20.993115 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:20.992096 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f45ad4f-0e3f-432e-8328-715ec5f97a69-metrics-certs\") pod \"network-metrics-daemon-b7sxk\" (UID: \"6f45ad4f-0e3f-432e-8328-715ec5f97a69\") " pod="openshift-multus/network-metrics-daemon-b7sxk" Jul 09 16:15:20.993115 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:20.992133 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r99g2\" (UniqueName: \"kubernetes.io/projected/473a10a3-ceca-4c00-915e-a4a21820ce3e-kube-api-access-r99g2\") pod \"network-check-target-kn7sp\" (UID: \"473a10a3-ceca-4c00-915e-a4a21820ce3e\") " pod="openshift-network-diagnostics/network-check-target-kn7sp" Jul 09 16:15:20.993115 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:20.992332 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 16:15:20.993115 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:20.992355 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 16:15:20.993115 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:20.992369 2569 projected.go:196] Error preparing data for projected volume kube-api-access-r99g2 for pod openshift-network-diagnostics/network-check-target-kn7sp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 16:15:20.993115 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:20.992432 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/473a10a3-ceca-4c00-915e-a4a21820ce3e-kube-api-access-r99g2 podName:473a10a3-ceca-4c00-915e-a4a21820ce3e nodeName:}" failed. No retries permitted until 2026-07-09 16:15:24.992412901 +0000 UTC m=+10.185519267 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-r99g2" (UniqueName: "kubernetes.io/projected/473a10a3-ceca-4c00-915e-a4a21820ce3e-kube-api-access-r99g2") pod "network-check-target-kn7sp" (UID: "473a10a3-ceca-4c00-915e-a4a21820ce3e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 16:15:20.993115 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:20.992883 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 16:15:20.993115 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:20.992940 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3bd017f3-bffb-4016-bdac-48c6fe9fa352-original-pull-secret podName:3bd017f3-bffb-4016-bdac-48c6fe9fa352 nodeName:}" failed. No retries permitted until 2026-07-09 16:15:24.992923565 +0000 UTC m=+10.186029929 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3bd017f3-bffb-4016-bdac-48c6fe9fa352-original-pull-secret") pod "global-pull-secret-syncer-f6wzq" (UID: "3bd017f3-bffb-4016-bdac-48c6fe9fa352") : object "kube-system"/"original-pull-secret" not registered Jul 09 16:15:20.993115 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:20.993029 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 16:15:20.993115 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:20.993068 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f45ad4f-0e3f-432e-8328-715ec5f97a69-metrics-certs podName:6f45ad4f-0e3f-432e-8328-715ec5f97a69 nodeName:}" failed. No retries permitted until 2026-07-09 16:15:24.993053557 +0000 UTC m=+10.186159925 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6f45ad4f-0e3f-432e-8328-715ec5f97a69-metrics-certs") pod "network-metrics-daemon-b7sxk" (UID: "6f45ad4f-0e3f-432e-8328-715ec5f97a69") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 16:15:21.383629 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:21.383545 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6wzq" Jul 09 16:15:21.383815 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:21.383682 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6wzq" podUID="3bd017f3-bffb-4016-bdac-48c6fe9fa352" Jul 09 16:15:21.383815 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:21.383545 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kn7sp" Jul 09 16:15:21.383815 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:21.383802 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kn7sp" podUID="473a10a3-ceca-4c00-915e-a4a21820ce3e" Jul 09 16:15:22.383141 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:22.383110 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b7sxk" Jul 09 16:15:22.383603 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:22.383262 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b7sxk" podUID="6f45ad4f-0e3f-432e-8328-715ec5f97a69" Jul 09 16:15:23.390618 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:23.390536 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6wzq" Jul 09 16:15:23.391078 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:23.390670 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6wzq" podUID="3bd017f3-bffb-4016-bdac-48c6fe9fa352" Jul 09 16:15:23.391078 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:23.390691 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kn7sp" Jul 09 16:15:23.391078 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:23.390817 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kn7sp" podUID="473a10a3-ceca-4c00-915e-a4a21820ce3e" Jul 09 16:15:24.383250 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:24.383211 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b7sxk" Jul 09 16:15:24.383436 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:24.383350 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b7sxk" podUID="6f45ad4f-0e3f-432e-8328-715ec5f97a69" Jul 09 16:15:25.025448 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:25.025131 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f45ad4f-0e3f-432e-8328-715ec5f97a69-metrics-certs\") pod \"network-metrics-daemon-b7sxk\" (UID: \"6f45ad4f-0e3f-432e-8328-715ec5f97a69\") " pod="openshift-multus/network-metrics-daemon-b7sxk" Jul 09 16:15:25.025448 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:25.025188 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r99g2\" (UniqueName: \"kubernetes.io/projected/473a10a3-ceca-4c00-915e-a4a21820ce3e-kube-api-access-r99g2\") pod \"network-check-target-kn7sp\" (UID: \"473a10a3-ceca-4c00-915e-a4a21820ce3e\") " pod="openshift-network-diagnostics/network-check-target-kn7sp" Jul 09 16:15:25.025448 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:25.025258 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3bd017f3-bffb-4016-bdac-48c6fe9fa352-original-pull-secret\") pod \"global-pull-secret-syncer-f6wzq\" (UID: \"3bd017f3-bffb-4016-bdac-48c6fe9fa352\") " pod="kube-system/global-pull-secret-syncer-f6wzq" Jul 09 16:15:25.025448 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:25.025284 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 16:15:25.025448 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:25.025356 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 16:15:25.025448 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:25.025364 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 16:15:25.025448 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:25.025386 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 16:15:25.025448 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:25.025360 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f45ad4f-0e3f-432e-8328-715ec5f97a69-metrics-certs podName:6f45ad4f-0e3f-432e-8328-715ec5f97a69 nodeName:}" failed. No retries permitted until 2026-07-09 16:15:33.025340408 +0000 UTC m=+18.218446771 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6f45ad4f-0e3f-432e-8328-715ec5f97a69-metrics-certs") pod "network-metrics-daemon-b7sxk" (UID: "6f45ad4f-0e3f-432e-8328-715ec5f97a69") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 16:15:25.025448 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:25.025400 2569 projected.go:196] Error preparing data for projected volume kube-api-access-r99g2 for pod openshift-network-diagnostics/network-check-target-kn7sp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 16:15:25.025448 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:25.025418 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3bd017f3-bffb-4016-bdac-48c6fe9fa352-original-pull-secret podName:3bd017f3-bffb-4016-bdac-48c6fe9fa352 nodeName:}" failed. No retries permitted until 2026-07-09 16:15:33.02540109 +0000 UTC m=+18.218507465 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3bd017f3-bffb-4016-bdac-48c6fe9fa352-original-pull-secret") pod "global-pull-secret-syncer-f6wzq" (UID: "3bd017f3-bffb-4016-bdac-48c6fe9fa352") : object "kube-system"/"original-pull-secret" not registered Jul 09 16:15:25.025448 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:25.025446 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/473a10a3-ceca-4c00-915e-a4a21820ce3e-kube-api-access-r99g2 podName:473a10a3-ceca-4c00-915e-a4a21820ce3e nodeName:}" failed. No retries permitted until 2026-07-09 16:15:33.025432683 +0000 UTC m=+18.218539058 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-r99g2" (UniqueName: "kubernetes.io/projected/473a10a3-ceca-4c00-915e-a4a21820ce3e-kube-api-access-r99g2") pod "network-check-target-kn7sp" (UID: "473a10a3-ceca-4c00-915e-a4a21820ce3e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 16:15:25.386609 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:25.386512 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kn7sp" Jul 09 16:15:25.386793 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:25.386637 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kn7sp" podUID="473a10a3-ceca-4c00-915e-a4a21820ce3e" Jul 09 16:15:25.387144 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:25.387009 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6wzq" Jul 09 16:15:25.387144 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:25.387121 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6wzq" podUID="3bd017f3-bffb-4016-bdac-48c6fe9fa352" Jul 09 16:15:26.383691 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:26.383659 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b7sxk" Jul 09 16:15:26.384072 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:26.383785 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b7sxk" podUID="6f45ad4f-0e3f-432e-8328-715ec5f97a69" Jul 09 16:15:27.389608 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:27.388832 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kn7sp" Jul 09 16:15:27.389608 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:27.389025 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kn7sp" podUID="473a10a3-ceca-4c00-915e-a4a21820ce3e" Jul 09 16:15:27.389608 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:27.389101 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6wzq" Jul 09 16:15:27.389608 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:27.389213 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6wzq" podUID="3bd017f3-bffb-4016-bdac-48c6fe9fa352" Jul 09 16:15:27.442118 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:27.442034 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-114.ec2.internal" event={"ID":"ef080aa212d8ff70e6e7c4e03505fc8a","Type":"ContainerStarted","Data":"a5d2dd586a833474d1d0d5a2790ab38298ebc7f2f29d4864e92c936dec95ceee"} Jul 09 16:15:27.443625 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:27.443584 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-q4flk" event={"ID":"98ab9c4f-0fd3-485f-8d0e-af1257f14d51","Type":"ContainerStarted","Data":"f6af0136da7e98d89b069dc72eef26270e0df500a2517dd497b81ef083fbc754"} Jul 09 16:15:27.479941 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:27.479163 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-114.ec2.internal" podStartSLOduration=11.479139666 podStartE2EDuration="11.479139666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 16:15:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:15:27.459494946 +0000 UTC m=+12.652601329" watchObservedRunningTime="2026-07-09 16:15:27.479139666 +0000 UTC m=+12.672246049" Jul 09 16:15:27.479941 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:27.479349 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-q4flk" podStartSLOduration=2.007050822 podStartE2EDuration="11.479339396s" totalImagesPullingTime="9.472288574s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 16:15:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:15:27.478929503 +0000 UTC m=+12.672035885" watchObservedRunningTime="2026-07-09 16:15:27.479339396 +0000 UTC m=+12.672445780" Jul 09 16:15:28.383804 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:28.383581 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b7sxk" Jul 09 16:15:28.383979 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:28.383954 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b7sxk" podUID="6f45ad4f-0e3f-432e-8328-715ec5f97a69" Jul 09 16:15:28.447827 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:28.447790 2569 generic.go:356] "Generic (PLEG): container finished" podID="68e180d1aa487674ae66c09f4bc0df95" containerID="bc0c8727a366e450850261783a1b2d22290f8e54148530d85e5e46b4701cd063" exitCode=0 Jul 09 16:15:28.448343 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:28.447898 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-114.ec2.internal" event={"ID":"68e180d1aa487674ae66c09f4bc0df95","Type":"ContainerDied","Data":"bc0c8727a366e450850261783a1b2d22290f8e54148530d85e5e46b4701cd063"} Jul 09 16:15:28.449656 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:28.449625 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ckqxw" event={"ID":"09e70b88-01bd-4114-93c0-fb224d649e1d","Type":"ContainerStarted","Data":"31f8badd3d4dab632c289a20d1db7f74e708876b9b266cfd5c9a6c65077d997d"} Jul 09 16:15:28.452816 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:28.452748 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xx9mg" event={"ID":"88be5834-3fd7-45da-83d5-4cb1d0f84c75","Type":"ContainerStarted","Data":"33fc0b17cbd0642713ede612ec6e3b97691cd7818df9b8e266589a829adab981"} Jul 09 16:15:28.454890 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:28.454852 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zj5qc" event={"ID":"e6d9b732-f9d0-48e1-b3a2-b9641eda1c0b","Type":"ContainerStarted","Data":"f5cea8badcad187b36ec35fe46e4e6540f27aa5e3751a44f4cce6cf61d58f12b"} Jul 09 16:15:28.456674 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:28.456647 2569 generic.go:356] "Generic (PLEG): container finished" podID="1a7bbb58-c6a9-4763-a12b-c2ae62a5beaf" containerID="1bec179cb79f6e1d761909f51076e674cdcf3a417cfa9690df5c27f990ddfc9c" exitCode=0 Jul 09 16:15:28.456778 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:28.456730 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6q72v" event={"ID":"1a7bbb58-c6a9-4763-a12b-c2ae62a5beaf","Type":"ContainerDied","Data":"1bec179cb79f6e1d761909f51076e674cdcf3a417cfa9690df5c27f990ddfc9c"} Jul 09 16:15:28.458403 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:28.458383 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jj6qs" event={"ID":"c17a000a-a79a-4dd8-af71-1fc47077bea7","Type":"ContainerStarted","Data":"ff35ab1558ec2b3a587af7ee719c417216b1975cf1e176792060d7f88fb2f4ca"} Jul 09 16:15:28.483047 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:28.482143 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-xx9mg" podStartSLOduration=4.758935276 podStartE2EDuration="13.482122008s" totalImagesPullingTime="8.723186732s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 16:15:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:15:28.481744573 +0000 UTC m=+13.674850956" watchObservedRunningTime="2026-07-09 16:15:28.482122008 +0000 UTC m=+13.675228390" Jul 09 16:15:28.527984 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:28.527927 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-zj5qc" podStartSLOduration=3.103904617 podStartE2EDuration="12.527909107s" totalImagesPullingTime="9.42400449s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:15:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:15:28.527455873 +0000 UTC m=+13.720562254" watchObservedRunningTime="2026-07-09 16:15:28.527909107 +0000 UTC m=+13.721015491" Jul 09 16:15:28.542029 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:28.541970 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-jj6qs" podStartSLOduration=3.930554844 podStartE2EDuration="13.541952026s" totalImagesPullingTime="9.611397182s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 16:15:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:15:28.541935177 +0000 UTC m=+13.735041559" watchObservedRunningTime="2026-07-09 16:15:28.541952026 +0000 UTC m=+13.735058404" Jul 09 16:15:29.160199 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:29.160165 2569 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 09 16:15:29.303011 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:29.302896 2569 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-09T16:15:29.160191118Z","UUID":"afe50ef8-c986-4580-9693-7f7e3f84b3a4","Handler":null,"Name":"","Endpoint":""} Jul 09 16:15:29.304262 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:29.304239 2569 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 09 16:15:29.304382 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:29.304280 2569 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 09 16:15:29.383940 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:29.383820 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kn7sp" Jul 09 16:15:29.384101 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:29.383969 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kn7sp" podUID="473a10a3-ceca-4c00-915e-a4a21820ce3e" Jul 09 16:15:29.384443 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:29.384418 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6wzq" Jul 09 16:15:29.384553 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:29.384531 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6wzq" podUID="3bd017f3-bffb-4016-bdac-48c6fe9fa352" Jul 09 16:15:29.462146 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:29.462113 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ckqxw" event={"ID":"09e70b88-01bd-4114-93c0-fb224d649e1d","Type":"ContainerStarted","Data":"78d1ffb6e4c51ddaf8294e33bae2b4a70a68ca6ce42097ca566b9f8c84eb3ad3"} Jul 09 16:15:29.463377 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:29.463352 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hgqgw" event={"ID":"29d74a8d-7acf-4663-ba5f-40e4884064bd","Type":"ContainerStarted","Data":"358609c15f322581b34b54ddea77dc41338848b7869f6234932158200220cbc4"} Jul 09 16:15:29.464972 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:29.464939 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-114.ec2.internal" event={"ID":"68e180d1aa487674ae66c09f4bc0df95","Type":"ContainerStarted","Data":"d9bc7825ccf41c36939e36ecdc9036eb7eb307d68deb400c6fe7850d601c404e"} Jul 09 16:15:29.479549 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:29.479494 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-hgqgw" podStartSLOduration=8.826743215 podStartE2EDuration="14.479478978s" totalImagesPullingTime="5.652735763s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 16:15:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:15:29.478952866 +0000 UTC m=+14.672059249" watchObservedRunningTime="2026-07-09 16:15:29.479478978 +0000 UTC m=+14.672585357" Jul 09 16:15:29.495846 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:29.495796 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-114.ec2.internal" podStartSLOduration=13.495778982 podStartE2EDuration="13.495778982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 16:15:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:15:29.495380097 +0000 UTC m=+14.688486477" watchObservedRunningTime="2026-07-09 16:15:29.495778982 +0000 UTC m=+14.688885360" Jul 09 16:15:30.383728 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:30.383689 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b7sxk" Jul 09 16:15:30.383927 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:30.383849 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b7sxk" podUID="6f45ad4f-0e3f-432e-8328-715ec5f97a69" Jul 09 16:15:31.388132 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:31.388103 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6wzq" Jul 09 16:15:31.388737 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:31.388267 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6wzq" podUID="3bd017f3-bffb-4016-bdac-48c6fe9fa352" Jul 09 16:15:31.388737 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:31.388103 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kn7sp" Jul 09 16:15:31.388737 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:31.388696 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kn7sp" podUID="473a10a3-ceca-4c00-915e-a4a21820ce3e" Jul 09 16:15:32.383672 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:32.383636 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b7sxk" Jul 09 16:15:32.383912 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:32.383777 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b7sxk" podUID="6f45ad4f-0e3f-432e-8328-715ec5f97a69" Jul 09 16:15:32.634561 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:32.633863 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-zj5qc" Jul 09 16:15:32.634561 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:32.634372 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-zj5qc" Jul 09 16:15:33.089282 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:33.089249 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3bd017f3-bffb-4016-bdac-48c6fe9fa352-original-pull-secret\") pod \"global-pull-secret-syncer-f6wzq\" (UID: \"3bd017f3-bffb-4016-bdac-48c6fe9fa352\") " pod="kube-system/global-pull-secret-syncer-f6wzq" Jul 09 16:15:33.089565 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:33.089306 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f45ad4f-0e3f-432e-8328-715ec5f97a69-metrics-certs\") pod \"network-metrics-daemon-b7sxk\" (UID: \"6f45ad4f-0e3f-432e-8328-715ec5f97a69\") " pod="openshift-multus/network-metrics-daemon-b7sxk" Jul 09 16:15:33.089565 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:33.089341 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r99g2\" (UniqueName: \"kubernetes.io/projected/473a10a3-ceca-4c00-915e-a4a21820ce3e-kube-api-access-r99g2\") pod \"network-check-target-kn7sp\" (UID: \"473a10a3-ceca-4c00-915e-a4a21820ce3e\") " pod="openshift-network-diagnostics/network-check-target-kn7sp" Jul 09 16:15:33.089565 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:33.089436 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 16:15:33.089565 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:33.089491 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 16:15:33.089565 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:33.089512 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 16:15:33.089565 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:33.089436 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 16:15:33.089565 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:33.089527 2569 projected.go:196] Error preparing data for projected volume kube-api-access-r99g2 for pod openshift-network-diagnostics/network-check-target-kn7sp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 16:15:33.089565 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:33.089521 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f45ad4f-0e3f-432e-8328-715ec5f97a69-metrics-certs podName:6f45ad4f-0e3f-432e-8328-715ec5f97a69 nodeName:}" failed. No retries permitted until 2026-07-09 16:15:49.089500568 +0000 UTC m=+34.282606949 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6f45ad4f-0e3f-432e-8328-715ec5f97a69-metrics-certs") pod "network-metrics-daemon-b7sxk" (UID: "6f45ad4f-0e3f-432e-8328-715ec5f97a69") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 16:15:33.090005 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:33.089594 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/473a10a3-ceca-4c00-915e-a4a21820ce3e-kube-api-access-r99g2 podName:473a10a3-ceca-4c00-915e-a4a21820ce3e nodeName:}" failed. No retries permitted until 2026-07-09 16:15:49.089573023 +0000 UTC m=+34.282679386 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-r99g2" (UniqueName: "kubernetes.io/projected/473a10a3-ceca-4c00-915e-a4a21820ce3e-kube-api-access-r99g2") pod "network-check-target-kn7sp" (UID: "473a10a3-ceca-4c00-915e-a4a21820ce3e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 16:15:33.090005 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:33.089610 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3bd017f3-bffb-4016-bdac-48c6fe9fa352-original-pull-secret podName:3bd017f3-bffb-4016-bdac-48c6fe9fa352 nodeName:}" failed. No retries permitted until 2026-07-09 16:15:49.089601705 +0000 UTC m=+34.282708064 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3bd017f3-bffb-4016-bdac-48c6fe9fa352-original-pull-secret") pod "global-pull-secret-syncer-f6wzq" (UID: "3bd017f3-bffb-4016-bdac-48c6fe9fa352") : object "kube-system"/"original-pull-secret" not registered Jul 09 16:15:33.147758 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:33.147723 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-zj5qc" Jul 09 16:15:33.148500 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:33.148476 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-zj5qc" Jul 09 16:15:33.383136 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:33.383044 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6wzq" Jul 09 16:15:33.383299 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:33.383172 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6wzq" podUID="3bd017f3-bffb-4016-bdac-48c6fe9fa352" Jul 09 16:15:33.383362 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:33.383049 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kn7sp" Jul 09 16:15:33.383501 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:33.383448 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kn7sp" podUID="473a10a3-ceca-4c00-915e-a4a21820ce3e" Jul 09 16:15:34.384067 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:34.384029 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b7sxk" Jul 09 16:15:34.384592 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:34.384167 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b7sxk" podUID="6f45ad4f-0e3f-432e-8328-715ec5f97a69" Jul 09 16:15:35.386467 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:35.386284 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6wzq" Jul 09 16:15:35.386911 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:35.386301 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kn7sp" Jul 09 16:15:35.386911 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:35.386565 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6wzq" podUID="3bd017f3-bffb-4016-bdac-48c6fe9fa352" Jul 09 16:15:35.386911 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:35.386643 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kn7sp" podUID="473a10a3-ceca-4c00-915e-a4a21820ce3e" Jul 09 16:15:36.384040 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:36.384003 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b7sxk" Jul 09 16:15:36.384238 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:36.384149 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b7sxk" podUID="6f45ad4f-0e3f-432e-8328-715ec5f97a69" Jul 09 16:15:37.384034 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:37.383995 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6wzq" Jul 09 16:15:37.384478 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:37.383996 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kn7sp" Jul 09 16:15:37.384478 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:37.384116 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6wzq" podUID="3bd017f3-bffb-4016-bdac-48c6fe9fa352" Jul 09 16:15:37.384478 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:37.384222 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kn7sp" podUID="473a10a3-ceca-4c00-915e-a4a21820ce3e" Jul 09 16:15:38.383617 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:38.383579 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b7sxk" Jul 09 16:15:38.383794 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:38.383697 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b7sxk" podUID="6f45ad4f-0e3f-432e-8328-715ec5f97a69" Jul 09 16:15:39.383830 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:39.383793 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kn7sp" Jul 09 16:15:39.384365 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:39.383801 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6wzq" Jul 09 16:15:39.384365 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:39.383939 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kn7sp" podUID="473a10a3-ceca-4c00-915e-a4a21820ce3e" Jul 09 16:15:39.384365 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:39.384008 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6wzq" podUID="3bd017f3-bffb-4016-bdac-48c6fe9fa352" Jul 09 16:15:40.383155 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:40.383124 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b7sxk" Jul 09 16:15:40.383309 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:40.383234 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b7sxk" podUID="6f45ad4f-0e3f-432e-8328-715ec5f97a69" Jul 09 16:15:41.383772 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:41.383575 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kn7sp" Jul 09 16:15:41.384363 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:41.383644 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6wzq" Jul 09 16:15:41.384363 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:41.383894 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kn7sp" podUID="473a10a3-ceca-4c00-915e-a4a21820ce3e" Jul 09 16:15:41.384363 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:41.383936 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6wzq" podUID="3bd017f3-bffb-4016-bdac-48c6fe9fa352" Jul 09 16:15:41.492791 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:41.492697 2569 generic.go:356] "Generic (PLEG): container finished" podID="1a7bbb58-c6a9-4763-a12b-c2ae62a5beaf" containerID="dab4a803c02063c8bd39b69b4ece7ecd5ca6785984a0d03c394f017c78a21744" exitCode=0 Jul 09 16:15:41.492791 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:41.492742 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6q72v" event={"ID":"1a7bbb58-c6a9-4763-a12b-c2ae62a5beaf","Type":"ContainerDied","Data":"dab4a803c02063c8bd39b69b4ece7ecd5ca6785984a0d03c394f017c78a21744"} Jul 09 16:15:41.494379 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:41.494353 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xrbgl" event={"ID":"bed43223-f252-48e2-b014-b917007e39dd","Type":"ContainerStarted","Data":"9f2521c76df5b6017b66e3ed02ebbb84a86ee833127a13c5c2a6b674f4863940"} Jul 09 16:15:41.496543 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:41.496520 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ckqxw" event={"ID":"09e70b88-01bd-4114-93c0-fb224d649e1d","Type":"ContainerStarted","Data":"378a4c1ff74f87e89f7beff42b58168ff94529fb34b0bd0b71f84f59fa7f676c"} Jul 09 16:15:41.499377 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:41.499361 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-49sqd_17313664-7a0f-472b-a597-39f8c5412bc3/ovn-acl-logging/0.log" Jul 09 16:15:41.499710 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:41.499691 2569 generic.go:356] "Generic (PLEG): container finished" podID="17313664-7a0f-472b-a597-39f8c5412bc3" containerID="8fd098942ceb64ad16ba43daa456a1b9fed9491384cdeca7bcf1f1a6f7afb898" exitCode=1 Jul 09 16:15:41.499782 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:41.499721 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-49sqd" event={"ID":"17313664-7a0f-472b-a597-39f8c5412bc3","Type":"ContainerStarted","Data":"80252fcf04a92c83f82b13e87b0f15bf445fc59643d0fe91af20bb8e9911edaf"} Jul 09 16:15:41.499782 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:41.499769 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-49sqd" event={"ID":"17313664-7a0f-472b-a597-39f8c5412bc3","Type":"ContainerStarted","Data":"7305474fee692f514ff5481696b6dcfbdbbf026f7ec26b1cf32e9ad53eb1ef57"} Jul 09 16:15:41.499866 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:41.499798 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-49sqd" event={"ID":"17313664-7a0f-472b-a597-39f8c5412bc3","Type":"ContainerStarted","Data":"1115b59859589014ca1be6cef8f3d511a36e637611dd20f7e9ab2b2b679c871b"} Jul 09 16:15:41.499866 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:41.499825 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-49sqd" event={"ID":"17313664-7a0f-472b-a597-39f8c5412bc3","Type":"ContainerStarted","Data":"20aaa69b6c7eb468357e03f2ca8c68c5034f6cbd6a16c8a0e61e8f317d0cbb39"} Jul 09 16:15:41.499866 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:41.499853 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-49sqd" event={"ID":"17313664-7a0f-472b-a597-39f8c5412bc3","Type":"ContainerDied","Data":"8fd098942ceb64ad16ba43daa456a1b9fed9491384cdeca7bcf1f1a6f7afb898"} Jul 09 16:15:41.500002 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:41.499903 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-49sqd" event={"ID":"17313664-7a0f-472b-a597-39f8c5412bc3","Type":"ContainerStarted","Data":"524a31a05e988d25c73a680309262eb3a503f3e497b88476dab076c1a3187890"} Jul 09 16:15:41.534193 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:41.534147 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-xrbgl" podStartSLOduration=3.479276302 podStartE2EDuration="26.534131032s" totalImagesPullingTime="23.05485473s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 16:15:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:15:41.534020288 +0000 UTC m=+26.727126682" watchObservedRunningTime="2026-07-09 16:15:41.534131032 +0000 UTC m=+26.727237426" Jul 09 16:15:42.383147 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:42.383111 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b7sxk" Jul 09 16:15:42.383341 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:42.383223 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b7sxk" podUID="6f45ad4f-0e3f-432e-8328-715ec5f97a69" Jul 09 16:15:43.383999 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:43.383965 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kn7sp" Jul 09 16:15:43.384513 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:43.384072 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kn7sp" podUID="473a10a3-ceca-4c00-915e-a4a21820ce3e" Jul 09 16:15:43.384513 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:43.384129 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6wzq" Jul 09 16:15:43.384513 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:43.384249 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6wzq" podUID="3bd017f3-bffb-4016-bdac-48c6fe9fa352" Jul 09 16:15:43.506534 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:43.506497 2569 generic.go:356] "Generic (PLEG): container finished" podID="1a7bbb58-c6a9-4763-a12b-c2ae62a5beaf" containerID="416dede5d879ae1645a0adeba95b58c79438b1e6240b2b07157f8cbfdc634e17" exitCode=0 Jul 09 16:15:43.506718 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:43.506630 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6q72v" event={"ID":"1a7bbb58-c6a9-4763-a12b-c2ae62a5beaf","Type":"ContainerDied","Data":"416dede5d879ae1645a0adeba95b58c79438b1e6240b2b07157f8cbfdc634e17"} Jul 09 16:15:43.510125 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:43.510103 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-49sqd_17313664-7a0f-472b-a597-39f8c5412bc3/ovn-acl-logging/0.log" Jul 09 16:15:43.510561 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:43.510533 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-49sqd" event={"ID":"17313664-7a0f-472b-a597-39f8c5412bc3","Type":"ContainerStarted","Data":"d22926710d59e31ed84e7eef05f9d3b107ad7fc191094f957a69781e4c088b9f"} Jul 09 16:15:43.530612 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:43.530547 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ckqxw" podStartSLOduration=5.230400269 podStartE2EDuration="27.530530867s" totalImagesPullingTime="22.300130598s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 16:15:16 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:15:41.555474919 +0000 UTC m=+26.748581301" watchObservedRunningTime="2026-07-09 16:15:43.530530867 +0000 UTC m=+28.723637289" Jul 09 16:15:44.383582 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:44.383551 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b7sxk" Jul 09 16:15:44.383732 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:44.383665 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b7sxk" podUID="6f45ad4f-0e3f-432e-8328-715ec5f97a69" Jul 09 16:15:45.384151 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:45.384126 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kn7sp" Jul 09 16:15:45.384614 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:45.384206 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kn7sp" podUID="473a10a3-ceca-4c00-915e-a4a21820ce3e" Jul 09 16:15:45.384614 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:45.384212 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6wzq" Jul 09 16:15:45.384614 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:45.384265 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6wzq" podUID="3bd017f3-bffb-4016-bdac-48c6fe9fa352" Jul 09 16:15:45.520035 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:45.519748 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-49sqd_17313664-7a0f-472b-a597-39f8c5412bc3/ovn-acl-logging/0.log" Jul 09 16:15:45.520644 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:45.520435 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-49sqd" event={"ID":"17313664-7a0f-472b-a597-39f8c5412bc3","Type":"ContainerStarted","Data":"d5384419d23033c7d0158d6ca394ee333e4608d2c3fdfdcb4057fe2ae888ef02"} Jul 09 16:15:45.520896 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:45.520796 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-49sqd" Jul 09 16:15:45.520955 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:45.520923 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-49sqd" Jul 09 16:15:45.521107 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:45.520976 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-49sqd" Jul 09 16:15:45.521107 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:45.521048 2569 scope.go:117] "RemoveContainer" containerID="8fd098942ceb64ad16ba43daa456a1b9fed9491384cdeca7bcf1f1a6f7afb898" Jul 09 16:15:45.525471 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:45.524939 2569 generic.go:356] "Generic (PLEG): container finished" podID="1a7bbb58-c6a9-4763-a12b-c2ae62a5beaf" containerID="4942ee9de34a401ff457e074c4e6c875c0919b362c62c7a2caaa9198388dccc5" exitCode=0 Jul 09 16:15:45.525471 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:45.524995 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6q72v" event={"ID":"1a7bbb58-c6a9-4763-a12b-c2ae62a5beaf","Type":"ContainerDied","Data":"4942ee9de34a401ff457e074c4e6c875c0919b362c62c7a2caaa9198388dccc5"} Jul 09 16:15:45.540343 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:45.540204 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-49sqd" Jul 09 16:15:45.541015 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:45.540994 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-49sqd" Jul 09 16:15:46.383683 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:46.383650 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b7sxk" Jul 09 16:15:46.383863 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:46.383761 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b7sxk" podUID="6f45ad4f-0e3f-432e-8328-715ec5f97a69" Jul 09 16:15:46.530342 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:46.530313 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-49sqd_17313664-7a0f-472b-a597-39f8c5412bc3/ovn-acl-logging/0.log" Jul 09 16:15:46.530765 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:46.530666 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-49sqd" event={"ID":"17313664-7a0f-472b-a597-39f8c5412bc3","Type":"ContainerStarted","Data":"3541b917e76aa3c3341fab6840e05c5d0136120dbd0c2429f97dd22695a51c04"} Jul 09 16:15:46.576227 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:46.576167 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-49sqd" podStartSLOduration=7.47814586 podStartE2EDuration="30.576144899s" totalImagesPullingTime="23.097999039s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 16:15:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:15:46.5757567 +0000 UTC m=+31.768863083" watchObservedRunningTime="2026-07-09 16:15:46.576144899 +0000 UTC m=+31.769251282" Jul 09 16:15:47.384025 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:47.383980 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kn7sp" Jul 09 16:15:47.384025 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:47.384021 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6wzq" Jul 09 16:15:47.384247 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:47.384122 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kn7sp" podUID="473a10a3-ceca-4c00-915e-a4a21820ce3e" Jul 09 16:15:47.384303 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:47.384246 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6wzq" podUID="3bd017f3-bffb-4016-bdac-48c6fe9fa352" Jul 09 16:15:47.465021 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:47.464973 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-b7sxk"] Jul 09 16:15:47.465171 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:47.465087 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b7sxk" Jul 09 16:15:47.465211 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:47.465176 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b7sxk" podUID="6f45ad4f-0e3f-432e-8328-715ec5f97a69" Jul 09 16:15:47.468813 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:47.468782 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kn7sp"] Jul 09 16:15:47.469469 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:47.469437 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-f6wzq"] Jul 09 16:15:47.533793 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:47.533767 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kn7sp" Jul 09 16:15:47.534326 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:47.533767 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6wzq" Jul 09 16:15:47.534326 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:47.533904 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kn7sp" podUID="473a10a3-ceca-4c00-915e-a4a21820ce3e" Jul 09 16:15:47.534326 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:47.534016 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6wzq" podUID="3bd017f3-bffb-4016-bdac-48c6fe9fa352" Jul 09 16:15:49.108339 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:49.108299 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r99g2\" (UniqueName: \"kubernetes.io/projected/473a10a3-ceca-4c00-915e-a4a21820ce3e-kube-api-access-r99g2\") pod \"network-check-target-kn7sp\" (UID: \"473a10a3-ceca-4c00-915e-a4a21820ce3e\") " pod="openshift-network-diagnostics/network-check-target-kn7sp" Jul 09 16:15:49.108945 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:49.108399 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3bd017f3-bffb-4016-bdac-48c6fe9fa352-original-pull-secret\") pod \"global-pull-secret-syncer-f6wzq\" (UID: \"3bd017f3-bffb-4016-bdac-48c6fe9fa352\") " pod="kube-system/global-pull-secret-syncer-f6wzq" Jul 09 16:15:49.108945 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:49.108435 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f45ad4f-0e3f-432e-8328-715ec5f97a69-metrics-certs\") pod \"network-metrics-daemon-b7sxk\" (UID: \"6f45ad4f-0e3f-432e-8328-715ec5f97a69\") " pod="openshift-multus/network-metrics-daemon-b7sxk" Jul 09 16:15:49.108945 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:49.108498 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 16:15:49.108945 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:49.108524 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 16:15:49.108945 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:49.108534 2569 projected.go:196] Error preparing data for projected volume kube-api-access-r99g2 for pod openshift-network-diagnostics/network-check-target-kn7sp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 16:15:49.108945 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:49.108559 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 16:15:49.108945 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:49.108583 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/473a10a3-ceca-4c00-915e-a4a21820ce3e-kube-api-access-r99g2 podName:473a10a3-ceca-4c00-915e-a4a21820ce3e nodeName:}" failed. No retries permitted until 2026-07-09 16:16:21.108568279 +0000 UTC m=+66.301674667 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-r99g2" (UniqueName: "kubernetes.io/projected/473a10a3-ceca-4c00-915e-a4a21820ce3e-kube-api-access-r99g2") pod "network-check-target-kn7sp" (UID: "473a10a3-ceca-4c00-915e-a4a21820ce3e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 16:15:49.108945 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:49.108608 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f45ad4f-0e3f-432e-8328-715ec5f97a69-metrics-certs podName:6f45ad4f-0e3f-432e-8328-715ec5f97a69 nodeName:}" failed. No retries permitted until 2026-07-09 16:16:21.108593903 +0000 UTC m=+66.301700268 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6f45ad4f-0e3f-432e-8328-715ec5f97a69-metrics-certs") pod "network-metrics-daemon-b7sxk" (UID: "6f45ad4f-0e3f-432e-8328-715ec5f97a69") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 16:15:49.108945 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:49.108560 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 16:15:49.108945 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:49.108642 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3bd017f3-bffb-4016-bdac-48c6fe9fa352-original-pull-secret podName:3bd017f3-bffb-4016-bdac-48c6fe9fa352 nodeName:}" failed. No retries permitted until 2026-07-09 16:16:21.108633335 +0000 UTC m=+66.301739698 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3bd017f3-bffb-4016-bdac-48c6fe9fa352-original-pull-secret") pod "global-pull-secret-syncer-f6wzq" (UID: "3bd017f3-bffb-4016-bdac-48c6fe9fa352") : object "kube-system"/"original-pull-secret" not registered Jul 09 16:15:49.383580 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:49.383497 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kn7sp" Jul 09 16:15:49.383747 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:49.383497 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b7sxk" Jul 09 16:15:49.383747 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:49.383626 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kn7sp" podUID="473a10a3-ceca-4c00-915e-a4a21820ce3e" Jul 09 16:15:49.383747 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:49.383661 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6wzq" Jul 09 16:15:49.383926 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:49.383785 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b7sxk" podUID="6f45ad4f-0e3f-432e-8328-715ec5f97a69" Jul 09 16:15:49.383926 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:49.383862 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6wzq" podUID="3bd017f3-bffb-4016-bdac-48c6fe9fa352" Jul 09 16:15:51.384125 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:51.383965 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kn7sp" Jul 09 16:15:51.384487 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:51.383965 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b7sxk" Jul 09 16:15:51.384487 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:51.384236 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kn7sp" podUID="473a10a3-ceca-4c00-915e-a4a21820ce3e" Jul 09 16:15:51.384487 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:51.384009 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6wzq" Jul 09 16:15:51.384487 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:51.384317 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b7sxk" podUID="6f45ad4f-0e3f-432e-8328-715ec5f97a69" Jul 09 16:15:51.384487 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:51.384401 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6wzq" podUID="3bd017f3-bffb-4016-bdac-48c6fe9fa352" Jul 09 16:15:51.546906 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:51.546811 2569 generic.go:356] "Generic (PLEG): container finished" podID="1a7bbb58-c6a9-4763-a12b-c2ae62a5beaf" containerID="b7af8721a7b6538634136c13feffda2d4da2c6db12b8217bc020f995941e47f9" exitCode=0 Jul 09 16:15:51.546906 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:51.546863 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6q72v" event={"ID":"1a7bbb58-c6a9-4763-a12b-c2ae62a5beaf","Type":"ContainerDied","Data":"b7af8721a7b6538634136c13feffda2d4da2c6db12b8217bc020f995941e47f9"} Jul 09 16:15:52.552014 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:52.551934 2569 generic.go:356] "Generic (PLEG): container finished" podID="1a7bbb58-c6a9-4763-a12b-c2ae62a5beaf" containerID="75eb8a79015d53c7b55f8bb079d8ebfaf566fad9b288bdc2fccaab5e909cbed9" exitCode=0 Jul 09 16:15:52.552014 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:52.551996 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6q72v" event={"ID":"1a7bbb58-c6a9-4763-a12b-c2ae62a5beaf","Type":"ContainerDied","Data":"75eb8a79015d53c7b55f8bb079d8ebfaf566fad9b288bdc2fccaab5e909cbed9"} Jul 09 16:15:53.383655 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:53.383618 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6wzq" Jul 09 16:15:53.383655 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:53.383635 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kn7sp" Jul 09 16:15:53.383954 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:53.383635 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b7sxk" Jul 09 16:15:53.383954 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:53.383746 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6wzq" podUID="3bd017f3-bffb-4016-bdac-48c6fe9fa352" Jul 09 16:15:53.383954 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:53.383798 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kn7sp" podUID="473a10a3-ceca-4c00-915e-a4a21820ce3e" Jul 09 16:15:53.383954 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:53.383899 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b7sxk" podUID="6f45ad4f-0e3f-432e-8328-715ec5f97a69" Jul 09 16:15:53.557890 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:53.557852 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6q72v" event={"ID":"1a7bbb58-c6a9-4763-a12b-c2ae62a5beaf","Type":"ContainerStarted","Data":"65ad3359dccdc036f30c3b0ea8f411a1264c553c75705be34a9679909be8ac78"} Jul 09 16:15:53.582951 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:53.582867 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-6q72v" podStartSLOduration=8.516265238999999 podStartE2EDuration="38.582847218s" totalImagesPullingTime="30.066581979s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 16:15:15 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:15:53.581941372 +0000 UTC m=+38.775047752" watchObservedRunningTime="2026-07-09 16:15:53.582847218 +0000 UTC m=+38.775953598" Jul 09 16:15:54.622761 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.622731 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-114.ec2.internal" event="NodeReady" Jul 09 16:15:54.623185 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.622857 2569 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 09 16:15:54.665703 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.665669 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-558d6fd964-d9xbs"] Jul 09 16:15:54.690189 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.690161 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-vdbkl"] Jul 09 16:15:54.690372 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.690247 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-558d6fd964-d9xbs" Jul 09 16:15:54.694531 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.694504 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 09 16:15:54.694662 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.694582 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 09 16:15:54.694662 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.694586 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 09 16:15:54.694662 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.694642 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6v4p5\"" Jul 09 16:15:54.706165 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.706141 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-84d7f6c55d-59vjr"] Jul 09 16:15:54.706313 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.706266 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-vdbkl" Jul 09 16:15:54.714781 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.714751 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 09 16:15:54.715739 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.715716 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 09 16:15:54.716040 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.716022 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 09 16:15:54.718167 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.716569 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 09 16:15:54.718167 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.716656 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-pf9kt\"" Jul 09 16:15:54.719963 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.719551 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 09 16:15:54.722287 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.722244 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-pf6gv"] Jul 09 16:15:54.723344 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.723095 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-84d7f6c55d-59vjr" Jul 09 16:15:54.726195 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.726174 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 09 16:15:54.726309 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.726272 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 09 16:15:54.726428 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.726406 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-p7mfm\"" Jul 09 16:15:54.726554 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.726406 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 09 16:15:54.726743 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.726726 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 09 16:15:54.726837 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.726771 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 09 16:15:54.727077 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.727061 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 09 16:15:54.738350 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.738324 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-q8zll"] Jul 09 16:15:54.738694 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.738491 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-pf6gv" Jul 09 16:15:54.740992 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.740970 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 09 16:15:54.741105 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.741029 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-x7rzj\"" Jul 09 16:15:54.741105 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.741033 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 09 16:15:54.741105 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.741063 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 09 16:15:54.755499 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.755478 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-vzds7"] Jul 09 16:15:54.755622 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.755608 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-q8zll" Jul 09 16:15:54.759039 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.759013 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 09 16:15:54.759039 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.759016 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 09 16:15:54.759039 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.759049 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6w5fn\"" Jul 09 16:15:54.769542 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.769521 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-6jksf"] Jul 09 16:15:54.769646 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.769633 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-vzds7" Jul 09 16:15:54.773307 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.773280 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 09 16:15:54.773307 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.773292 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-fph4t\"" Jul 09 16:15:54.773594 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.773579 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 09 16:15:54.781624 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.781602 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qfsjp"] Jul 09 16:15:54.781744 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.781714 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-6jksf" Jul 09 16:15:54.791358 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.791338 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 09 16:15:54.791822 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.791801 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-xjcmf"] Jul 09 16:15:54.791940 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.791926 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qfsjp" Jul 09 16:15:54.794194 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.794176 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 09 16:15:54.798452 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.798433 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 09 16:15:54.798663 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.798648 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 09 16:15:54.798741 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.798685 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-b8vcn\"" Jul 09 16:15:54.799886 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.799862 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 09 16:15:54.800050 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.800037 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 09 16:15:54.800127 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.800113 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 09 16:15:54.800888 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.800857 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 09 16:15:54.801121 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.801108 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-gwzfb\"" Jul 09 16:15:54.805585 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.805570 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 09 16:15:54.805955 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.805935 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7866497494-nzrk9"] Jul 09 16:15:54.806052 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.806043 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-xjcmf" Jul 09 16:15:54.813440 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.813408 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 09 16:15:54.813554 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.813412 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 09 16:15:54.813701 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.813687 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-j8jl2\"" Jul 09 16:15:54.813900 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.813887 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 09 16:15:54.814199 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.814186 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 09 16:15:54.815991 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.815975 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-558d6fd964-d9xbs"] Jul 09 16:15:54.816040 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.816004 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-vdbkl"] Jul 09 16:15:54.816040 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.816020 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-84d7f6c55d-59vjr"] Jul 09 16:15:54.816040 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.816037 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-pf6gv"] Jul 09 16:15:54.816125 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.816056 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-pzpbf"] Jul 09 16:15:54.816125 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.816082 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-nzrk9" Jul 09 16:15:54.830559 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.830533 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 09 16:15:54.830559 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.830533 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 09 16:15:54.830853 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.830839 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 09 16:15:54.831158 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.831142 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 09 16:15:54.831269 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.831255 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-krxlc\"" Jul 09 16:15:54.832229 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.832213 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-4rch5"] Jul 09 16:15:54.832335 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.832321 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-pzpbf" Jul 09 16:15:54.836504 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.836484 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 09 16:15:54.841300 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.841275 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 09 16:15:54.841300 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.841280 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 09 16:15:54.841467 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.841357 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-sxrhb\"" Jul 09 16:15:54.842171 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.842157 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qfsjp"] Jul 09 16:15:54.842232 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.842185 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-vzds7"] Jul 09 16:15:54.842232 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.842200 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-6jksf"] Jul 09 16:15:54.842232 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.842213 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-nzrk9"] Jul 09 16:15:54.842232 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.842224 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-q8zll"] Jul 09 16:15:54.842357 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.842245 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4rch5"] Jul 09 16:15:54.842357 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.842256 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-xjcmf"] Jul 09 16:15:54.842357 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.842268 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-pzpbf"] Jul 09 16:15:54.842357 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.842279 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4rch5" Jul 09 16:15:54.845906 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.845889 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 09 16:15:54.845966 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.845940 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 09 16:15:54.846027 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.845989 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 09 16:15:54.846280 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.846264 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-6z7vc\"" Jul 09 16:15:54.849849 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.849827 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e1cf555d-4290-4caa-a5f6-eaca63d05a1d-metrics-tls\") pod \"dns-default-q8zll\" (UID: \"e1cf555d-4290-4caa-a5f6-eaca63d05a1d\") " pod="openshift-dns/dns-default-q8zll" Jul 09 16:15:54.849976 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.849859 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nj4t\" (UniqueName: \"kubernetes.io/projected/e1cf555d-4290-4caa-a5f6-eaca63d05a1d-kube-api-access-9nj4t\") pod \"dns-default-q8zll\" (UID: \"e1cf555d-4290-4caa-a5f6-eaca63d05a1d\") " pod="openshift-dns/dns-default-q8zll" Jul 09 16:15:54.849976 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.849920 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1032414d-f6cd-4936-a52f-4f57feddba8f-bound-sa-token\") pod \"image-registry-558d6fd964-d9xbs\" (UID: \"1032414d-f6cd-4936-a52f-4f57feddba8f\") " pod="openshift-image-registry/image-registry-558d6fd964-d9xbs" Jul 09 16:15:54.849976 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.849947 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe900560-5002-4708-98e4-237e41014fbf-service-ca-bundle\") pod \"router-default-84d7f6c55d-59vjr\" (UID: \"fe900560-5002-4708-98e4-237e41014fbf\") " pod="openshift-ingress/router-default-84d7f6c55d-59vjr" Jul 09 16:15:54.849976 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.849968 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wm6k5\" (UniqueName: \"kubernetes.io/projected/022557ee-2bed-48ef-a717-3bf8a3c53dc5-kube-api-access-wm6k5\") pod \"cluster-monitoring-operator-5f7f494dd4-vdbkl\" (UID: \"022557ee-2bed-48ef-a717-3bf8a3c53dc5\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-vdbkl" Jul 09 16:15:54.850102 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.849991 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58bvj\" (UniqueName: \"kubernetes.io/projected/a0f6347a-b82d-4c94-b411-1b621bd1101c-kube-api-access-58bvj\") pod \"cluster-samples-operator-65db858894-pf6gv\" (UID: \"a0f6347a-b82d-4c94-b411-1b621bd1101c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-pf6gv" Jul 09 16:15:54.850102 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.850026 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1032414d-f6cd-4936-a52f-4f57feddba8f-registry-tls\") pod \"image-registry-558d6fd964-d9xbs\" (UID: \"1032414d-f6cd-4936-a52f-4f57feddba8f\") " pod="openshift-image-registry/image-registry-558d6fd964-d9xbs" Jul 09 16:15:54.850102 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.850051 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/fe900560-5002-4708-98e4-237e41014fbf-stats-auth\") pod \"router-default-84d7f6c55d-59vjr\" (UID: \"fe900560-5002-4708-98e4-237e41014fbf\") " pod="openshift-ingress/router-default-84d7f6c55d-59vjr" Jul 09 16:15:54.850188 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.850110 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1032414d-f6cd-4936-a52f-4f57feddba8f-ca-trust-extracted\") pod \"image-registry-558d6fd964-d9xbs\" (UID: \"1032414d-f6cd-4936-a52f-4f57feddba8f\") " pod="openshift-image-registry/image-registry-558d6fd964-d9xbs" Jul 09 16:15:54.850188 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.850131 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1032414d-f6cd-4936-a52f-4f57feddba8f-installation-pull-secrets\") pod \"image-registry-558d6fd964-d9xbs\" (UID: \"1032414d-f6cd-4936-a52f-4f57feddba8f\") " pod="openshift-image-registry/image-registry-558d6fd964-d9xbs" Jul 09 16:15:54.850188 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.850148 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe900560-5002-4708-98e4-237e41014fbf-metrics-certs\") pod \"router-default-84d7f6c55d-59vjr\" (UID: \"fe900560-5002-4708-98e4-237e41014fbf\") " pod="openshift-ingress/router-default-84d7f6c55d-59vjr" Jul 09 16:15:54.850188 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.850165 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shb9r\" (UniqueName: \"kubernetes.io/projected/fe900560-5002-4708-98e4-237e41014fbf-kube-api-access-shb9r\") pod \"router-default-84d7f6c55d-59vjr\" (UID: \"fe900560-5002-4708-98e4-237e41014fbf\") " pod="openshift-ingress/router-default-84d7f6c55d-59vjr" Jul 09 16:15:54.850188 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.850180 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/022557ee-2bed-48ef-a717-3bf8a3c53dc5-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-vdbkl\" (UID: \"022557ee-2bed-48ef-a717-3bf8a3c53dc5\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-vdbkl" Jul 09 16:15:54.850357 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.850201 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/022557ee-2bed-48ef-a717-3bf8a3c53dc5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-vdbkl\" (UID: \"022557ee-2bed-48ef-a717-3bf8a3c53dc5\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-vdbkl" Jul 09 16:15:54.850357 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.850247 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1032414d-f6cd-4936-a52f-4f57feddba8f-registry-certificates\") pod \"image-registry-558d6fd964-d9xbs\" (UID: \"1032414d-f6cd-4936-a52f-4f57feddba8f\") " pod="openshift-image-registry/image-registry-558d6fd964-d9xbs" Jul 09 16:15:54.850357 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.850264 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rpw2\" (UniqueName: \"kubernetes.io/projected/1032414d-f6cd-4936-a52f-4f57feddba8f-kube-api-access-6rpw2\") pod \"image-registry-558d6fd964-d9xbs\" (UID: \"1032414d-f6cd-4936-a52f-4f57feddba8f\") " pod="openshift-image-registry/image-registry-558d6fd964-d9xbs" Jul 09 16:15:54.850357 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.850282 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e1cf555d-4290-4caa-a5f6-eaca63d05a1d-config-volume\") pod \"dns-default-q8zll\" (UID: \"e1cf555d-4290-4caa-a5f6-eaca63d05a1d\") " pod="openshift-dns/dns-default-q8zll" Jul 09 16:15:54.850357 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.850298 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1032414d-f6cd-4936-a52f-4f57feddba8f-image-registry-private-configuration\") pod \"image-registry-558d6fd964-d9xbs\" (UID: \"1032414d-f6cd-4936-a52f-4f57feddba8f\") " pod="openshift-image-registry/image-registry-558d6fd964-d9xbs" Jul 09 16:15:54.850357 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.850314 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1032414d-f6cd-4936-a52f-4f57feddba8f-trusted-ca\") pod \"image-registry-558d6fd964-d9xbs\" (UID: \"1032414d-f6cd-4936-a52f-4f57feddba8f\") " pod="openshift-image-registry/image-registry-558d6fd964-d9xbs" Jul 09 16:15:54.850357 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.850329 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/fe900560-5002-4708-98e4-237e41014fbf-default-certificate\") pod \"router-default-84d7f6c55d-59vjr\" (UID: \"fe900560-5002-4708-98e4-237e41014fbf\") " pod="openshift-ingress/router-default-84d7f6c55d-59vjr" Jul 09 16:15:54.850571 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.850359 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a0f6347a-b82d-4c94-b411-1b621bd1101c-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-pf6gv\" (UID: \"a0f6347a-b82d-4c94-b411-1b621bd1101c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-pf6gv" Jul 09 16:15:54.850571 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.850388 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1cf555d-4290-4caa-a5f6-eaca63d05a1d-tmp-dir\") pod \"dns-default-q8zll\" (UID: \"e1cf555d-4290-4caa-a5f6-eaca63d05a1d\") " pod="openshift-dns/dns-default-q8zll" Jul 09 16:15:54.951097 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.951012 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a0f6347a-b82d-4c94-b411-1b621bd1101c-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-pf6gv\" (UID: \"a0f6347a-b82d-4c94-b411-1b621bd1101c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-pf6gv" Jul 09 16:15:54.951097 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.951051 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prpjc\" (UniqueName: \"kubernetes.io/projected/3d316a09-f8b0-49cd-b8cb-ff0746dff65c-kube-api-access-prpjc\") pod \"network-check-source-869dcfcdcd-pzpbf\" (UID: \"3d316a09-f8b0-49cd-b8cb-ff0746dff65c\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-pzpbf" Jul 09 16:15:54.951097 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.951073 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bb610783-4a86-4291-a9f0-42deb83b322d-tmp\") pod \"insights-operator-68d9dd94cf-6jksf\" (UID: \"bb610783-4a86-4291-a9f0-42deb83b322d\") " pod="openshift-insights/insights-operator-68d9dd94cf-6jksf" Jul 09 16:15:54.951097 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.951095 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/bb610783-4a86-4291-a9f0-42deb83b322d-snapshots\") pod \"insights-operator-68d9dd94cf-6jksf\" (UID: \"bb610783-4a86-4291-a9f0-42deb83b322d\") " pod="openshift-insights/insights-operator-68d9dd94cf-6jksf" Jul 09 16:15:54.951334 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.951154 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1032414d-f6cd-4936-a52f-4f57feddba8f-bound-sa-token\") pod \"image-registry-558d6fd964-d9xbs\" (UID: \"1032414d-f6cd-4936-a52f-4f57feddba8f\") " pod="openshift-image-registry/image-registry-558d6fd964-d9xbs" Jul 09 16:15:54.951334 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:54.951164 2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 09 16:15:54.951334 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:54.951227 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0f6347a-b82d-4c94-b411-1b621bd1101c-samples-operator-tls podName:a0f6347a-b82d-4c94-b411-1b621bd1101c nodeName:}" failed. No retries permitted until 2026-07-09 16:15:55.451212464 +0000 UTC m=+40.644318826 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a0f6347a-b82d-4c94-b411-1b621bd1101c-samples-operator-tls") pod "cluster-samples-operator-65db858894-pf6gv" (UID: "a0f6347a-b82d-4c94-b411-1b621bd1101c") : secret "samples-operator-tls" not found Jul 09 16:15:54.951334 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.951258 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe900560-5002-4708-98e4-237e41014fbf-service-ca-bundle\") pod \"router-default-84d7f6c55d-59vjr\" (UID: \"fe900560-5002-4708-98e4-237e41014fbf\") " pod="openshift-ingress/router-default-84d7f6c55d-59vjr" Jul 09 16:15:54.951334 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.951277 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a284ea64-610c-40f7-837b-fa0975c0a290-serving-cert\") pod \"console-operator-7866497494-nzrk9\" (UID: \"a284ea64-610c-40f7-837b-fa0975c0a290\") " pod="openshift-console-operator/console-operator-7866497494-nzrk9" Jul 09 16:15:54.951334 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.951306 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1032414d-f6cd-4936-a52f-4f57feddba8f-registry-tls\") pod \"image-registry-558d6fd964-d9xbs\" (UID: \"1032414d-f6cd-4936-a52f-4f57feddba8f\") " pod="openshift-image-registry/image-registry-558d6fd964-d9xbs" Jul 09 16:15:54.951334 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.951331 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb610783-4a86-4291-a9f0-42deb83b322d-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-6jksf\" (UID: \"bb610783-4a86-4291-a9f0-42deb83b322d\") " pod="openshift-insights/insights-operator-68d9dd94cf-6jksf" Jul 09 16:15:54.951619 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.951353 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/79ec4c79-9eb6-427a-b2f3-e66a6cd57dcb-serving-cert\") pod \"service-ca-operator-7695fc6c8c-qfsjp\" (UID: \"79ec4c79-9eb6-427a-b2f3-e66a6cd57dcb\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qfsjp" Jul 09 16:15:54.951619 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.951396 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e1cf555d-4290-4caa-a5f6-eaca63d05a1d-metrics-tls\") pod \"dns-default-q8zll\" (UID: \"e1cf555d-4290-4caa-a5f6-eaca63d05a1d\") " pod="openshift-dns/dns-default-q8zll" Jul 09 16:15:54.951619 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:54.951407 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 16:15:54.951619 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:54.951424 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-558d6fd964-d9xbs: secret "image-registry-tls" not found Jul 09 16:15:54.951619 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:54.951458 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fe900560-5002-4708-98e4-237e41014fbf-service-ca-bundle podName:fe900560-5002-4708-98e4-237e41014fbf nodeName:}" failed. No retries permitted until 2026-07-09 16:15:55.451444834 +0000 UTC m=+40.644551196 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/fe900560-5002-4708-98e4-237e41014fbf-service-ca-bundle") pod "router-default-84d7f6c55d-59vjr" (UID: "fe900560-5002-4708-98e4-237e41014fbf") : configmap references non-existent config key: service-ca.crt Jul 09 16:15:54.951619 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:54.951484 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1032414d-f6cd-4936-a52f-4f57feddba8f-registry-tls podName:1032414d-f6cd-4936-a52f-4f57feddba8f nodeName:}" failed. No retries permitted until 2026-07-09 16:15:55.45146981 +0000 UTC m=+40.644576176 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1032414d-f6cd-4936-a52f-4f57feddba8f-registry-tls") pod "image-registry-558d6fd964-d9xbs" (UID: "1032414d-f6cd-4936-a52f-4f57feddba8f") : secret "image-registry-tls" not found Jul 09 16:15:54.951619 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:54.951528 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 16:15:54.951619 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:54.951572 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1cf555d-4290-4caa-a5f6-eaca63d05a1d-metrics-tls podName:e1cf555d-4290-4caa-a5f6-eaca63d05a1d nodeName:}" failed. No retries permitted until 2026-07-09 16:15:55.451562297 +0000 UTC m=+40.644668658 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e1cf555d-4290-4caa-a5f6-eaca63d05a1d-metrics-tls") pod "dns-default-q8zll" (UID: "e1cf555d-4290-4caa-a5f6-eaca63d05a1d") : secret "dns-default-metrics-tls" not found Jul 09 16:15:54.951619 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.951593 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe900560-5002-4708-98e4-237e41014fbf-metrics-certs\") pod \"router-default-84d7f6c55d-59vjr\" (UID: \"fe900560-5002-4708-98e4-237e41014fbf\") " pod="openshift-ingress/router-default-84d7f6c55d-59vjr" Jul 09 16:15:54.952076 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.951630 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/022557ee-2bed-48ef-a717-3bf8a3c53dc5-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-vdbkl\" (UID: \"022557ee-2bed-48ef-a717-3bf8a3c53dc5\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-vdbkl" Jul 09 16:15:54.952076 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:54.951666 2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 09 16:15:54.952076 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.951669 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xj7tc\" (UniqueName: \"kubernetes.io/projected/f54c8925-c09f-46d8-84fb-9f3af042dc41-kube-api-access-xj7tc\") pod \"kube-storage-version-migrator-operator-799cb7f958-xjcmf\" (UID: \"f54c8925-c09f-46d8-84fb-9f3af042dc41\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-xjcmf" Jul 09 16:15:54.952076 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:54.951692 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe900560-5002-4708-98e4-237e41014fbf-metrics-certs podName:fe900560-5002-4708-98e4-237e41014fbf nodeName:}" failed. No retries permitted until 2026-07-09 16:15:55.451684727 +0000 UTC m=+40.644791089 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fe900560-5002-4708-98e4-237e41014fbf-metrics-certs") pod "router-default-84d7f6c55d-59vjr" (UID: "fe900560-5002-4708-98e4-237e41014fbf") : secret "router-metrics-certs-default" not found Jul 09 16:15:54.952076 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.951707 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a284ea64-610c-40f7-837b-fa0975c0a290-trusted-ca\") pod \"console-operator-7866497494-nzrk9\" (UID: \"a284ea64-610c-40f7-837b-fa0975c0a290\") " pod="openshift-console-operator/console-operator-7866497494-nzrk9" Jul 09 16:15:54.952076 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.951733 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb610783-4a86-4291-a9f0-42deb83b322d-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-6jksf\" (UID: \"bb610783-4a86-4291-a9f0-42deb83b322d\") " pod="openshift-insights/insights-operator-68d9dd94cf-6jksf" Jul 09 16:15:54.952076 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.951748 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bb610783-4a86-4291-a9f0-42deb83b322d-serving-cert\") pod \"insights-operator-68d9dd94cf-6jksf\" (UID: \"bb610783-4a86-4291-a9f0-42deb83b322d\") " pod="openshift-insights/insights-operator-68d9dd94cf-6jksf" Jul 09 16:15:54.952076 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.951768 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shb9r\" (UniqueName: \"kubernetes.io/projected/fe900560-5002-4708-98e4-237e41014fbf-kube-api-access-shb9r\") pod \"router-default-84d7f6c55d-59vjr\" (UID: \"fe900560-5002-4708-98e4-237e41014fbf\") " pod="openshift-ingress/router-default-84d7f6c55d-59vjr" Jul 09 16:15:54.952076 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.951809 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1032414d-f6cd-4936-a52f-4f57feddba8f-installation-pull-secrets\") pod \"image-registry-558d6fd964-d9xbs\" (UID: \"1032414d-f6cd-4936-a52f-4f57feddba8f\") " pod="openshift-image-registry/image-registry-558d6fd964-d9xbs" Jul 09 16:15:54.952076 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.951911 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rpw2\" (UniqueName: \"kubernetes.io/projected/1032414d-f6cd-4936-a52f-4f57feddba8f-kube-api-access-6rpw2\") pod \"image-registry-558d6fd964-d9xbs\" (UID: \"1032414d-f6cd-4936-a52f-4f57feddba8f\") " pod="openshift-image-registry/image-registry-558d6fd964-d9xbs" Jul 09 16:15:54.952076 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.951928 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/fe900560-5002-4708-98e4-237e41014fbf-default-certificate\") pod \"router-default-84d7f6c55d-59vjr\" (UID: \"fe900560-5002-4708-98e4-237e41014fbf\") " pod="openshift-ingress/router-default-84d7f6c55d-59vjr" Jul 09 16:15:54.952076 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.951950 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1cf555d-4290-4caa-a5f6-eaca63d05a1d-tmp-dir\") pod \"dns-default-q8zll\" (UID: \"e1cf555d-4290-4caa-a5f6-eaca63d05a1d\") " pod="openshift-dns/dns-default-q8zll" Jul 09 16:15:54.952076 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.951972 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f54c8925-c09f-46d8-84fb-9f3af042dc41-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-xjcmf\" (UID: \"f54c8925-c09f-46d8-84fb-9f3af042dc41\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-xjcmf" Jul 09 16:15:54.952076 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.951994 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1032414d-f6cd-4936-a52f-4f57feddba8f-registry-certificates\") pod \"image-registry-558d6fd964-d9xbs\" (UID: \"1032414d-f6cd-4936-a52f-4f57feddba8f\") " pod="openshift-image-registry/image-registry-558d6fd964-d9xbs" Jul 09 16:15:54.952076 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.952011 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e1cf555d-4290-4caa-a5f6-eaca63d05a1d-config-volume\") pod \"dns-default-q8zll\" (UID: \"e1cf555d-4290-4caa-a5f6-eaca63d05a1d\") " pod="openshift-dns/dns-default-q8zll" Jul 09 16:15:54.952809 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.952033 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cc6ebce3-3ee7-4b8d-a11d-1cb084a6b780-cert\") pod \"ingress-canary-4rch5\" (UID: \"cc6ebce3-3ee7-4b8d-a11d-1cb084a6b780\") " pod="openshift-ingress-canary/ingress-canary-4rch5" Jul 09 16:15:54.952809 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.952061 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1032414d-f6cd-4936-a52f-4f57feddba8f-image-registry-private-configuration\") pod \"image-registry-558d6fd964-d9xbs\" (UID: \"1032414d-f6cd-4936-a52f-4f57feddba8f\") " pod="openshift-image-registry/image-registry-558d6fd964-d9xbs" Jul 09 16:15:54.952809 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.952085 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1032414d-f6cd-4936-a52f-4f57feddba8f-trusted-ca\") pod \"image-registry-558d6fd964-d9xbs\" (UID: \"1032414d-f6cd-4936-a52f-4f57feddba8f\") " pod="openshift-image-registry/image-registry-558d6fd964-d9xbs" Jul 09 16:15:54.952809 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.952111 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a284ea64-610c-40f7-837b-fa0975c0a290-config\") pod \"console-operator-7866497494-nzrk9\" (UID: \"a284ea64-610c-40f7-837b-fa0975c0a290\") " pod="openshift-console-operator/console-operator-7866497494-nzrk9" Jul 09 16:15:54.952809 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.952135 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mgkw\" (UniqueName: \"kubernetes.io/projected/cc6ebce3-3ee7-4b8d-a11d-1cb084a6b780-kube-api-access-7mgkw\") pod \"ingress-canary-4rch5\" (UID: \"cc6ebce3-3ee7-4b8d-a11d-1cb084a6b780\") " pod="openshift-ingress-canary/ingress-canary-4rch5" Jul 09 16:15:54.952809 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.952159 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/79ec4c79-9eb6-427a-b2f3-e66a6cd57dcb-config\") pod \"service-ca-operator-7695fc6c8c-qfsjp\" (UID: \"79ec4c79-9eb6-427a-b2f3-e66a6cd57dcb\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qfsjp" Jul 09 16:15:54.952809 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.952189 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58bvj\" (UniqueName: \"kubernetes.io/projected/a0f6347a-b82d-4c94-b411-1b621bd1101c-kube-api-access-58bvj\") pod \"cluster-samples-operator-65db858894-pf6gv\" (UID: \"a0f6347a-b82d-4c94-b411-1b621bd1101c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-pf6gv" Jul 09 16:15:54.952809 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.952215 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm6k5\" (UniqueName: \"kubernetes.io/projected/022557ee-2bed-48ef-a717-3bf8a3c53dc5-kube-api-access-wm6k5\") pod \"cluster-monitoring-operator-5f7f494dd4-vdbkl\" (UID: \"022557ee-2bed-48ef-a717-3bf8a3c53dc5\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-vdbkl" Jul 09 16:15:54.952809 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.952308 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/fe900560-5002-4708-98e4-237e41014fbf-stats-auth\") pod \"router-default-84d7f6c55d-59vjr\" (UID: \"fe900560-5002-4708-98e4-237e41014fbf\") " pod="openshift-ingress/router-default-84d7f6c55d-59vjr" Jul 09 16:15:54.952809 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.952556 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1cf555d-4290-4caa-a5f6-eaca63d05a1d-tmp-dir\") pod \"dns-default-q8zll\" (UID: \"e1cf555d-4290-4caa-a5f6-eaca63d05a1d\") " pod="openshift-dns/dns-default-q8zll" Jul 09 16:15:54.952809 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.952600 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e1cf555d-4290-4caa-a5f6-eaca63d05a1d-config-volume\") pod \"dns-default-q8zll\" (UID: \"e1cf555d-4290-4caa-a5f6-eaca63d05a1d\") " pod="openshift-dns/dns-default-q8zll" Jul 09 16:15:54.952809 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.952728 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nj4t\" (UniqueName: \"kubernetes.io/projected/e1cf555d-4290-4caa-a5f6-eaca63d05a1d-kube-api-access-9nj4t\") pod \"dns-default-q8zll\" (UID: \"e1cf555d-4290-4caa-a5f6-eaca63d05a1d\") " pod="openshift-dns/dns-default-q8zll" Jul 09 16:15:54.952809 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.952757 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f54c8925-c09f-46d8-84fb-9f3af042dc41-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-xjcmf\" (UID: \"f54c8925-c09f-46d8-84fb-9f3af042dc41\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-xjcmf" Jul 09 16:15:54.952809 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.952801 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/022557ee-2bed-48ef-a717-3bf8a3c53dc5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-vdbkl\" (UID: \"022557ee-2bed-48ef-a717-3bf8a3c53dc5\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-vdbkl" Jul 09 16:15:54.953479 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.952829 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlnq8\" (UniqueName: \"kubernetes.io/projected/a284ea64-610c-40f7-837b-fa0975c0a290-kube-api-access-tlnq8\") pod \"console-operator-7866497494-nzrk9\" (UID: \"a284ea64-610c-40f7-837b-fa0975c0a290\") " pod="openshift-console-operator/console-operator-7866497494-nzrk9" Jul 09 16:15:54.953479 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.952854 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1032414d-f6cd-4936-a52f-4f57feddba8f-ca-trust-extracted\") pod \"image-registry-558d6fd964-d9xbs\" (UID: \"1032414d-f6cd-4936-a52f-4f57feddba8f\") " pod="openshift-image-registry/image-registry-558d6fd964-d9xbs" Jul 09 16:15:54.953479 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.952902 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qqnc\" (UniqueName: \"kubernetes.io/projected/4b943693-db9f-4291-90a6-1cf6cc00a714-kube-api-access-7qqnc\") pod \"volume-data-source-validator-78495597f6-vzds7\" (UID: \"4b943693-db9f-4291-90a6-1cf6cc00a714\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-vzds7" Jul 09 16:15:54.953479 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.952925 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8wcc\" (UniqueName: \"kubernetes.io/projected/bb610783-4a86-4291-a9f0-42deb83b322d-kube-api-access-l8wcc\") pod \"insights-operator-68d9dd94cf-6jksf\" (UID: \"bb610783-4a86-4291-a9f0-42deb83b322d\") " pod="openshift-insights/insights-operator-68d9dd94cf-6jksf" Jul 09 16:15:54.953479 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.952946 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v65gf\" (UniqueName: \"kubernetes.io/projected/79ec4c79-9eb6-427a-b2f3-e66a6cd57dcb-kube-api-access-v65gf\") pod \"service-ca-operator-7695fc6c8c-qfsjp\" (UID: \"79ec4c79-9eb6-427a-b2f3-e66a6cd57dcb\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qfsjp" Jul 09 16:15:54.953479 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.953035 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1032414d-f6cd-4936-a52f-4f57feddba8f-registry-certificates\") pod \"image-registry-558d6fd964-d9xbs\" (UID: \"1032414d-f6cd-4936-a52f-4f57feddba8f\") " pod="openshift-image-registry/image-registry-558d6fd964-d9xbs" Jul 09 16:15:54.953479 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:54.953074 2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 09 16:15:54.953479 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:54.953125 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/022557ee-2bed-48ef-a717-3bf8a3c53dc5-cluster-monitoring-operator-tls podName:022557ee-2bed-48ef-a717-3bf8a3c53dc5 nodeName:}" failed. No retries permitted until 2026-07-09 16:15:55.45311105 +0000 UTC m=+40.646217410 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/022557ee-2bed-48ef-a717-3bf8a3c53dc5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-vdbkl" (UID: "022557ee-2bed-48ef-a717-3bf8a3c53dc5") : secret "cluster-monitoring-operator-tls" not found Jul 09 16:15:54.953479 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.953274 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1032414d-f6cd-4936-a52f-4f57feddba8f-trusted-ca\") pod \"image-registry-558d6fd964-d9xbs\" (UID: \"1032414d-f6cd-4936-a52f-4f57feddba8f\") " pod="openshift-image-registry/image-registry-558d6fd964-d9xbs" Jul 09 16:15:54.953479 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.953324 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1032414d-f6cd-4936-a52f-4f57feddba8f-ca-trust-extracted\") pod \"image-registry-558d6fd964-d9xbs\" (UID: \"1032414d-f6cd-4936-a52f-4f57feddba8f\") " pod="openshift-image-registry/image-registry-558d6fd964-d9xbs" Jul 09 16:15:54.956240 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.956217 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1032414d-f6cd-4936-a52f-4f57feddba8f-installation-pull-secrets\") pod \"image-registry-558d6fd964-d9xbs\" (UID: \"1032414d-f6cd-4936-a52f-4f57feddba8f\") " pod="openshift-image-registry/image-registry-558d6fd964-d9xbs" Jul 09 16:15:54.956356 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.956248 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1032414d-f6cd-4936-a52f-4f57feddba8f-image-registry-private-configuration\") pod \"image-registry-558d6fd964-d9xbs\" (UID: \"1032414d-f6cd-4936-a52f-4f57feddba8f\") " pod="openshift-image-registry/image-registry-558d6fd964-d9xbs" Jul 09 16:15:54.956356 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.956280 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/fe900560-5002-4708-98e4-237e41014fbf-default-certificate\") pod \"router-default-84d7f6c55d-59vjr\" (UID: \"fe900560-5002-4708-98e4-237e41014fbf\") " pod="openshift-ingress/router-default-84d7f6c55d-59vjr" Jul 09 16:15:54.956356 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.956304 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/fe900560-5002-4708-98e4-237e41014fbf-stats-auth\") pod \"router-default-84d7f6c55d-59vjr\" (UID: \"fe900560-5002-4708-98e4-237e41014fbf\") " pod="openshift-ingress/router-default-84d7f6c55d-59vjr" Jul 09 16:15:54.962730 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.962699 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1032414d-f6cd-4936-a52f-4f57feddba8f-bound-sa-token\") pod \"image-registry-558d6fd964-d9xbs\" (UID: \"1032414d-f6cd-4936-a52f-4f57feddba8f\") " pod="openshift-image-registry/image-registry-558d6fd964-d9xbs" Jul 09 16:15:54.963184 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.963162 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58bvj\" (UniqueName: \"kubernetes.io/projected/a0f6347a-b82d-4c94-b411-1b621bd1101c-kube-api-access-58bvj\") pod \"cluster-samples-operator-65db858894-pf6gv\" (UID: \"a0f6347a-b82d-4c94-b411-1b621bd1101c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-pf6gv" Jul 09 16:15:54.963309 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.963290 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shb9r\" (UniqueName: \"kubernetes.io/projected/fe900560-5002-4708-98e4-237e41014fbf-kube-api-access-shb9r\") pod \"router-default-84d7f6c55d-59vjr\" (UID: \"fe900560-5002-4708-98e4-237e41014fbf\") " pod="openshift-ingress/router-default-84d7f6c55d-59vjr" Jul 09 16:15:54.963381 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.963366 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rpw2\" (UniqueName: \"kubernetes.io/projected/1032414d-f6cd-4936-a52f-4f57feddba8f-kube-api-access-6rpw2\") pod \"image-registry-558d6fd964-d9xbs\" (UID: \"1032414d-f6cd-4936-a52f-4f57feddba8f\") " pod="openshift-image-registry/image-registry-558d6fd964-d9xbs" Jul 09 16:15:54.963557 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.963538 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nj4t\" (UniqueName: \"kubernetes.io/projected/e1cf555d-4290-4caa-a5f6-eaca63d05a1d-kube-api-access-9nj4t\") pod \"dns-default-q8zll\" (UID: \"e1cf555d-4290-4caa-a5f6-eaca63d05a1d\") " pod="openshift-dns/dns-default-q8zll" Jul 09 16:15:54.966628 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.966605 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/022557ee-2bed-48ef-a717-3bf8a3c53dc5-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-vdbkl\" (UID: \"022557ee-2bed-48ef-a717-3bf8a3c53dc5\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-vdbkl" Jul 09 16:15:54.968181 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:54.968161 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm6k5\" (UniqueName: \"kubernetes.io/projected/022557ee-2bed-48ef-a717-3bf8a3c53dc5-kube-api-access-wm6k5\") pod \"cluster-monitoring-operator-5f7f494dd4-vdbkl\" (UID: \"022557ee-2bed-48ef-a717-3bf8a3c53dc5\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-vdbkl" Jul 09 16:15:55.054064 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:55.054015 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f54c8925-c09f-46d8-84fb-9f3af042dc41-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-xjcmf\" (UID: \"f54c8925-c09f-46d8-84fb-9f3af042dc41\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-xjcmf" Jul 09 16:15:55.054064 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:55.054066 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cc6ebce3-3ee7-4b8d-a11d-1cb084a6b780-cert\") pod \"ingress-canary-4rch5\" (UID: \"cc6ebce3-3ee7-4b8d-a11d-1cb084a6b780\") " pod="openshift-ingress-canary/ingress-canary-4rch5" Jul 09 16:15:55.054333 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:55.054096 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a284ea64-610c-40f7-837b-fa0975c0a290-config\") pod \"console-operator-7866497494-nzrk9\" (UID: \"a284ea64-610c-40f7-837b-fa0975c0a290\") " pod="openshift-console-operator/console-operator-7866497494-nzrk9" Jul 09 16:15:55.054333 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:55.054118 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mgkw\" (UniqueName: \"kubernetes.io/projected/cc6ebce3-3ee7-4b8d-a11d-1cb084a6b780-kube-api-access-7mgkw\") pod \"ingress-canary-4rch5\" (UID: \"cc6ebce3-3ee7-4b8d-a11d-1cb084a6b780\") " pod="openshift-ingress-canary/ingress-canary-4rch5" Jul 09 16:15:55.054333 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:55.054139 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/79ec4c79-9eb6-427a-b2f3-e66a6cd57dcb-config\") pod \"service-ca-operator-7695fc6c8c-qfsjp\" (UID: \"79ec4c79-9eb6-427a-b2f3-e66a6cd57dcb\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qfsjp" Jul 09 16:15:55.054333 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:55.054177 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f54c8925-c09f-46d8-84fb-9f3af042dc41-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-xjcmf\" (UID: \"f54c8925-c09f-46d8-84fb-9f3af042dc41\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-xjcmf" Jul 09 16:15:55.054593 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:55.054420 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlnq8\" (UniqueName: \"kubernetes.io/projected/a284ea64-610c-40f7-837b-fa0975c0a290-kube-api-access-tlnq8\") pod \"console-operator-7866497494-nzrk9\" (UID: \"a284ea64-610c-40f7-837b-fa0975c0a290\") " pod="openshift-console-operator/console-operator-7866497494-nzrk9" Jul 09 16:15:55.054593 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:55.054472 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qqnc\" (UniqueName: \"kubernetes.io/projected/4b943693-db9f-4291-90a6-1cf6cc00a714-kube-api-access-7qqnc\") pod \"volume-data-source-validator-78495597f6-vzds7\" (UID: \"4b943693-db9f-4291-90a6-1cf6cc00a714\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-vzds7" Jul 09 16:15:55.054593 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:55.054502 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8wcc\" (UniqueName: \"kubernetes.io/projected/bb610783-4a86-4291-a9f0-42deb83b322d-kube-api-access-l8wcc\") pod \"insights-operator-68d9dd94cf-6jksf\" (UID: \"bb610783-4a86-4291-a9f0-42deb83b322d\") " pod="openshift-insights/insights-operator-68d9dd94cf-6jksf" Jul 09 16:15:55.054793 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:55.054764 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v65gf\" (UniqueName: \"kubernetes.io/projected/79ec4c79-9eb6-427a-b2f3-e66a6cd57dcb-kube-api-access-v65gf\") pod \"service-ca-operator-7695fc6c8c-qfsjp\" (UID: \"79ec4c79-9eb6-427a-b2f3-e66a6cd57dcb\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qfsjp" Jul 09 16:15:55.056067 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:55.054833 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prpjc\" (UniqueName: \"kubernetes.io/projected/3d316a09-f8b0-49cd-b8cb-ff0746dff65c-kube-api-access-prpjc\") pod \"network-check-source-869dcfcdcd-pzpbf\" (UID: \"3d316a09-f8b0-49cd-b8cb-ff0746dff65c\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-pzpbf" Jul 09 16:15:55.056067 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:55.054870 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bb610783-4a86-4291-a9f0-42deb83b322d-tmp\") pod \"insights-operator-68d9dd94cf-6jksf\" (UID: \"bb610783-4a86-4291-a9f0-42deb83b322d\") " pod="openshift-insights/insights-operator-68d9dd94cf-6jksf" Jul 09 16:15:55.056067 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:55.054830 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f54c8925-c09f-46d8-84fb-9f3af042dc41-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-xjcmf\" (UID: \"f54c8925-c09f-46d8-84fb-9f3af042dc41\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-xjcmf" Jul 09 16:15:55.056067 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:55.054921 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/bb610783-4a86-4291-a9f0-42deb83b322d-snapshots\") pod \"insights-operator-68d9dd94cf-6jksf\" (UID: \"bb610783-4a86-4291-a9f0-42deb83b322d\") " pod="openshift-insights/insights-operator-68d9dd94cf-6jksf" Jul 09 16:15:55.056067 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:55.054959 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/79ec4c79-9eb6-427a-b2f3-e66a6cd57dcb-config\") pod \"service-ca-operator-7695fc6c8c-qfsjp\" (UID: \"79ec4c79-9eb6-427a-b2f3-e66a6cd57dcb\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qfsjp" Jul 09 16:15:55.056067 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:55.054977 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a284ea64-610c-40f7-837b-fa0975c0a290-config\") pod \"console-operator-7866497494-nzrk9\" (UID: \"a284ea64-610c-40f7-837b-fa0975c0a290\") " pod="openshift-console-operator/console-operator-7866497494-nzrk9" Jul 09 16:15:55.056067 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:55.054983 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a284ea64-610c-40f7-837b-fa0975c0a290-serving-cert\") pod \"console-operator-7866497494-nzrk9\" (UID: \"a284ea64-610c-40f7-837b-fa0975c0a290\") " pod="openshift-console-operator/console-operator-7866497494-nzrk9" Jul 09 16:15:55.056067 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:55.055060 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb610783-4a86-4291-a9f0-42deb83b322d-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-6jksf\" (UID: \"bb610783-4a86-4291-a9f0-42deb83b322d\") " pod="openshift-insights/insights-operator-68d9dd94cf-6jksf" Jul 09 16:15:55.056067 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:55.055087 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/79ec4c79-9eb6-427a-b2f3-e66a6cd57dcb-serving-cert\") pod \"service-ca-operator-7695fc6c8c-qfsjp\" (UID: \"79ec4c79-9eb6-427a-b2f3-e66a6cd57dcb\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qfsjp" Jul 09 16:15:55.056067 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:55.055146 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj7tc\" (UniqueName: \"kubernetes.io/projected/f54c8925-c09f-46d8-84fb-9f3af042dc41-kube-api-access-xj7tc\") pod \"kube-storage-version-migrator-operator-799cb7f958-xjcmf\" (UID: \"f54c8925-c09f-46d8-84fb-9f3af042dc41\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-xjcmf" Jul 09 16:15:55.056067 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:55.055173 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a284ea64-610c-40f7-837b-fa0975c0a290-trusted-ca\") pod \"console-operator-7866497494-nzrk9\" (UID: \"a284ea64-610c-40f7-837b-fa0975c0a290\") " pod="openshift-console-operator/console-operator-7866497494-nzrk9" Jul 09 16:15:55.056067 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:55.055197 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb610783-4a86-4291-a9f0-42deb83b322d-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-6jksf\" (UID: \"bb610783-4a86-4291-a9f0-42deb83b322d\") " pod="openshift-insights/insights-operator-68d9dd94cf-6jksf" Jul 09 16:15:55.056067 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:55.055227 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bb610783-4a86-4291-a9f0-42deb83b322d-serving-cert\") pod \"insights-operator-68d9dd94cf-6jksf\" (UID: \"bb610783-4a86-4291-a9f0-42deb83b322d\") " pod="openshift-insights/insights-operator-68d9dd94cf-6jksf" Jul 09 16:15:55.056067 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:55.055225 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bb610783-4a86-4291-a9f0-42deb83b322d-tmp\") pod \"insights-operator-68d9dd94cf-6jksf\" (UID: \"bb610783-4a86-4291-a9f0-42deb83b322d\") " pod="openshift-insights/insights-operator-68d9dd94cf-6jksf" Jul 09 16:15:55.056067 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:55.055570 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/bb610783-4a86-4291-a9f0-42deb83b322d-snapshots\") pod \"insights-operator-68d9dd94cf-6jksf\" (UID: \"bb610783-4a86-4291-a9f0-42deb83b322d\") " pod="openshift-insights/insights-operator-68d9dd94cf-6jksf" Jul 09 16:15:55.056067 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:55.055967 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a284ea64-610c-40f7-837b-fa0975c0a290-trusted-ca\") pod \"console-operator-7866497494-nzrk9\" (UID: \"a284ea64-610c-40f7-837b-fa0975c0a290\") " pod="openshift-console-operator/console-operator-7866497494-nzrk9" Jul 09 16:15:55.056992 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:55.056066 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb610783-4a86-4291-a9f0-42deb83b322d-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-6jksf\" (UID: \"bb610783-4a86-4291-a9f0-42deb83b322d\") " pod="openshift-insights/insights-operator-68d9dd94cf-6jksf" Jul 09 16:15:55.056992 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:55.056299 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb610783-4a86-4291-a9f0-42deb83b322d-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-6jksf\" (UID: \"bb610783-4a86-4291-a9f0-42deb83b322d\") " pod="openshift-insights/insights-operator-68d9dd94cf-6jksf" Jul 09 16:15:55.056992 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:55.056831 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f54c8925-c09f-46d8-84fb-9f3af042dc41-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-xjcmf\" (UID: \"f54c8925-c09f-46d8-84fb-9f3af042dc41\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-xjcmf" Jul 09 16:15:55.057147 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:55.057103 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cc6ebce3-3ee7-4b8d-a11d-1cb084a6b780-cert\") pod \"ingress-canary-4rch5\" (UID: \"cc6ebce3-3ee7-4b8d-a11d-1cb084a6b780\") " pod="openshift-ingress-canary/ingress-canary-4rch5" Jul 09 16:15:55.057294 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:55.057275 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a284ea64-610c-40f7-837b-fa0975c0a290-serving-cert\") pod \"console-operator-7866497494-nzrk9\" (UID: \"a284ea64-610c-40f7-837b-fa0975c0a290\") " pod="openshift-console-operator/console-operator-7866497494-nzrk9" Jul 09 16:15:55.057594 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:55.057575 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/79ec4c79-9eb6-427a-b2f3-e66a6cd57dcb-serving-cert\") pod \"service-ca-operator-7695fc6c8c-qfsjp\" (UID: \"79ec4c79-9eb6-427a-b2f3-e66a6cd57dcb\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qfsjp" Jul 09 16:15:55.057985 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:55.057970 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bb610783-4a86-4291-a9f0-42deb83b322d-serving-cert\") pod \"insights-operator-68d9dd94cf-6jksf\" (UID: \"bb610783-4a86-4291-a9f0-42deb83b322d\") " pod="openshift-insights/insights-operator-68d9dd94cf-6jksf" Jul 09 16:15:55.063585 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:55.063480 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v65gf\" (UniqueName: \"kubernetes.io/projected/79ec4c79-9eb6-427a-b2f3-e66a6cd57dcb-kube-api-access-v65gf\") pod \"service-ca-operator-7695fc6c8c-qfsjp\" (UID: \"79ec4c79-9eb6-427a-b2f3-e66a6cd57dcb\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qfsjp" Jul 09 16:15:55.064098 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:55.063828 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qqnc\" (UniqueName: \"kubernetes.io/projected/4b943693-db9f-4291-90a6-1cf6cc00a714-kube-api-access-7qqnc\") pod \"volume-data-source-validator-78495597f6-vzds7\" (UID: \"4b943693-db9f-4291-90a6-1cf6cc00a714\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-vzds7" Jul 09 16:15:55.064098 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:55.063841 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mgkw\" (UniqueName: \"kubernetes.io/projected/cc6ebce3-3ee7-4b8d-a11d-1cb084a6b780-kube-api-access-7mgkw\") pod \"ingress-canary-4rch5\" (UID: \"cc6ebce3-3ee7-4b8d-a11d-1cb084a6b780\") " pod="openshift-ingress-canary/ingress-canary-4rch5" Jul 09 16:15:55.064296 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:55.064274 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xj7tc\" (UniqueName: \"kubernetes.io/projected/f54c8925-c09f-46d8-84fb-9f3af042dc41-kube-api-access-xj7tc\") pod \"kube-storage-version-migrator-operator-799cb7f958-xjcmf\" (UID: \"f54c8925-c09f-46d8-84fb-9f3af042dc41\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-xjcmf" Jul 09 16:15:55.064372 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:55.064335 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-prpjc\" (UniqueName: \"kubernetes.io/projected/3d316a09-f8b0-49cd-b8cb-ff0746dff65c-kube-api-access-prpjc\") pod \"network-check-source-869dcfcdcd-pzpbf\" (UID: \"3d316a09-f8b0-49cd-b8cb-ff0746dff65c\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-pzpbf" Jul 09 16:15:55.064441 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:55.064425 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8wcc\" (UniqueName: \"kubernetes.io/projected/bb610783-4a86-4291-a9f0-42deb83b322d-kube-api-access-l8wcc\") pod \"insights-operator-68d9dd94cf-6jksf\" (UID: \"bb610783-4a86-4291-a9f0-42deb83b322d\") " pod="openshift-insights/insights-operator-68d9dd94cf-6jksf" Jul 09 16:15:55.065338 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:55.065316 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlnq8\" (UniqueName: \"kubernetes.io/projected/a284ea64-610c-40f7-837b-fa0975c0a290-kube-api-access-tlnq8\") pod \"console-operator-7866497494-nzrk9\" (UID: \"a284ea64-610c-40f7-837b-fa0975c0a290\") " pod="openshift-console-operator/console-operator-7866497494-nzrk9" Jul 09 16:15:55.077532 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:55.077508 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-vzds7" Jul 09 16:15:55.089726 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:55.089701 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-6jksf" Jul 09 16:15:55.099647 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:55.099626 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qfsjp" Jul 09 16:15:55.114488 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:55.114463 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-xjcmf" Jul 09 16:15:55.123779 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:55.123754 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-nzrk9" Jul 09 16:15:55.141223 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:55.141190 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-pzpbf" Jul 09 16:15:55.150447 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:55.150392 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4rch5" Jul 09 16:15:55.348452 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:55.343953 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-vzds7"] Jul 09 16:15:55.360856 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:55.360820 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b943693_db9f_4291_90a6_1cf6cc00a714.slice/crio-9deb86c588c4afd43f2114198c22e8aa61b78788cc30e585a99ad1851b7ccec2 WatchSource:0}: Error finding container 9deb86c588c4afd43f2114198c22e8aa61b78788cc30e585a99ad1851b7ccec2: Status 404 returned error can't find the container with id 9deb86c588c4afd43f2114198c22e8aa61b78788cc30e585a99ad1851b7ccec2 Jul 09 16:15:55.371185 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:55.371072 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qfsjp"] Jul 09 16:15:55.374744 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:55.374674 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-6jksf"] Jul 09 16:15:55.385900 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:55.385177 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b7sxk" Jul 09 16:15:55.387585 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:55.386128 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kn7sp" Jul 09 16:15:55.387585 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:55.386501 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6wzq" Jul 09 16:15:55.390029 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:55.388290 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-k799q\"" Jul 09 16:15:55.390029 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:55.388560 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 09 16:15:55.390029 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:55.389545 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 09 16:15:55.390029 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:55.389588 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-c54pk\"" Jul 09 16:15:55.419652 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:55.419508 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-xjcmf"] Jul 09 16:15:55.422391 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:55.422367 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf54c8925_c09f_46d8_84fb_9f3af042dc41.slice/crio-4bad9d94078db6dc655db680ead79baa656b49a9ee252903bf266a7dd9947005 WatchSource:0}: Error finding container 4bad9d94078db6dc655db680ead79baa656b49a9ee252903bf266a7dd9947005: Status 404 returned error can't find the container with id 4bad9d94078db6dc655db680ead79baa656b49a9ee252903bf266a7dd9947005 Jul 09 16:15:55.425595 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:55.425376 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-nzrk9"] Jul 09 16:15:55.429340 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:55.429314 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda284ea64_610c_40f7_837b_fa0975c0a290.slice/crio-4bd923ad934ca77a90c76ed039fff8fd0086ba476d7ac0962b8f39ba0a4e3da5 WatchSource:0}: Error finding container 4bd923ad934ca77a90c76ed039fff8fd0086ba476d7ac0962b8f39ba0a4e3da5: Status 404 returned error can't find the container with id 4bd923ad934ca77a90c76ed039fff8fd0086ba476d7ac0962b8f39ba0a4e3da5 Jul 09 16:15:55.437942 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:55.437915 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-pzpbf"] Jul 09 16:15:55.440934 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:55.440907 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d316a09_f8b0_49cd_b8cb_ff0746dff65c.slice/crio-2dcd8d014663d19c7fc1333c236dd8eda6521977a6d442eb98dcca78b0e7b405 WatchSource:0}: Error finding container 2dcd8d014663d19c7fc1333c236dd8eda6521977a6d442eb98dcca78b0e7b405: Status 404 returned error can't find the container with id 2dcd8d014663d19c7fc1333c236dd8eda6521977a6d442eb98dcca78b0e7b405 Jul 09 16:15:55.442498 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:55.442407 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4rch5"] Jul 09 16:15:55.445972 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:15:55.445952 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc6ebce3_3ee7_4b8d_a11d_1cb084a6b780.slice/crio-3fefb67eb5b3545c6a901db65beaba571518ac3143969b4623349ed51da14222 WatchSource:0}: Error finding container 3fefb67eb5b3545c6a901db65beaba571518ac3143969b4623349ed51da14222: Status 404 returned error can't find the container with id 3fefb67eb5b3545c6a901db65beaba571518ac3143969b4623349ed51da14222 Jul 09 16:15:55.458968 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:55.458944 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/022557ee-2bed-48ef-a717-3bf8a3c53dc5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-vdbkl\" (UID: \"022557ee-2bed-48ef-a717-3bf8a3c53dc5\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-vdbkl" Jul 09 16:15:55.459154 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:55.458991 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a0f6347a-b82d-4c94-b411-1b621bd1101c-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-pf6gv\" (UID: \"a0f6347a-b82d-4c94-b411-1b621bd1101c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-pf6gv" Jul 09 16:15:55.459154 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:55.459024 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe900560-5002-4708-98e4-237e41014fbf-service-ca-bundle\") pod \"router-default-84d7f6c55d-59vjr\" (UID: \"fe900560-5002-4708-98e4-237e41014fbf\") " pod="openshift-ingress/router-default-84d7f6c55d-59vjr" Jul 09 16:15:55.459154 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:55.459050 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1032414d-f6cd-4936-a52f-4f57feddba8f-registry-tls\") pod \"image-registry-558d6fd964-d9xbs\" (UID: \"1032414d-f6cd-4936-a52f-4f57feddba8f\") " pod="openshift-image-registry/image-registry-558d6fd964-d9xbs" Jul 09 16:15:55.459154 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:55.459067 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e1cf555d-4290-4caa-a5f6-eaca63d05a1d-metrics-tls\") pod \"dns-default-q8zll\" (UID: \"e1cf555d-4290-4caa-a5f6-eaca63d05a1d\") " pod="openshift-dns/dns-default-q8zll" Jul 09 16:15:55.459154 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:55.459088 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe900560-5002-4708-98e4-237e41014fbf-metrics-certs\") pod \"router-default-84d7f6c55d-59vjr\" (UID: \"fe900560-5002-4708-98e4-237e41014fbf\") " pod="openshift-ingress/router-default-84d7f6c55d-59vjr" Jul 09 16:15:55.459154 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:55.459111 2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 09 16:15:55.459389 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:55.459178 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/022557ee-2bed-48ef-a717-3bf8a3c53dc5-cluster-monitoring-operator-tls podName:022557ee-2bed-48ef-a717-3bf8a3c53dc5 nodeName:}" failed. No retries permitted until 2026-07-09 16:15:56.459158738 +0000 UTC m=+41.652265111 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/022557ee-2bed-48ef-a717-3bf8a3c53dc5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-vdbkl" (UID: "022557ee-2bed-48ef-a717-3bf8a3c53dc5") : secret "cluster-monitoring-operator-tls" not found Jul 09 16:15:55.459389 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:55.459182 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 16:15:55.459389 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:55.459198 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fe900560-5002-4708-98e4-237e41014fbf-service-ca-bundle podName:fe900560-5002-4708-98e4-237e41014fbf nodeName:}" failed. No retries permitted until 2026-07-09 16:15:56.459188916 +0000 UTC m=+41.652295276 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/fe900560-5002-4708-98e4-237e41014fbf-service-ca-bundle") pod "router-default-84d7f6c55d-59vjr" (UID: "fe900560-5002-4708-98e4-237e41014fbf") : configmap references non-existent config key: service-ca.crt Jul 09 16:15:55.459389 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:55.459199 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-558d6fd964-d9xbs: secret "image-registry-tls" not found Jul 09 16:15:55.459389 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:55.459203 2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 09 16:15:55.459389 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:55.459188 2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 09 16:15:55.459389 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:55.459204 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 16:15:55.459389 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:55.459236 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1032414d-f6cd-4936-a52f-4f57feddba8f-registry-tls podName:1032414d-f6cd-4936-a52f-4f57feddba8f nodeName:}" failed. No retries permitted until 2026-07-09 16:15:56.459226719 +0000 UTC m=+41.652333109 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1032414d-f6cd-4936-a52f-4f57feddba8f-registry-tls") pod "image-registry-558d6fd964-d9xbs" (UID: "1032414d-f6cd-4936-a52f-4f57feddba8f") : secret "image-registry-tls" not found Jul 09 16:15:55.459389 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:55.459305 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe900560-5002-4708-98e4-237e41014fbf-metrics-certs podName:fe900560-5002-4708-98e4-237e41014fbf nodeName:}" failed. No retries permitted until 2026-07-09 16:15:56.459287298 +0000 UTC m=+41.652393671 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fe900560-5002-4708-98e4-237e41014fbf-metrics-certs") pod "router-default-84d7f6c55d-59vjr" (UID: "fe900560-5002-4708-98e4-237e41014fbf") : secret "router-metrics-certs-default" not found Jul 09 16:15:55.459389 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:55.459328 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0f6347a-b82d-4c94-b411-1b621bd1101c-samples-operator-tls podName:a0f6347a-b82d-4c94-b411-1b621bd1101c nodeName:}" failed. No retries permitted until 2026-07-09 16:15:56.45931763 +0000 UTC m=+41.652423989 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a0f6347a-b82d-4c94-b411-1b621bd1101c-samples-operator-tls") pod "cluster-samples-operator-65db858894-pf6gv" (UID: "a0f6347a-b82d-4c94-b411-1b621bd1101c") : secret "samples-operator-tls" not found Jul 09 16:15:55.459389 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:55.459345 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1cf555d-4290-4caa-a5f6-eaca63d05a1d-metrics-tls podName:e1cf555d-4290-4caa-a5f6-eaca63d05a1d nodeName:}" failed. No retries permitted until 2026-07-09 16:15:56.459338063 +0000 UTC m=+41.652444422 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e1cf555d-4290-4caa-a5f6-eaca63d05a1d-metrics-tls") pod "dns-default-q8zll" (UID: "e1cf555d-4290-4caa-a5f6-eaca63d05a1d") : secret "dns-default-metrics-tls" not found Jul 09 16:15:55.563565 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:55.563455 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-xjcmf" event={"ID":"f54c8925-c09f-46d8-84fb-9f3af042dc41","Type":"ContainerStarted","Data":"4bad9d94078db6dc655db680ead79baa656b49a9ee252903bf266a7dd9947005"} Jul 09 16:15:55.564562 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:55.564530 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-6jksf" event={"ID":"bb610783-4a86-4291-a9f0-42deb83b322d","Type":"ContainerStarted","Data":"50558ca2031adf48039692b149236812224d18e0425e12bc5b6f4d96cf280762"} Jul 09 16:15:55.565579 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:55.565554 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-vzds7" event={"ID":"4b943693-db9f-4291-90a6-1cf6cc00a714","Type":"ContainerStarted","Data":"9deb86c588c4afd43f2114198c22e8aa61b78788cc30e585a99ad1851b7ccec2"} Jul 09 16:15:55.566512 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:55.566484 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qfsjp" event={"ID":"79ec4c79-9eb6-427a-b2f3-e66a6cd57dcb","Type":"ContainerStarted","Data":"8088819972b9857aa95f8a5c26e215172075cfc60d576cdeb1c6bd08544e2d58"} Jul 09 16:15:55.567482 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:55.567460 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-nzrk9" event={"ID":"a284ea64-610c-40f7-837b-fa0975c0a290","Type":"ContainerStarted","Data":"4bd923ad934ca77a90c76ed039fff8fd0086ba476d7ac0962b8f39ba0a4e3da5"} Jul 09 16:15:55.568545 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:55.568522 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4rch5" event={"ID":"cc6ebce3-3ee7-4b8d-a11d-1cb084a6b780","Type":"ContainerStarted","Data":"3fefb67eb5b3545c6a901db65beaba571518ac3143969b4623349ed51da14222"} Jul 09 16:15:55.569526 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:55.569507 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-pzpbf" event={"ID":"3d316a09-f8b0-49cd-b8cb-ff0746dff65c","Type":"ContainerStarted","Data":"2dcd8d014663d19c7fc1333c236dd8eda6521977a6d442eb98dcca78b0e7b405"} Jul 09 16:15:56.468505 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:56.468464 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/022557ee-2bed-48ef-a717-3bf8a3c53dc5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-vdbkl\" (UID: \"022557ee-2bed-48ef-a717-3bf8a3c53dc5\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-vdbkl" Jul 09 16:15:56.469406 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:56.468531 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a0f6347a-b82d-4c94-b411-1b621bd1101c-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-pf6gv\" (UID: \"a0f6347a-b82d-4c94-b411-1b621bd1101c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-pf6gv" Jul 09 16:15:56.469406 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:56.468575 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe900560-5002-4708-98e4-237e41014fbf-service-ca-bundle\") pod \"router-default-84d7f6c55d-59vjr\" (UID: \"fe900560-5002-4708-98e4-237e41014fbf\") " pod="openshift-ingress/router-default-84d7f6c55d-59vjr" Jul 09 16:15:56.469406 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:56.468619 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1032414d-f6cd-4936-a52f-4f57feddba8f-registry-tls\") pod \"image-registry-558d6fd964-d9xbs\" (UID: \"1032414d-f6cd-4936-a52f-4f57feddba8f\") " pod="openshift-image-registry/image-registry-558d6fd964-d9xbs" Jul 09 16:15:56.469406 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:56.468652 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e1cf555d-4290-4caa-a5f6-eaca63d05a1d-metrics-tls\") pod \"dns-default-q8zll\" (UID: \"e1cf555d-4290-4caa-a5f6-eaca63d05a1d\") " pod="openshift-dns/dns-default-q8zll" Jul 09 16:15:56.469406 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:56.468680 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe900560-5002-4708-98e4-237e41014fbf-metrics-certs\") pod \"router-default-84d7f6c55d-59vjr\" (UID: \"fe900560-5002-4708-98e4-237e41014fbf\") " pod="openshift-ingress/router-default-84d7f6c55d-59vjr" Jul 09 16:15:56.469406 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:56.468825 2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 09 16:15:56.469406 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:56.468914 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe900560-5002-4708-98e4-237e41014fbf-metrics-certs podName:fe900560-5002-4708-98e4-237e41014fbf nodeName:}" failed. No retries permitted until 2026-07-09 16:15:58.46889022 +0000 UTC m=+43.661996593 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fe900560-5002-4708-98e4-237e41014fbf-metrics-certs") pod "router-default-84d7f6c55d-59vjr" (UID: "fe900560-5002-4708-98e4-237e41014fbf") : secret "router-metrics-certs-default" not found Jul 09 16:15:56.469406 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:56.469340 2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 09 16:15:56.469406 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:56.469396 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/022557ee-2bed-48ef-a717-3bf8a3c53dc5-cluster-monitoring-operator-tls podName:022557ee-2bed-48ef-a717-3bf8a3c53dc5 nodeName:}" failed. No retries permitted until 2026-07-09 16:15:58.469380008 +0000 UTC m=+43.662486368 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/022557ee-2bed-48ef-a717-3bf8a3c53dc5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-vdbkl" (UID: "022557ee-2bed-48ef-a717-3bf8a3c53dc5") : secret "cluster-monitoring-operator-tls" not found Jul 09 16:15:56.469941 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:56.469489 2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 09 16:15:56.469941 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:56.469527 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0f6347a-b82d-4c94-b411-1b621bd1101c-samples-operator-tls podName:a0f6347a-b82d-4c94-b411-1b621bd1101c nodeName:}" failed. No retries permitted until 2026-07-09 16:15:58.469515228 +0000 UTC m=+43.662621591 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a0f6347a-b82d-4c94-b411-1b621bd1101c-samples-operator-tls") pod "cluster-samples-operator-65db858894-pf6gv" (UID: "a0f6347a-b82d-4c94-b411-1b621bd1101c") : secret "samples-operator-tls" not found Jul 09 16:15:56.469941 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:56.469594 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fe900560-5002-4708-98e4-237e41014fbf-service-ca-bundle podName:fe900560-5002-4708-98e4-237e41014fbf nodeName:}" failed. No retries permitted until 2026-07-09 16:15:58.469582812 +0000 UTC m=+43.662689172 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/fe900560-5002-4708-98e4-237e41014fbf-service-ca-bundle") pod "router-default-84d7f6c55d-59vjr" (UID: "fe900560-5002-4708-98e4-237e41014fbf") : configmap references non-existent config key: service-ca.crt Jul 09 16:15:56.469941 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:56.469658 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 16:15:56.469941 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:56.469670 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-558d6fd964-d9xbs: secret "image-registry-tls" not found Jul 09 16:15:56.469941 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:56.469701 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1032414d-f6cd-4936-a52f-4f57feddba8f-registry-tls podName:1032414d-f6cd-4936-a52f-4f57feddba8f nodeName:}" failed. No retries permitted until 2026-07-09 16:15:58.469689645 +0000 UTC m=+43.662796020 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1032414d-f6cd-4936-a52f-4f57feddba8f-registry-tls") pod "image-registry-558d6fd964-d9xbs" (UID: "1032414d-f6cd-4936-a52f-4f57feddba8f") : secret "image-registry-tls" not found Jul 09 16:15:56.469941 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:56.469757 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 16:15:56.469941 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:56.469788 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1cf555d-4290-4caa-a5f6-eaca63d05a1d-metrics-tls podName:e1cf555d-4290-4caa-a5f6-eaca63d05a1d nodeName:}" failed. No retries permitted until 2026-07-09 16:15:58.469776235 +0000 UTC m=+43.662882597 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e1cf555d-4290-4caa-a5f6-eaca63d05a1d-metrics-tls") pod "dns-default-q8zll" (UID: "e1cf555d-4290-4caa-a5f6-eaca63d05a1d") : secret "dns-default-metrics-tls" not found Jul 09 16:15:58.491673 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:58.491473 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1032414d-f6cd-4936-a52f-4f57feddba8f-registry-tls\") pod \"image-registry-558d6fd964-d9xbs\" (UID: \"1032414d-f6cd-4936-a52f-4f57feddba8f\") " pod="openshift-image-registry/image-registry-558d6fd964-d9xbs" Jul 09 16:15:58.491673 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:58.491527 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e1cf555d-4290-4caa-a5f6-eaca63d05a1d-metrics-tls\") pod \"dns-default-q8zll\" (UID: \"e1cf555d-4290-4caa-a5f6-eaca63d05a1d\") " pod="openshift-dns/dns-default-q8zll" Jul 09 16:15:58.491673 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:58.491566 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe900560-5002-4708-98e4-237e41014fbf-metrics-certs\") pod \"router-default-84d7f6c55d-59vjr\" (UID: \"fe900560-5002-4708-98e4-237e41014fbf\") " pod="openshift-ingress/router-default-84d7f6c55d-59vjr" Jul 09 16:15:58.491673 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:58.491637 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 16:15:58.491673 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:58.491653 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 16:15:58.491673 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:58.491672 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/022557ee-2bed-48ef-a717-3bf8a3c53dc5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-vdbkl\" (UID: \"022557ee-2bed-48ef-a717-3bf8a3c53dc5\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-vdbkl" Jul 09 16:15:58.492256 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:58.491720 2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 09 16:15:58.492256 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:58.491730 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1cf555d-4290-4caa-a5f6-eaca63d05a1d-metrics-tls podName:e1cf555d-4290-4caa-a5f6-eaca63d05a1d nodeName:}" failed. No retries permitted until 2026-07-09 16:16:02.491707851 +0000 UTC m=+47.684814211 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e1cf555d-4290-4caa-a5f6-eaca63d05a1d-metrics-tls") pod "dns-default-q8zll" (UID: "e1cf555d-4290-4caa-a5f6-eaca63d05a1d") : secret "dns-default-metrics-tls" not found Jul 09 16:15:58.492256 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:58.491759 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a0f6347a-b82d-4c94-b411-1b621bd1101c-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-pf6gv\" (UID: \"a0f6347a-b82d-4c94-b411-1b621bd1101c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-pf6gv" Jul 09 16:15:58.492256 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:58.491767 2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 09 16:15:58.492256 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:58.491660 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-558d6fd964-d9xbs: secret "image-registry-tls" not found Jul 09 16:15:58.492256 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:58.491825 2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 09 16:15:58.492256 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:58.491771 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe900560-5002-4708-98e4-237e41014fbf-metrics-certs podName:fe900560-5002-4708-98e4-237e41014fbf nodeName:}" failed. No retries permitted until 2026-07-09 16:16:02.491754944 +0000 UTC m=+47.684861308 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fe900560-5002-4708-98e4-237e41014fbf-metrics-certs") pod "router-default-84d7f6c55d-59vjr" (UID: "fe900560-5002-4708-98e4-237e41014fbf") : secret "router-metrics-certs-default" not found Jul 09 16:15:58.492256 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:58.491902 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/022557ee-2bed-48ef-a717-3bf8a3c53dc5-cluster-monitoring-operator-tls podName:022557ee-2bed-48ef-a717-3bf8a3c53dc5 nodeName:}" failed. No retries permitted until 2026-07-09 16:16:02.491864142 +0000 UTC m=+47.684970510 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/022557ee-2bed-48ef-a717-3bf8a3c53dc5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-vdbkl" (UID: "022557ee-2bed-48ef-a717-3bf8a3c53dc5") : secret "cluster-monitoring-operator-tls" not found Jul 09 16:15:58.492256 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:58.491935 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1032414d-f6cd-4936-a52f-4f57feddba8f-registry-tls podName:1032414d-f6cd-4936-a52f-4f57feddba8f nodeName:}" failed. No retries permitted until 2026-07-09 16:16:02.491921349 +0000 UTC m=+47.685027707 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1032414d-f6cd-4936-a52f-4f57feddba8f-registry-tls") pod "image-registry-558d6fd964-d9xbs" (UID: "1032414d-f6cd-4936-a52f-4f57feddba8f") : secret "image-registry-tls" not found Jul 09 16:15:58.492256 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:58.491951 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0f6347a-b82d-4c94-b411-1b621bd1101c-samples-operator-tls podName:a0f6347a-b82d-4c94-b411-1b621bd1101c nodeName:}" failed. No retries permitted until 2026-07-09 16:16:02.491945138 +0000 UTC m=+47.685051497 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a0f6347a-b82d-4c94-b411-1b621bd1101c-samples-operator-tls") pod "cluster-samples-operator-65db858894-pf6gv" (UID: "a0f6347a-b82d-4c94-b411-1b621bd1101c") : secret "samples-operator-tls" not found Jul 09 16:15:58.492256 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:15:58.492004 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe900560-5002-4708-98e4-237e41014fbf-service-ca-bundle\") pod \"router-default-84d7f6c55d-59vjr\" (UID: \"fe900560-5002-4708-98e4-237e41014fbf\") " pod="openshift-ingress/router-default-84d7f6c55d-59vjr" Jul 09 16:15:58.492256 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:15:58.492106 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fe900560-5002-4708-98e4-237e41014fbf-service-ca-bundle podName:fe900560-5002-4708-98e4-237e41014fbf nodeName:}" failed. No retries permitted until 2026-07-09 16:16:02.492096922 +0000 UTC m=+47.685203284 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/fe900560-5002-4708-98e4-237e41014fbf-service-ca-bundle") pod "router-default-84d7f6c55d-59vjr" (UID: "fe900560-5002-4708-98e4-237e41014fbf") : configmap references non-existent config key: service-ca.crt Jul 09 16:16:02.529864 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:02.529611 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/022557ee-2bed-48ef-a717-3bf8a3c53dc5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-vdbkl\" (UID: \"022557ee-2bed-48ef-a717-3bf8a3c53dc5\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-vdbkl" Jul 09 16:16:02.530322 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:02.529920 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a0f6347a-b82d-4c94-b411-1b621bd1101c-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-pf6gv\" (UID: \"a0f6347a-b82d-4c94-b411-1b621bd1101c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-pf6gv" Jul 09 16:16:02.530322 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:02.529964 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe900560-5002-4708-98e4-237e41014fbf-service-ca-bundle\") pod \"router-default-84d7f6c55d-59vjr\" (UID: \"fe900560-5002-4708-98e4-237e41014fbf\") " pod="openshift-ingress/router-default-84d7f6c55d-59vjr" Jul 09 16:16:02.530322 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:02.530000 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1032414d-f6cd-4936-a52f-4f57feddba8f-registry-tls\") pod \"image-registry-558d6fd964-d9xbs\" (UID: \"1032414d-f6cd-4936-a52f-4f57feddba8f\") " pod="openshift-image-registry/image-registry-558d6fd964-d9xbs" Jul 09 16:16:02.530322 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:16:02.529699 2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 09 16:16:02.530322 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:16:02.530037 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fe900560-5002-4708-98e4-237e41014fbf-service-ca-bundle podName:fe900560-5002-4708-98e4-237e41014fbf nodeName:}" failed. No retries permitted until 2026-07-09 16:16:10.530023196 +0000 UTC m=+55.723129556 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/fe900560-5002-4708-98e4-237e41014fbf-service-ca-bundle") pod "router-default-84d7f6c55d-59vjr" (UID: "fe900560-5002-4708-98e4-237e41014fbf") : configmap references non-existent config key: service-ca.crt Jul 09 16:16:02.530322 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:16:02.529965 2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 09 16:16:02.530322 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:02.530062 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e1cf555d-4290-4caa-a5f6-eaca63d05a1d-metrics-tls\") pod \"dns-default-q8zll\" (UID: \"e1cf555d-4290-4caa-a5f6-eaca63d05a1d\") " pod="openshift-dns/dns-default-q8zll" Jul 09 16:16:02.530322 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:16:02.530083 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0f6347a-b82d-4c94-b411-1b621bd1101c-samples-operator-tls podName:a0f6347a-b82d-4c94-b411-1b621bd1101c nodeName:}" failed. No retries permitted until 2026-07-09 16:16:10.530075879 +0000 UTC m=+55.723182238 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a0f6347a-b82d-4c94-b411-1b621bd1101c-samples-operator-tls") pod "cluster-samples-operator-65db858894-pf6gv" (UID: "a0f6347a-b82d-4c94-b411-1b621bd1101c") : secret "samples-operator-tls" not found Jul 09 16:16:02.530322 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:16:02.530099 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 16:16:02.530322 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:16:02.530115 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-558d6fd964-d9xbs: secret "image-registry-tls" not found Jul 09 16:16:02.530322 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:16:02.530125 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/022557ee-2bed-48ef-a717-3bf8a3c53dc5-cluster-monitoring-operator-tls podName:022557ee-2bed-48ef-a717-3bf8a3c53dc5 nodeName:}" failed. No retries permitted until 2026-07-09 16:16:10.530097854 +0000 UTC m=+55.723204222 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/022557ee-2bed-48ef-a717-3bf8a3c53dc5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-vdbkl" (UID: "022557ee-2bed-48ef-a717-3bf8a3c53dc5") : secret "cluster-monitoring-operator-tls" not found Jul 09 16:16:02.530322 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:02.530099 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe900560-5002-4708-98e4-237e41014fbf-metrics-certs\") pod \"router-default-84d7f6c55d-59vjr\" (UID: \"fe900560-5002-4708-98e4-237e41014fbf\") " pod="openshift-ingress/router-default-84d7f6c55d-59vjr" Jul 09 16:16:02.530322 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:16:02.530131 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 16:16:02.530322 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:16:02.530143 2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 09 16:16:02.530322 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:16:02.530152 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1032414d-f6cd-4936-a52f-4f57feddba8f-registry-tls podName:1032414d-f6cd-4936-a52f-4f57feddba8f nodeName:}" failed. No retries permitted until 2026-07-09 16:16:10.530143052 +0000 UTC m=+55.723249410 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1032414d-f6cd-4936-a52f-4f57feddba8f-registry-tls") pod "image-registry-558d6fd964-d9xbs" (UID: "1032414d-f6cd-4936-a52f-4f57feddba8f") : secret "image-registry-tls" not found Jul 09 16:16:02.530322 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:16:02.530180 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1cf555d-4290-4caa-a5f6-eaca63d05a1d-metrics-tls podName:e1cf555d-4290-4caa-a5f6-eaca63d05a1d nodeName:}" failed. No retries permitted until 2026-07-09 16:16:10.530171941 +0000 UTC m=+55.723278306 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e1cf555d-4290-4caa-a5f6-eaca63d05a1d-metrics-tls") pod "dns-default-q8zll" (UID: "e1cf555d-4290-4caa-a5f6-eaca63d05a1d") : secret "dns-default-metrics-tls" not found Jul 09 16:16:02.530822 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:16:02.530196 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe900560-5002-4708-98e4-237e41014fbf-metrics-certs podName:fe900560-5002-4708-98e4-237e41014fbf nodeName:}" failed. No retries permitted until 2026-07-09 16:16:10.53018856 +0000 UTC m=+55.723294919 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fe900560-5002-4708-98e4-237e41014fbf-metrics-certs") pod "router-default-84d7f6c55d-59vjr" (UID: "fe900560-5002-4708-98e4-237e41014fbf") : secret "router-metrics-certs-default" not found Jul 09 16:16:03.601735 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:03.601691 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-nzrk9_a284ea64-610c-40f7-837b-fa0975c0a290/console-operator/0.log" Jul 09 16:16:03.602233 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:03.601743 2569 generic.go:356] "Generic (PLEG): container finished" podID="a284ea64-610c-40f7-837b-fa0975c0a290" containerID="79c2bcad1cfa0af8a08b77f0c9a5f81d5031cb98a0401a6147e676c06d233804" exitCode=255 Jul 09 16:16:03.602233 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:03.601844 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-nzrk9" event={"ID":"a284ea64-610c-40f7-837b-fa0975c0a290","Type":"ContainerDied","Data":"79c2bcad1cfa0af8a08b77f0c9a5f81d5031cb98a0401a6147e676c06d233804"} Jul 09 16:16:03.602233 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:03.602181 2569 scope.go:117] "RemoveContainer" containerID="79c2bcad1cfa0af8a08b77f0c9a5f81d5031cb98a0401a6147e676c06d233804" Jul 09 16:16:03.604233 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:03.604203 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4rch5" event={"ID":"cc6ebce3-3ee7-4b8d-a11d-1cb084a6b780","Type":"ContainerStarted","Data":"b6ce4c65623463b0cee38cb34c4acf204ada65391265b88a7e40242d796d5046"} Jul 09 16:16:03.606223 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:03.606195 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-pzpbf" event={"ID":"3d316a09-f8b0-49cd-b8cb-ff0746dff65c","Type":"ContainerStarted","Data":"94a047cafc3310e94da58bca973034c7e561ce91ccdf5cbe735ebcc49d2142d8"} Jul 09 16:16:03.608407 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:03.608377 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-xjcmf" event={"ID":"f54c8925-c09f-46d8-84fb-9f3af042dc41","Type":"ContainerStarted","Data":"3562980d29b75d34d213ad74c95f5a34b87cc98ce953262972e5af5be3a4d338"} Jul 09 16:16:03.610152 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:03.609863 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-6jksf" event={"ID":"bb610783-4a86-4291-a9f0-42deb83b322d","Type":"ContainerStarted","Data":"182e8256b448b6b5aacaaf85377f7879b8a3446b5cedcef6361768242b9c0043"} Jul 09 16:16:03.611517 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:03.611493 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-vzds7" event={"ID":"4b943693-db9f-4291-90a6-1cf6cc00a714","Type":"ContainerStarted","Data":"ac7e7a71b54971d682754c8afcd61988361ee2607cb04d9fc1315953e4ee0ac0"} Jul 09 16:16:03.613393 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:03.613364 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qfsjp" event={"ID":"79ec4c79-9eb6-427a-b2f3-e66a6cd57dcb","Type":"ContainerStarted","Data":"8f881a8a49d27bed47d97f70fb59fa88bdfcb263bf280abfe7f8b5616c712359"} Jul 09 16:16:03.635634 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:03.635581 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qfsjp" podStartSLOduration=13.324735197 podStartE2EDuration="20.635559553s" totalImagesPullingTime="7.310824356s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:15:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:16:03.634865331 +0000 UTC m=+48.827971714" watchObservedRunningTime="2026-07-09 16:16:03.635559553 +0000 UTC m=+48.828665932" Jul 09 16:16:03.655313 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:03.655262 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-pzpbf" podStartSLOduration=13.400653046 podStartE2EDuration="20.655240035s" totalImagesPullingTime="7.254586989s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:15:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:16:03.653983612 +0000 UTC m=+48.847089994" watchObservedRunningTime="2026-07-09 16:16:03.655240035 +0000 UTC m=+48.848346418" Jul 09 16:16:03.676279 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:03.676219 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-4rch5" podStartSLOduration=2.426378548 podStartE2EDuration="9.676195969s" totalImagesPullingTime="7.249817421s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:15:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:16:03.675012469 +0000 UTC m=+48.868118853" watchObservedRunningTime="2026-07-09 16:16:03.676195969 +0000 UTC m=+48.869302352" Jul 09 16:16:03.697227 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:03.697122 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-68d9dd94cf-6jksf" podStartSLOduration=13.392314633 podStartE2EDuration="20.697100191s" totalImagesPullingTime="7.304785558s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:15:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:16:03.696615872 +0000 UTC m=+48.889722254" watchObservedRunningTime="2026-07-09 16:16:03.697100191 +0000 UTC m=+48.890206563" Jul 09 16:16:03.717281 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:03.717164 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-xjcmf" podStartSLOduration=13.445231837 podStartE2EDuration="20.717139099s" totalImagesPullingTime="7.271907262s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:15:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:16:03.715657455 +0000 UTC m=+48.908763837" watchObservedRunningTime="2026-07-09 16:16:03.717139099 +0000 UTC m=+48.910245481" Jul 09 16:16:03.739413 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:03.739141 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-vzds7" podStartSLOduration=13.525659326 podStartE2EDuration="20.739101795s" totalImagesPullingTime="7.213442469s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:15:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:16:03.739024778 +0000 UTC m=+48.932131160" watchObservedRunningTime="2026-07-09 16:16:03.739101795 +0000 UTC m=+48.932208177" Jul 09 16:16:03.990565 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:03.990473 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-rkqp6"] Jul 09 16:16:04.012029 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:04.011999 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rkqp6"] Jul 09 16:16:04.012208 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:04.012155 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rkqp6" Jul 09 16:16:04.015931 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:04.015904 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 09 16:16:04.016076 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:04.015959 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 09 16:16:04.016076 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:04.015988 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-4w6wt\"" Jul 09 16:16:04.045994 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:04.045949 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/031a3833-1870-44a3-abfb-9a840df32ac3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rkqp6\" (UID: \"031a3833-1870-44a3-abfb-9a840df32ac3\") " pod="openshift-insights/insights-runtime-extractor-rkqp6" Jul 09 16:16:04.046176 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:04.046059 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/031a3833-1870-44a3-abfb-9a840df32ac3-crio-socket\") pod \"insights-runtime-extractor-rkqp6\" (UID: \"031a3833-1870-44a3-abfb-9a840df32ac3\") " pod="openshift-insights/insights-runtime-extractor-rkqp6" Jul 09 16:16:04.046176 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:04.046120 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/031a3833-1870-44a3-abfb-9a840df32ac3-data-volume\") pod \"insights-runtime-extractor-rkqp6\" (UID: \"031a3833-1870-44a3-abfb-9a840df32ac3\") " pod="openshift-insights/insights-runtime-extractor-rkqp6" Jul 09 16:16:04.046176 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:04.046151 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pl4w\" (UniqueName: \"kubernetes.io/projected/031a3833-1870-44a3-abfb-9a840df32ac3-kube-api-access-7pl4w\") pod \"insights-runtime-extractor-rkqp6\" (UID: \"031a3833-1870-44a3-abfb-9a840df32ac3\") " pod="openshift-insights/insights-runtime-extractor-rkqp6" Jul 09 16:16:04.046324 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:04.046268 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/031a3833-1870-44a3-abfb-9a840df32ac3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rkqp6\" (UID: \"031a3833-1870-44a3-abfb-9a840df32ac3\") " pod="openshift-insights/insights-runtime-extractor-rkqp6" Jul 09 16:16:04.146738 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:04.146697 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/031a3833-1870-44a3-abfb-9a840df32ac3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rkqp6\" (UID: \"031a3833-1870-44a3-abfb-9a840df32ac3\") " pod="openshift-insights/insights-runtime-extractor-rkqp6" Jul 09 16:16:04.146914 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:04.146780 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/031a3833-1870-44a3-abfb-9a840df32ac3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rkqp6\" (UID: \"031a3833-1870-44a3-abfb-9a840df32ac3\") " pod="openshift-insights/insights-runtime-extractor-rkqp6" Jul 09 16:16:04.146914 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:16:04.146887 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 16:16:04.147039 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:16:04.146940 2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 16:16:04.147039 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:04.146887 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/031a3833-1870-44a3-abfb-9a840df32ac3-crio-socket\") pod \"insights-runtime-extractor-rkqp6\" (UID: \"031a3833-1870-44a3-abfb-9a840df32ac3\") " pod="openshift-insights/insights-runtime-extractor-rkqp6" Jul 09 16:16:04.147039 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:04.146950 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/031a3833-1870-44a3-abfb-9a840df32ac3-crio-socket\") pod \"insights-runtime-extractor-rkqp6\" (UID: \"031a3833-1870-44a3-abfb-9a840df32ac3\") " pod="openshift-insights/insights-runtime-extractor-rkqp6" Jul 09 16:16:04.147039 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:16:04.146956 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/031a3833-1870-44a3-abfb-9a840df32ac3-insights-runtime-extractor-tls podName:031a3833-1870-44a3-abfb-9a840df32ac3 nodeName:}" failed. No retries permitted until 2026-07-09 16:16:04.64693802 +0000 UTC m=+49.840044394 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/031a3833-1870-44a3-abfb-9a840df32ac3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rkqp6" (UID: "031a3833-1870-44a3-abfb-9a840df32ac3") : secret "insights-runtime-extractor-tls" not found Jul 09 16:16:04.147039 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:16:04.147028 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/031a3833-1870-44a3-abfb-9a840df32ac3-kube-rbac-proxy-cm podName:031a3833-1870-44a3-abfb-9a840df32ac3 nodeName:}" failed. No retries permitted until 2026-07-09 16:16:04.647008757 +0000 UTC m=+49.840115129 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/031a3833-1870-44a3-abfb-9a840df32ac3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rkqp6" (UID: "031a3833-1870-44a3-abfb-9a840df32ac3") : configmap "kube-rbac-proxy" not found Jul 09 16:16:04.147218 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:04.147088 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/031a3833-1870-44a3-abfb-9a840df32ac3-data-volume\") pod \"insights-runtime-extractor-rkqp6\" (UID: \"031a3833-1870-44a3-abfb-9a840df32ac3\") " pod="openshift-insights/insights-runtime-extractor-rkqp6" Jul 09 16:16:04.147218 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:04.147135 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pl4w\" (UniqueName: \"kubernetes.io/projected/031a3833-1870-44a3-abfb-9a840df32ac3-kube-api-access-7pl4w\") pod \"insights-runtime-extractor-rkqp6\" (UID: \"031a3833-1870-44a3-abfb-9a840df32ac3\") " pod="openshift-insights/insights-runtime-extractor-rkqp6" Jul 09 16:16:04.147416 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:04.147399 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/031a3833-1870-44a3-abfb-9a840df32ac3-data-volume\") pod \"insights-runtime-extractor-rkqp6\" (UID: \"031a3833-1870-44a3-abfb-9a840df32ac3\") " pod="openshift-insights/insights-runtime-extractor-rkqp6" Jul 09 16:16:04.157788 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:04.157760 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pl4w\" (UniqueName: \"kubernetes.io/projected/031a3833-1870-44a3-abfb-9a840df32ac3-kube-api-access-7pl4w\") pod \"insights-runtime-extractor-rkqp6\" (UID: \"031a3833-1870-44a3-abfb-9a840df32ac3\") " pod="openshift-insights/insights-runtime-extractor-rkqp6" Jul 09 16:16:04.619643 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:04.619284 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-nzrk9_a284ea64-610c-40f7-837b-fa0975c0a290/console-operator/1.log" Jul 09 16:16:04.620999 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:04.620377 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-nzrk9_a284ea64-610c-40f7-837b-fa0975c0a290/console-operator/0.log" Jul 09 16:16:04.620999 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:04.620418 2569 generic.go:356] "Generic (PLEG): container finished" podID="a284ea64-610c-40f7-837b-fa0975c0a290" containerID="d0e6b4d7e71720cddd1920aa6e1bc5359483537173f8796e2414efc89fd6f9ae" exitCode=255 Jul 09 16:16:04.621182 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:04.621028 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-nzrk9" event={"ID":"a284ea64-610c-40f7-837b-fa0975c0a290","Type":"ContainerDied","Data":"d0e6b4d7e71720cddd1920aa6e1bc5359483537173f8796e2414efc89fd6f9ae"} Jul 09 16:16:04.621182 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:04.621089 2569 scope.go:117] "RemoveContainer" containerID="79c2bcad1cfa0af8a08b77f0c9a5f81d5031cb98a0401a6147e676c06d233804" Jul 09 16:16:04.621525 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:04.621409 2569 scope.go:117] "RemoveContainer" containerID="d0e6b4d7e71720cddd1920aa6e1bc5359483537173f8796e2414efc89fd6f9ae" Jul 09 16:16:04.621641 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:16:04.621578 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-nzrk9_openshift-console-operator(a284ea64-610c-40f7-837b-fa0975c0a290)\"" pod="openshift-console-operator/console-operator-7866497494-nzrk9" podUID="a284ea64-610c-40f7-837b-fa0975c0a290" Jul 09 16:16:04.652545 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:04.652502 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/031a3833-1870-44a3-abfb-9a840df32ac3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rkqp6\" (UID: \"031a3833-1870-44a3-abfb-9a840df32ac3\") " pod="openshift-insights/insights-runtime-extractor-rkqp6" Jul 09 16:16:04.652740 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:16:04.652631 2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 16:16:04.652815 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:16:04.652763 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/031a3833-1870-44a3-abfb-9a840df32ac3-kube-rbac-proxy-cm podName:031a3833-1870-44a3-abfb-9a840df32ac3 nodeName:}" failed. No retries permitted until 2026-07-09 16:16:05.652686081 +0000 UTC m=+50.845792439 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/031a3833-1870-44a3-abfb-9a840df32ac3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rkqp6" (UID: "031a3833-1870-44a3-abfb-9a840df32ac3") : configmap "kube-rbac-proxy" not found Jul 09 16:16:04.653523 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:04.653487 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/031a3833-1870-44a3-abfb-9a840df32ac3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rkqp6\" (UID: \"031a3833-1870-44a3-abfb-9a840df32ac3\") " pod="openshift-insights/insights-runtime-extractor-rkqp6" Jul 09 16:16:04.653990 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:16:04.653969 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 16:16:04.654080 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:16:04.654024 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/031a3833-1870-44a3-abfb-9a840df32ac3-insights-runtime-extractor-tls podName:031a3833-1870-44a3-abfb-9a840df32ac3 nodeName:}" failed. No retries permitted until 2026-07-09 16:16:05.654008471 +0000 UTC m=+50.847114832 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/031a3833-1870-44a3-abfb-9a840df32ac3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rkqp6" (UID: "031a3833-1870-44a3-abfb-9a840df32ac3") : secret "insights-runtime-extractor-tls" not found Jul 09 16:16:04.719459 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:04.719428 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-fr9br"] Jul 09 16:16:04.742586 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:04.742557 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-fr9br"] Jul 09 16:16:04.742735 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:04.742684 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-fr9br" Jul 09 16:16:04.745595 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:04.745569 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 09 16:16:04.745733 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:04.745629 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-wxh8j\"" Jul 09 16:16:04.745733 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:04.745632 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 09 16:16:04.855800 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:04.855757 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpw52\" (UniqueName: \"kubernetes.io/projected/c2cc6efd-36ec-42da-9025-c01334ab3bc0-kube-api-access-hpw52\") pod \"migrator-7f59b54f6c-fr9br\" (UID: \"c2cc6efd-36ec-42da-9025-c01334ab3bc0\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-fr9br" Jul 09 16:16:04.957001 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:04.956915 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpw52\" (UniqueName: \"kubernetes.io/projected/c2cc6efd-36ec-42da-9025-c01334ab3bc0-kube-api-access-hpw52\") pod \"migrator-7f59b54f6c-fr9br\" (UID: \"c2cc6efd-36ec-42da-9025-c01334ab3bc0\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-fr9br" Jul 09 16:16:04.965537 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:04.965506 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpw52\" (UniqueName: \"kubernetes.io/projected/c2cc6efd-36ec-42da-9025-c01334ab3bc0-kube-api-access-hpw52\") pod \"migrator-7f59b54f6c-fr9br\" (UID: \"c2cc6efd-36ec-42da-9025-c01334ab3bc0\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-fr9br" Jul 09 16:16:05.052901 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:05.052840 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-fr9br" Jul 09 16:16:05.124371 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:05.124293 2569 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7866497494-nzrk9" Jul 09 16:16:05.124615 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:05.124577 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-nzrk9" Jul 09 16:16:05.202584 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:05.202558 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-fr9br"] Jul 09 16:16:05.204957 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:16:05.204926 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2cc6efd_36ec_42da_9025_c01334ab3bc0.slice/crio-fbc852b57d829683c7f728d94ba3ded881cf44cb4a492e4ccd2697d50559ea6b WatchSource:0}: Error finding container fbc852b57d829683c7f728d94ba3ded881cf44cb4a492e4ccd2697d50559ea6b: Status 404 returned error can't find the container with id fbc852b57d829683c7f728d94ba3ded881cf44cb4a492e4ccd2697d50559ea6b Jul 09 16:16:05.475593 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:05.475356 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xx9mg_88be5834-3fd7-45da-83d5-4cb1d0f84c75/dns-node-resolver/0.log" Jul 09 16:16:05.625358 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:05.625320 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-fr9br" event={"ID":"c2cc6efd-36ec-42da-9025-c01334ab3bc0","Type":"ContainerStarted","Data":"fbc852b57d829683c7f728d94ba3ded881cf44cb4a492e4ccd2697d50559ea6b"} Jul 09 16:16:05.627035 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:05.627014 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-nzrk9_a284ea64-610c-40f7-837b-fa0975c0a290/console-operator/1.log" Jul 09 16:16:05.627420 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:05.627405 2569 scope.go:117] "RemoveContainer" containerID="d0e6b4d7e71720cddd1920aa6e1bc5359483537173f8796e2414efc89fd6f9ae" Jul 09 16:16:05.627623 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:16:05.627601 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-nzrk9_openshift-console-operator(a284ea64-610c-40f7-837b-fa0975c0a290)\"" pod="openshift-console-operator/console-operator-7866497494-nzrk9" podUID="a284ea64-610c-40f7-837b-fa0975c0a290" Jul 09 16:16:05.664937 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:05.664899 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/031a3833-1870-44a3-abfb-9a840df32ac3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rkqp6\" (UID: \"031a3833-1870-44a3-abfb-9a840df32ac3\") " pod="openshift-insights/insights-runtime-extractor-rkqp6" Jul 09 16:16:05.665111 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:05.664966 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/031a3833-1870-44a3-abfb-9a840df32ac3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rkqp6\" (UID: \"031a3833-1870-44a3-abfb-9a840df32ac3\") " pod="openshift-insights/insights-runtime-extractor-rkqp6" Jul 09 16:16:05.665111 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:16:05.665089 2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 16:16:05.665238 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:16:05.665093 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 16:16:05.665238 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:16:05.665161 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/031a3833-1870-44a3-abfb-9a840df32ac3-kube-rbac-proxy-cm podName:031a3833-1870-44a3-abfb-9a840df32ac3 nodeName:}" failed. No retries permitted until 2026-07-09 16:16:07.665139504 +0000 UTC m=+52.858245876 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/031a3833-1870-44a3-abfb-9a840df32ac3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rkqp6" (UID: "031a3833-1870-44a3-abfb-9a840df32ac3") : configmap "kube-rbac-proxy" not found Jul 09 16:16:05.665319 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:16:05.665268 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/031a3833-1870-44a3-abfb-9a840df32ac3-insights-runtime-extractor-tls podName:031a3833-1870-44a3-abfb-9a840df32ac3 nodeName:}" failed. No retries permitted until 2026-07-09 16:16:07.665253209 +0000 UTC m=+52.858359567 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/031a3833-1870-44a3-abfb-9a840df32ac3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rkqp6" (UID: "031a3833-1870-44a3-abfb-9a840df32ac3") : secret "insights-runtime-extractor-tls" not found Jul 09 16:16:06.275268 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:06.275240 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jj6qs_c17a000a-a79a-4dd8-af71-1fc47077bea7/node-ca/0.log" Jul 09 16:16:06.567745 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:06.567661 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-2trt8"] Jul 09 16:16:06.580751 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:06.580704 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-2trt8" Jul 09 16:16:06.581930 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:06.581837 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-2trt8"] Jul 09 16:16:06.583988 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:06.583827 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 09 16:16:06.583988 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:06.583851 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-wvkqc\"" Jul 09 16:16:06.583988 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:06.583829 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 09 16:16:06.585169 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:06.584983 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 09 16:16:06.585169 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:06.585066 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 09 16:16:06.632074 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:06.632047 2569 scope.go:117] "RemoveContainer" containerID="d0e6b4d7e71720cddd1920aa6e1bc5359483537173f8796e2414efc89fd6f9ae" Jul 09 16:16:06.632523 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:16:06.632253 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-nzrk9_openshift-console-operator(a284ea64-610c-40f7-837b-fa0975c0a290)\"" pod="openshift-console-operator/console-operator-7866497494-nzrk9" podUID="a284ea64-610c-40f7-837b-fa0975c0a290" Jul 09 16:16:06.674805 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:06.674770 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/74595786-1e43-465f-bdbe-c6e7f765ae19-signing-key\") pod \"service-ca-79f68b9f4f-2trt8\" (UID: \"74595786-1e43-465f-bdbe-c6e7f765ae19\") " pod="openshift-service-ca/service-ca-79f68b9f4f-2trt8" Jul 09 16:16:06.675032 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:06.674838 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6p4v\" (UniqueName: \"kubernetes.io/projected/74595786-1e43-465f-bdbe-c6e7f765ae19-kube-api-access-s6p4v\") pod \"service-ca-79f68b9f4f-2trt8\" (UID: \"74595786-1e43-465f-bdbe-c6e7f765ae19\") " pod="openshift-service-ca/service-ca-79f68b9f4f-2trt8" Jul 09 16:16:06.675127 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:06.675027 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/74595786-1e43-465f-bdbe-c6e7f765ae19-signing-cabundle\") pod \"service-ca-79f68b9f4f-2trt8\" (UID: \"74595786-1e43-465f-bdbe-c6e7f765ae19\") " pod="openshift-service-ca/service-ca-79f68b9f4f-2trt8" Jul 09 16:16:06.776054 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:06.776011 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/74595786-1e43-465f-bdbe-c6e7f765ae19-signing-cabundle\") pod \"service-ca-79f68b9f4f-2trt8\" (UID: \"74595786-1e43-465f-bdbe-c6e7f765ae19\") " pod="openshift-service-ca/service-ca-79f68b9f4f-2trt8" Jul 09 16:16:06.776277 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:06.776255 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/74595786-1e43-465f-bdbe-c6e7f765ae19-signing-key\") pod \"service-ca-79f68b9f4f-2trt8\" (UID: \"74595786-1e43-465f-bdbe-c6e7f765ae19\") " pod="openshift-service-ca/service-ca-79f68b9f4f-2trt8" Jul 09 16:16:06.776320 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:06.776310 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6p4v\" (UniqueName: \"kubernetes.io/projected/74595786-1e43-465f-bdbe-c6e7f765ae19-kube-api-access-s6p4v\") pod \"service-ca-79f68b9f4f-2trt8\" (UID: \"74595786-1e43-465f-bdbe-c6e7f765ae19\") " pod="openshift-service-ca/service-ca-79f68b9f4f-2trt8" Jul 09 16:16:06.776741 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:06.776722 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/74595786-1e43-465f-bdbe-c6e7f765ae19-signing-cabundle\") pod \"service-ca-79f68b9f4f-2trt8\" (UID: \"74595786-1e43-465f-bdbe-c6e7f765ae19\") " pod="openshift-service-ca/service-ca-79f68b9f4f-2trt8" Jul 09 16:16:06.778658 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:06.778629 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/74595786-1e43-465f-bdbe-c6e7f765ae19-signing-key\") pod \"service-ca-79f68b9f4f-2trt8\" (UID: \"74595786-1e43-465f-bdbe-c6e7f765ae19\") " pod="openshift-service-ca/service-ca-79f68b9f4f-2trt8" Jul 09 16:16:06.795425 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:06.795398 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6p4v\" (UniqueName: \"kubernetes.io/projected/74595786-1e43-465f-bdbe-c6e7f765ae19-kube-api-access-s6p4v\") pod \"service-ca-79f68b9f4f-2trt8\" (UID: \"74595786-1e43-465f-bdbe-c6e7f765ae19\") " pod="openshift-service-ca/service-ca-79f68b9f4f-2trt8" Jul 09 16:16:06.875065 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:06.875012 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-4rch5_cc6ebce3-3ee7-4b8d-a11d-1cb084a6b780/serve-healthcheck-canary/0.log" Jul 09 16:16:06.893263 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:06.893231 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-2trt8" Jul 09 16:16:07.071292 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:07.071252 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-2trt8"] Jul 09 16:16:07.075418 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:16:07.075394 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74595786_1e43_465f_bdbe_c6e7f765ae19.slice/crio-68a884339a3a6104bd026d86724d7378ac8d8e5f43ff779cc9385c4da9124b07 WatchSource:0}: Error finding container 68a884339a3a6104bd026d86724d7378ac8d8e5f43ff779cc9385c4da9124b07: Status 404 returned error can't find the container with id 68a884339a3a6104bd026d86724d7378ac8d8e5f43ff779cc9385c4da9124b07 Jul 09 16:16:07.637791 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:07.637758 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-fr9br" event={"ID":"c2cc6efd-36ec-42da-9025-c01334ab3bc0","Type":"ContainerStarted","Data":"721c744c38ef1a79ca1734440a87967a674299c6d5f8c48f9d207fe895f5f5a9"} Jul 09 16:16:07.638231 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:07.637820 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-fr9br" event={"ID":"c2cc6efd-36ec-42da-9025-c01334ab3bc0","Type":"ContainerStarted","Data":"3486d363a305948faa53194a7144ccfdc7327d846810817995c8a4972396d865"} Jul 09 16:16:07.639329 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:07.639306 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-2trt8" event={"ID":"74595786-1e43-465f-bdbe-c6e7f765ae19","Type":"ContainerStarted","Data":"61768fd66195730e5abf57e4d41e760f99f7d886ddcd48956856c2e0d2e6e570"} Jul 09 16:16:07.639432 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:07.639340 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-2trt8" event={"ID":"74595786-1e43-465f-bdbe-c6e7f765ae19","Type":"ContainerStarted","Data":"68a884339a3a6104bd026d86724d7378ac8d8e5f43ff779cc9385c4da9124b07"} Jul 09 16:16:07.660888 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:07.660821 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-fr9br" podStartSLOduration=1.933320307 podStartE2EDuration="3.660799827s" totalImagesPullingTime="1.72747952s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:16:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:16:07.658320834 +0000 UTC m=+52.851427215" watchObservedRunningTime="2026-07-09 16:16:07.660799827 +0000 UTC m=+52.853906210" Jul 09 16:16:07.684838 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:07.684797 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/031a3833-1870-44a3-abfb-9a840df32ac3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rkqp6\" (UID: \"031a3833-1870-44a3-abfb-9a840df32ac3\") " pod="openshift-insights/insights-runtime-extractor-rkqp6" Jul 09 16:16:07.685051 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:07.684925 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/031a3833-1870-44a3-abfb-9a840df32ac3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rkqp6\" (UID: \"031a3833-1870-44a3-abfb-9a840df32ac3\") " pod="openshift-insights/insights-runtime-extractor-rkqp6" Jul 09 16:16:07.685051 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:16:07.684987 2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 16:16:07.685202 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:16:07.685176 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 16:16:07.685266 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:16:07.685195 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/031a3833-1870-44a3-abfb-9a840df32ac3-kube-rbac-proxy-cm podName:031a3833-1870-44a3-abfb-9a840df32ac3 nodeName:}" failed. No retries permitted until 2026-07-09 16:16:11.685169235 +0000 UTC m=+56.878275602 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/031a3833-1870-44a3-abfb-9a840df32ac3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rkqp6" (UID: "031a3833-1870-44a3-abfb-9a840df32ac3") : configmap "kube-rbac-proxy" not found Jul 09 16:16:07.685266 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:16:07.685260 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/031a3833-1870-44a3-abfb-9a840df32ac3-insights-runtime-extractor-tls podName:031a3833-1870-44a3-abfb-9a840df32ac3 nodeName:}" failed. No retries permitted until 2026-07-09 16:16:11.685241885 +0000 UTC m=+56.878348259 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/031a3833-1870-44a3-abfb-9a840df32ac3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rkqp6" (UID: "031a3833-1870-44a3-abfb-9a840df32ac3") : secret "insights-runtime-extractor-tls" not found Jul 09 16:16:07.692022 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:07.691978 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-79f68b9f4f-2trt8" podStartSLOduration=1.6919624770000001 podStartE2EDuration="1.691962477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:16:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:16:07.691952654 +0000 UTC m=+52.885059036" watchObservedRunningTime="2026-07-09 16:16:07.691962477 +0000 UTC m=+52.885068858" Jul 09 16:16:10.616486 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:10.616442 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a0f6347a-b82d-4c94-b411-1b621bd1101c-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-pf6gv\" (UID: \"a0f6347a-b82d-4c94-b411-1b621bd1101c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-pf6gv" Jul 09 16:16:10.617075 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:10.616498 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe900560-5002-4708-98e4-237e41014fbf-service-ca-bundle\") pod \"router-default-84d7f6c55d-59vjr\" (UID: \"fe900560-5002-4708-98e4-237e41014fbf\") " pod="openshift-ingress/router-default-84d7f6c55d-59vjr" Jul 09 16:16:10.617075 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:16:10.616597 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fe900560-5002-4708-98e4-237e41014fbf-service-ca-bundle podName:fe900560-5002-4708-98e4-237e41014fbf nodeName:}" failed. No retries permitted until 2026-07-09 16:16:26.616583175 +0000 UTC m=+71.809689535 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/fe900560-5002-4708-98e4-237e41014fbf-service-ca-bundle") pod "router-default-84d7f6c55d-59vjr" (UID: "fe900560-5002-4708-98e4-237e41014fbf") : configmap references non-existent config key: service-ca.crt Jul 09 16:16:10.617075 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:10.616636 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1032414d-f6cd-4936-a52f-4f57feddba8f-registry-tls\") pod \"image-registry-558d6fd964-d9xbs\" (UID: \"1032414d-f6cd-4936-a52f-4f57feddba8f\") " pod="openshift-image-registry/image-registry-558d6fd964-d9xbs" Jul 09 16:16:10.617075 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:10.616657 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e1cf555d-4290-4caa-a5f6-eaca63d05a1d-metrics-tls\") pod \"dns-default-q8zll\" (UID: \"e1cf555d-4290-4caa-a5f6-eaca63d05a1d\") " pod="openshift-dns/dns-default-q8zll" Jul 09 16:16:10.617075 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:10.616684 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe900560-5002-4708-98e4-237e41014fbf-metrics-certs\") pod \"router-default-84d7f6c55d-59vjr\" (UID: \"fe900560-5002-4708-98e4-237e41014fbf\") " pod="openshift-ingress/router-default-84d7f6c55d-59vjr" Jul 09 16:16:10.617075 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:16:10.616745 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 16:16:10.617075 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:16:10.616771 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-558d6fd964-d9xbs: secret "image-registry-tls" not found Jul 09 16:16:10.617075 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:10.616769 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/022557ee-2bed-48ef-a717-3bf8a3c53dc5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-vdbkl\" (UID: \"022557ee-2bed-48ef-a717-3bf8a3c53dc5\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-vdbkl" Jul 09 16:16:10.617075 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:16:10.616779 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 16:16:10.617075 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:16:10.616821 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1032414d-f6cd-4936-a52f-4f57feddba8f-registry-tls podName:1032414d-f6cd-4936-a52f-4f57feddba8f nodeName:}" failed. No retries permitted until 2026-07-09 16:16:26.616805046 +0000 UTC m=+71.809911405 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1032414d-f6cd-4936-a52f-4f57feddba8f-registry-tls") pod "image-registry-558d6fd964-d9xbs" (UID: "1032414d-f6cd-4936-a52f-4f57feddba8f") : secret "image-registry-tls" not found Jul 09 16:16:10.617075 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:16:10.616840 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1cf555d-4290-4caa-a5f6-eaca63d05a1d-metrics-tls podName:e1cf555d-4290-4caa-a5f6-eaca63d05a1d nodeName:}" failed. No retries permitted until 2026-07-09 16:16:26.61682974 +0000 UTC m=+71.809936099 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e1cf555d-4290-4caa-a5f6-eaca63d05a1d-metrics-tls") pod "dns-default-q8zll" (UID: "e1cf555d-4290-4caa-a5f6-eaca63d05a1d") : secret "dns-default-metrics-tls" not found Jul 09 16:16:10.617075 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:16:10.616858 2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 09 16:16:10.617075 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:16:10.616895 2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 09 16:16:10.617075 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:16:10.616928 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe900560-5002-4708-98e4-237e41014fbf-metrics-certs podName:fe900560-5002-4708-98e4-237e41014fbf nodeName:}" failed. No retries permitted until 2026-07-09 16:16:26.616914024 +0000 UTC m=+71.810020387 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fe900560-5002-4708-98e4-237e41014fbf-metrics-certs") pod "router-default-84d7f6c55d-59vjr" (UID: "fe900560-5002-4708-98e4-237e41014fbf") : secret "router-metrics-certs-default" not found Jul 09 16:16:10.617075 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:16:10.616946 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/022557ee-2bed-48ef-a717-3bf8a3c53dc5-cluster-monitoring-operator-tls podName:022557ee-2bed-48ef-a717-3bf8a3c53dc5 nodeName:}" failed. No retries permitted until 2026-07-09 16:16:26.616936849 +0000 UTC m=+71.810043213 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/022557ee-2bed-48ef-a717-3bf8a3c53dc5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-vdbkl" (UID: "022557ee-2bed-48ef-a717-3bf8a3c53dc5") : secret "cluster-monitoring-operator-tls" not found Jul 09 16:16:10.619035 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:10.619007 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a0f6347a-b82d-4c94-b411-1b621bd1101c-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-pf6gv\" (UID: \"a0f6347a-b82d-4c94-b411-1b621bd1101c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-pf6gv" Jul 09 16:16:10.646675 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:10.646642 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-pf6gv" Jul 09 16:16:10.814303 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:10.814189 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-pf6gv"] Jul 09 16:16:11.655280 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:11.655249 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-pf6gv" event={"ID":"a0f6347a-b82d-4c94-b411-1b621bd1101c","Type":"ContainerStarted","Data":"2c93ac7d71f743f583cb556066f8b4751034ed3378bc9cce650bb48185882399"} Jul 09 16:16:11.727265 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:11.727221 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/031a3833-1870-44a3-abfb-9a840df32ac3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rkqp6\" (UID: \"031a3833-1870-44a3-abfb-9a840df32ac3\") " pod="openshift-insights/insights-runtime-extractor-rkqp6" Jul 09 16:16:11.727458 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:11.727311 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/031a3833-1870-44a3-abfb-9a840df32ac3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rkqp6\" (UID: \"031a3833-1870-44a3-abfb-9a840df32ac3\") " pod="openshift-insights/insights-runtime-extractor-rkqp6" Jul 09 16:16:11.727458 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:16:11.727394 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 16:16:11.727458 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:16:11.727440 2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 16:16:11.727600 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:16:11.727471 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/031a3833-1870-44a3-abfb-9a840df32ac3-insights-runtime-extractor-tls podName:031a3833-1870-44a3-abfb-9a840df32ac3 nodeName:}" failed. No retries permitted until 2026-07-09 16:16:19.727449273 +0000 UTC m=+64.920555635 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/031a3833-1870-44a3-abfb-9a840df32ac3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rkqp6" (UID: "031a3833-1870-44a3-abfb-9a840df32ac3") : secret "insights-runtime-extractor-tls" not found Jul 09 16:16:11.727600 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:16:11.727498 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/031a3833-1870-44a3-abfb-9a840df32ac3-kube-rbac-proxy-cm podName:031a3833-1870-44a3-abfb-9a840df32ac3 nodeName:}" failed. No retries permitted until 2026-07-09 16:16:19.727482754 +0000 UTC m=+64.920589114 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/031a3833-1870-44a3-abfb-9a840df32ac3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rkqp6" (UID: "031a3833-1870-44a3-abfb-9a840df32ac3") : configmap "kube-rbac-proxy" not found Jul 09 16:16:13.665250 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:13.665181 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-pf6gv" event={"ID":"a0f6347a-b82d-4c94-b411-1b621bd1101c","Type":"ContainerStarted","Data":"47dbdbb024449bcf21295e533bb709b9fc5c4c693159960325ffd026594a6e06"} Jul 09 16:16:13.665250 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:13.665238 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-pf6gv" event={"ID":"a0f6347a-b82d-4c94-b411-1b621bd1101c","Type":"ContainerStarted","Data":"4071927a02727edc4b47b7282d25e284174ec714af3047dc15dcc9a325122e1f"} Jul 09 16:16:13.683959 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:13.683900 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-pf6gv" podStartSLOduration=28.19366515 podStartE2EDuration="30.683862624s" totalImagesPullingTime="2.490197474s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:15:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:16:13.683149876 +0000 UTC m=+58.876256259" watchObservedRunningTime="2026-07-09 16:16:13.683862624 +0000 UTC m=+58.876969004" Jul 09 16:16:17.544517 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:17.544484 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-49sqd" Jul 09 16:16:19.808855 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:19.808814 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/031a3833-1870-44a3-abfb-9a840df32ac3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rkqp6\" (UID: \"031a3833-1870-44a3-abfb-9a840df32ac3\") " pod="openshift-insights/insights-runtime-extractor-rkqp6" Jul 09 16:16:19.809258 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:19.808902 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/031a3833-1870-44a3-abfb-9a840df32ac3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rkqp6\" (UID: \"031a3833-1870-44a3-abfb-9a840df32ac3\") " pod="openshift-insights/insights-runtime-extractor-rkqp6" Jul 09 16:16:19.809508 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:19.809482 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/031a3833-1870-44a3-abfb-9a840df32ac3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rkqp6\" (UID: \"031a3833-1870-44a3-abfb-9a840df32ac3\") " pod="openshift-insights/insights-runtime-extractor-rkqp6" Jul 09 16:16:19.811199 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:19.811182 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/031a3833-1870-44a3-abfb-9a840df32ac3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rkqp6\" (UID: \"031a3833-1870-44a3-abfb-9a840df32ac3\") " pod="openshift-insights/insights-runtime-extractor-rkqp6" Jul 09 16:16:19.924953 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:19.924922 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-4w6wt\"" Jul 09 16:16:19.932470 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:19.932441 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rkqp6" Jul 09 16:16:20.060776 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:20.060750 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rkqp6"] Jul 09 16:16:20.062780 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:16:20.062750 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod031a3833_1870_44a3_abfb_9a840df32ac3.slice/crio-98bd5e75b742aa552c3272e74e4289b551cb8600b49810ca56e207ab7aadb242 WatchSource:0}: Error finding container 98bd5e75b742aa552c3272e74e4289b551cb8600b49810ca56e207ab7aadb242: Status 404 returned error can't find the container with id 98bd5e75b742aa552c3272e74e4289b551cb8600b49810ca56e207ab7aadb242 Jul 09 16:16:20.383348 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:20.383324 2569 scope.go:117] "RemoveContainer" containerID="d0e6b4d7e71720cddd1920aa6e1bc5359483537173f8796e2414efc89fd6f9ae" Jul 09 16:16:20.693585 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:20.693474 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rkqp6" event={"ID":"031a3833-1870-44a3-abfb-9a840df32ac3","Type":"ContainerStarted","Data":"98bd5e75b742aa552c3272e74e4289b551cb8600b49810ca56e207ab7aadb242"} Jul 09 16:16:20.695535 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:20.695498 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-nzrk9_a284ea64-610c-40f7-837b-fa0975c0a290/console-operator/2.log" Jul 09 16:16:20.696523 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:20.696498 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-nzrk9_a284ea64-610c-40f7-837b-fa0975c0a290/console-operator/1.log" Jul 09 16:16:20.696659 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:20.696540 2569 generic.go:356] "Generic (PLEG): container finished" podID="a284ea64-610c-40f7-837b-fa0975c0a290" containerID="4b3ee3e67675af9b52c91e3edfd3e7675ee25d214a69cc540a6c451c8b9385a2" exitCode=255 Jul 09 16:16:20.696659 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:20.696641 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-nzrk9" event={"ID":"a284ea64-610c-40f7-837b-fa0975c0a290","Type":"ContainerDied","Data":"4b3ee3e67675af9b52c91e3edfd3e7675ee25d214a69cc540a6c451c8b9385a2"} Jul 09 16:16:20.696780 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:20.696691 2569 scope.go:117] "RemoveContainer" containerID="d0e6b4d7e71720cddd1920aa6e1bc5359483537173f8796e2414efc89fd6f9ae" Jul 09 16:16:20.697332 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:20.697313 2569 scope.go:117] "RemoveContainer" containerID="4b3ee3e67675af9b52c91e3edfd3e7675ee25d214a69cc540a6c451c8b9385a2" Jul 09 16:16:20.697581 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:16:20.697525 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7866497494-nzrk9_openshift-console-operator(a284ea64-610c-40f7-837b-fa0975c0a290)\"" pod="openshift-console-operator/console-operator-7866497494-nzrk9" podUID="a284ea64-610c-40f7-837b-fa0975c0a290" Jul 09 16:16:21.122733 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:21.122698 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f45ad4f-0e3f-432e-8328-715ec5f97a69-metrics-certs\") pod \"network-metrics-daemon-b7sxk\" (UID: \"6f45ad4f-0e3f-432e-8328-715ec5f97a69\") " pod="openshift-multus/network-metrics-daemon-b7sxk" Jul 09 16:16:21.122733 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:21.122742 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r99g2\" (UniqueName: \"kubernetes.io/projected/473a10a3-ceca-4c00-915e-a4a21820ce3e-kube-api-access-r99g2\") pod \"network-check-target-kn7sp\" (UID: \"473a10a3-ceca-4c00-915e-a4a21820ce3e\") " pod="openshift-network-diagnostics/network-check-target-kn7sp" Jul 09 16:16:21.123304 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:21.122815 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3bd017f3-bffb-4016-bdac-48c6fe9fa352-original-pull-secret\") pod \"global-pull-secret-syncer-f6wzq\" (UID: \"3bd017f3-bffb-4016-bdac-48c6fe9fa352\") " pod="kube-system/global-pull-secret-syncer-f6wzq" Jul 09 16:16:21.125688 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:21.125654 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r99g2\" (UniqueName: \"kubernetes.io/projected/473a10a3-ceca-4c00-915e-a4a21820ce3e-kube-api-access-r99g2\") pod \"network-check-target-kn7sp\" (UID: \"473a10a3-ceca-4c00-915e-a4a21820ce3e\") " pod="openshift-network-diagnostics/network-check-target-kn7sp" Jul 09 16:16:21.125818 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:21.125802 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 09 16:16:21.126827 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:21.126809 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 09 16:16:21.135343 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:21.135310 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3bd017f3-bffb-4016-bdac-48c6fe9fa352-original-pull-secret\") pod \"global-pull-secret-syncer-f6wzq\" (UID: \"3bd017f3-bffb-4016-bdac-48c6fe9fa352\") " pod="kube-system/global-pull-secret-syncer-f6wzq" Jul 09 16:16:21.135489 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:21.135347 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f45ad4f-0e3f-432e-8328-715ec5f97a69-metrics-certs\") pod \"network-metrics-daemon-b7sxk\" (UID: \"6f45ad4f-0e3f-432e-8328-715ec5f97a69\") " pod="openshift-multus/network-metrics-daemon-b7sxk" Jul 09 16:16:21.208515 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:21.208473 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-k799q\"" Jul 09 16:16:21.215376 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:21.215339 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-c54pk\"" Jul 09 16:16:21.216345 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:21.216316 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b7sxk" Jul 09 16:16:21.219545 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:21.219526 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6wzq" Jul 09 16:16:21.223769 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:21.223728 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kn7sp" Jul 09 16:16:21.383476 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:21.382157 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-b7sxk"] Jul 09 16:16:21.400304 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:21.400277 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-f6wzq"] Jul 09 16:16:21.404169 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:16:21.404132 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bd017f3_bffb_4016_bdac_48c6fe9fa352.slice/crio-0eecab71e1165b1c37c3c2906354c65baba2876c24d717dacb8ae47c8ca59caa WatchSource:0}: Error finding container 0eecab71e1165b1c37c3c2906354c65baba2876c24d717dacb8ae47c8ca59caa: Status 404 returned error can't find the container with id 0eecab71e1165b1c37c3c2906354c65baba2876c24d717dacb8ae47c8ca59caa Jul 09 16:16:21.422201 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:21.422144 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kn7sp"] Jul 09 16:16:21.425207 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:16:21.425166 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod473a10a3_ceca_4c00_915e_a4a21820ce3e.slice/crio-bb0587a77f0b3961937e1725d9b61d79191162b09959a2300fcf835b983cf74f WatchSource:0}: Error finding container bb0587a77f0b3961937e1725d9b61d79191162b09959a2300fcf835b983cf74f: Status 404 returned error can't find the container with id bb0587a77f0b3961937e1725d9b61d79191162b09959a2300fcf835b983cf74f Jul 09 16:16:21.701230 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:21.701125 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-b7sxk" event={"ID":"6f45ad4f-0e3f-432e-8328-715ec5f97a69","Type":"ContainerStarted","Data":"c8605e78f038c5788d4f7da4db24e508cd2f57c7c3f053ae6be9d1047e878d86"} Jul 09 16:16:21.702582 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:21.702552 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kn7sp" event={"ID":"473a10a3-ceca-4c00-915e-a4a21820ce3e","Type":"ContainerStarted","Data":"1153f60674d9c44db407ddd62c25695bf8d201165d7569af0fdef20661ea8648"} Jul 09 16:16:21.702747 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:21.702603 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kn7sp" event={"ID":"473a10a3-ceca-4c00-915e-a4a21820ce3e","Type":"ContainerStarted","Data":"bb0587a77f0b3961937e1725d9b61d79191162b09959a2300fcf835b983cf74f"} Jul 09 16:16:21.702747 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:21.702678 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-kn7sp" Jul 09 16:16:21.704362 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:21.704342 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-nzrk9_a284ea64-610c-40f7-837b-fa0975c0a290/console-operator/2.log" Jul 09 16:16:21.705636 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:21.705616 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-f6wzq" event={"ID":"3bd017f3-bffb-4016-bdac-48c6fe9fa352","Type":"ContainerStarted","Data":"0eecab71e1165b1c37c3c2906354c65baba2876c24d717dacb8ae47c8ca59caa"} Jul 09 16:16:21.718200 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:21.718145 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-kn7sp" podStartSLOduration=65.718126783 podStartE2EDuration="1m5.718126783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:15:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:16:21.718065129 +0000 UTC m=+66.911171510" watchObservedRunningTime="2026-07-09 16:16:21.718126783 +0000 UTC m=+66.911233160" Jul 09 16:16:25.124670 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:25.124645 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-nzrk9" Jul 09 16:16:25.125261 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:25.124703 2569 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7866497494-nzrk9" Jul 09 16:16:25.125261 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:25.125103 2569 scope.go:117] "RemoveContainer" containerID="4b3ee3e67675af9b52c91e3edfd3e7675ee25d214a69cc540a6c451c8b9385a2" Jul 09 16:16:25.125380 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:16:25.125317 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7866497494-nzrk9_openshift-console-operator(a284ea64-610c-40f7-837b-fa0975c0a290)\"" pod="openshift-console-operator/console-operator-7866497494-nzrk9" podUID="a284ea64-610c-40f7-837b-fa0975c0a290" Jul 09 16:16:25.724433 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:25.724390 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-b7sxk" event={"ID":"6f45ad4f-0e3f-432e-8328-715ec5f97a69","Type":"ContainerStarted","Data":"befcef5093538ee7a91686e2cead83ddbac292c857413dedbefb7b925c59d11b"} Jul 09 16:16:25.724638 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:25.724499 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-b7sxk" event={"ID":"6f45ad4f-0e3f-432e-8328-715ec5f97a69","Type":"ContainerStarted","Data":"519b644b29dcca0374f03a182c71f1596e824c306db3b0bd676c9710eb76bfd7"} Jul 09 16:16:25.726344 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:25.726314 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rkqp6" event={"ID":"031a3833-1870-44a3-abfb-9a840df32ac3","Type":"ContainerStarted","Data":"ee13f3a8b7cd0eb06e29a9d0c62c6ec17ea838d547f9c525827560ed89a281d9"} Jul 09 16:16:25.742700 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:25.742643 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-b7sxk" podStartSLOduration=66.146064836 podStartE2EDuration="1m9.742621929s" totalImagesPullingTime="3.596557093s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:15:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:16:25.741523695 +0000 UTC m=+70.934630079" watchObservedRunningTime="2026-07-09 16:16:25.742621929 +0000 UTC m=+70.935728311" Jul 09 16:16:26.677565 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:26.677535 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e1cf555d-4290-4caa-a5f6-eaca63d05a1d-metrics-tls\") pod \"dns-default-q8zll\" (UID: \"e1cf555d-4290-4caa-a5f6-eaca63d05a1d\") " pod="openshift-dns/dns-default-q8zll" Jul 09 16:16:26.677885 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:26.677574 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe900560-5002-4708-98e4-237e41014fbf-metrics-certs\") pod \"router-default-84d7f6c55d-59vjr\" (UID: \"fe900560-5002-4708-98e4-237e41014fbf\") " pod="openshift-ingress/router-default-84d7f6c55d-59vjr" Jul 09 16:16:26.677885 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:26.677620 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/022557ee-2bed-48ef-a717-3bf8a3c53dc5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-vdbkl\" (UID: \"022557ee-2bed-48ef-a717-3bf8a3c53dc5\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-vdbkl" Jul 09 16:16:26.677885 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:26.677653 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe900560-5002-4708-98e4-237e41014fbf-service-ca-bundle\") pod \"router-default-84d7f6c55d-59vjr\" (UID: \"fe900560-5002-4708-98e4-237e41014fbf\") " pod="openshift-ingress/router-default-84d7f6c55d-59vjr" Jul 09 16:16:26.678043 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:26.677918 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1032414d-f6cd-4936-a52f-4f57feddba8f-registry-tls\") pod \"image-registry-558d6fd964-d9xbs\" (UID: \"1032414d-f6cd-4936-a52f-4f57feddba8f\") " pod="openshift-image-registry/image-registry-558d6fd964-d9xbs" Jul 09 16:16:26.678448 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:26.678420 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe900560-5002-4708-98e4-237e41014fbf-service-ca-bundle\") pod \"router-default-84d7f6c55d-59vjr\" (UID: \"fe900560-5002-4708-98e4-237e41014fbf\") " pod="openshift-ingress/router-default-84d7f6c55d-59vjr" Jul 09 16:16:26.680181 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:26.680149 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe900560-5002-4708-98e4-237e41014fbf-metrics-certs\") pod \"router-default-84d7f6c55d-59vjr\" (UID: \"fe900560-5002-4708-98e4-237e41014fbf\") " pod="openshift-ingress/router-default-84d7f6c55d-59vjr" Jul 09 16:16:26.680288 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:26.680149 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/022557ee-2bed-48ef-a717-3bf8a3c53dc5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-vdbkl\" (UID: \"022557ee-2bed-48ef-a717-3bf8a3c53dc5\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-vdbkl" Jul 09 16:16:26.680288 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:26.680194 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1032414d-f6cd-4936-a52f-4f57feddba8f-registry-tls\") pod \"image-registry-558d6fd964-d9xbs\" (UID: \"1032414d-f6cd-4936-a52f-4f57feddba8f\") " pod="openshift-image-registry/image-registry-558d6fd964-d9xbs" Jul 09 16:16:26.680421 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:26.680397 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e1cf555d-4290-4caa-a5f6-eaca63d05a1d-metrics-tls\") pod \"dns-default-q8zll\" (UID: \"e1cf555d-4290-4caa-a5f6-eaca63d05a1d\") " pod="openshift-dns/dns-default-q8zll" Jul 09 16:16:26.804720 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:26.804694 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6v4p5\"" Jul 09 16:16:26.811776 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:26.811527 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-558d6fd964-d9xbs" Jul 09 16:16:26.822933 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:26.822901 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-pf9kt\"" Jul 09 16:16:26.830754 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:26.830723 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-vdbkl" Jul 09 16:16:26.835792 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:26.835771 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-p7mfm\"" Jul 09 16:16:26.843215 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:26.843195 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-84d7f6c55d-59vjr" Jul 09 16:16:26.866202 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:26.866024 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6w5fn\"" Jul 09 16:16:26.875235 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:26.874976 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-q8zll" Jul 09 16:16:26.876025 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:26.875936 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-557897bb-cds89"] Jul 09 16:16:26.879534 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:26.879496 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6544fc656d-lqwlr"] Jul 09 16:16:26.879654 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:26.879597 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-557897bb-cds89" Jul 09 16:16:26.882965 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:26.882945 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 09 16:16:26.886499 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:26.883472 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68486b77b8-kd7p4"] Jul 09 16:16:26.886499 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:26.884964 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 09 16:16:26.886499 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:26.885229 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 09 16:16:26.886499 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:26.885416 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6544fc656d-lqwlr" Jul 09 16:16:26.886499 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:26.885501 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-rngzn\"" Jul 09 16:16:26.886499 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:26.885789 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 09 16:16:26.889251 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:26.889062 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 09 16:16:26.897121 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:26.893285 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-557897bb-cds89"] Jul 09 16:16:26.897121 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:26.893316 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6544fc656d-lqwlr"] Jul 09 16:16:26.897121 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:26.893426 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68486b77b8-kd7p4" Jul 09 16:16:26.897121 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:26.893816 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68486b77b8-kd7p4"] Jul 09 16:16:26.897121 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:26.896785 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 09 16:16:26.897121 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:26.897078 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 09 16:16:26.897473 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:26.897273 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 09 16:16:26.899486 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:26.899214 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 09 16:16:26.981152 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:26.980274 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtm66\" (UniqueName: \"kubernetes.io/projected/93836e42-c8eb-4b8e-b980-9c0ee5d46492-kube-api-access-vtm66\") pod \"managed-serviceaccount-addon-agent-557897bb-cds89\" (UID: \"93836e42-c8eb-4b8e-b980-9c0ee5d46492\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-557897bb-cds89" Jul 09 16:16:26.981152 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:26.980332 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/869b9f48-371a-4d51-9737-08235e444a84-tmp\") pod \"klusterlet-addon-workmgr-6544fc656d-lqwlr\" (UID: \"869b9f48-371a-4d51-9737-08235e444a84\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6544fc656d-lqwlr" Jul 09 16:16:26.981152 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:26.980365 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/93836e42-c8eb-4b8e-b980-9c0ee5d46492-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-557897bb-cds89\" (UID: \"93836e42-c8eb-4b8e-b980-9c0ee5d46492\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-557897bb-cds89" Jul 09 16:16:26.981152 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:26.980388 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/869b9f48-371a-4d51-9737-08235e444a84-klusterlet-config\") pod \"klusterlet-addon-workmgr-6544fc656d-lqwlr\" (UID: \"869b9f48-371a-4d51-9737-08235e444a84\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6544fc656d-lqwlr" Jul 09 16:16:26.981152 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:26.980451 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74p7c\" (UniqueName: \"kubernetes.io/projected/869b9f48-371a-4d51-9737-08235e444a84-kube-api-access-74p7c\") pod \"klusterlet-addon-workmgr-6544fc656d-lqwlr\" (UID: \"869b9f48-371a-4d51-9737-08235e444a84\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6544fc656d-lqwlr" Jul 09 16:16:26.986849 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:26.985346 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-2hrtq"] Jul 09 16:16:27.003910 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:27.003788 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-2hrtq"] Jul 09 16:16:27.004614 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:27.004132 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-2hrtq" Jul 09 16:16:27.016833 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:27.016287 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 09 16:16:27.016833 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:27.016582 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 09 16:16:27.021857 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:27.020470 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-jg2qp\"" Jul 09 16:16:27.082182 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:27.080934 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/d5ca1fe9-b3d3-45e9-9396-9515ac281088-hub\") pod \"cluster-proxy-proxy-agent-68486b77b8-kd7p4\" (UID: \"d5ca1fe9-b3d3-45e9-9396-9515ac281088\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68486b77b8-kd7p4" Jul 09 16:16:27.082182 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:27.081007 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74p7c\" (UniqueName: \"kubernetes.io/projected/869b9f48-371a-4d51-9737-08235e444a84-kube-api-access-74p7c\") pod \"klusterlet-addon-workmgr-6544fc656d-lqwlr\" (UID: \"869b9f48-371a-4d51-9737-08235e444a84\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6544fc656d-lqwlr" Jul 09 16:16:27.082182 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:27.081045 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbxx6\" (UniqueName: \"kubernetes.io/projected/d5ca1fe9-b3d3-45e9-9396-9515ac281088-kube-api-access-bbxx6\") pod \"cluster-proxy-proxy-agent-68486b77b8-kd7p4\" (UID: \"d5ca1fe9-b3d3-45e9-9396-9515ac281088\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68486b77b8-kd7p4" Jul 09 16:16:27.082182 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:27.081107 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d5ca1fe9-b3d3-45e9-9396-9515ac281088-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-68486b77b8-kd7p4\" (UID: \"d5ca1fe9-b3d3-45e9-9396-9515ac281088\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68486b77b8-kd7p4" Jul 09 16:16:27.082182 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:27.081138 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/d5ca1fe9-b3d3-45e9-9396-9515ac281088-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-68486b77b8-kd7p4\" (UID: \"d5ca1fe9-b3d3-45e9-9396-9515ac281088\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68486b77b8-kd7p4" Jul 09 16:16:27.082182 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:27.081174 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtm66\" (UniqueName: \"kubernetes.io/projected/93836e42-c8eb-4b8e-b980-9c0ee5d46492-kube-api-access-vtm66\") pod \"managed-serviceaccount-addon-agent-557897bb-cds89\" (UID: \"93836e42-c8eb-4b8e-b980-9c0ee5d46492\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-557897bb-cds89" Jul 09 16:16:27.082182 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:27.081223 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/869b9f48-371a-4d51-9737-08235e444a84-tmp\") pod \"klusterlet-addon-workmgr-6544fc656d-lqwlr\" (UID: \"869b9f48-371a-4d51-9737-08235e444a84\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6544fc656d-lqwlr" Jul 09 16:16:27.082182 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:27.081254 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/93836e42-c8eb-4b8e-b980-9c0ee5d46492-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-557897bb-cds89\" (UID: \"93836e42-c8eb-4b8e-b980-9c0ee5d46492\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-557897bb-cds89" Jul 09 16:16:27.082182 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:27.081276 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/869b9f48-371a-4d51-9737-08235e444a84-klusterlet-config\") pod \"klusterlet-addon-workmgr-6544fc656d-lqwlr\" (UID: \"869b9f48-371a-4d51-9737-08235e444a84\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6544fc656d-lqwlr" Jul 09 16:16:27.082182 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:27.081299 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/d5ca1fe9-b3d3-45e9-9396-9515ac281088-ca\") pod \"cluster-proxy-proxy-agent-68486b77b8-kd7p4\" (UID: \"d5ca1fe9-b3d3-45e9-9396-9515ac281088\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68486b77b8-kd7p4" Jul 09 16:16:27.082182 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:27.081345 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5ca1fe9-b3d3-45e9-9396-9515ac281088-additional-service-ca\") pod \"cluster-proxy-proxy-agent-68486b77b8-kd7p4\" (UID: \"d5ca1fe9-b3d3-45e9-9396-9515ac281088\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68486b77b8-kd7p4" Jul 09 16:16:27.082931 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:27.082528 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/869b9f48-371a-4d51-9737-08235e444a84-tmp\") pod \"klusterlet-addon-workmgr-6544fc656d-lqwlr\" (UID: \"869b9f48-371a-4d51-9737-08235e444a84\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6544fc656d-lqwlr" Jul 09 16:16:27.085831 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:27.085706 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/93836e42-c8eb-4b8e-b980-9c0ee5d46492-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-557897bb-cds89\" (UID: \"93836e42-c8eb-4b8e-b980-9c0ee5d46492\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-557897bb-cds89" Jul 09 16:16:27.085831 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:27.085706 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/869b9f48-371a-4d51-9737-08235e444a84-klusterlet-config\") pod \"klusterlet-addon-workmgr-6544fc656d-lqwlr\" (UID: \"869b9f48-371a-4d51-9737-08235e444a84\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6544fc656d-lqwlr" Jul 09 16:16:27.098356 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:27.098305 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74p7c\" (UniqueName: \"kubernetes.io/projected/869b9f48-371a-4d51-9737-08235e444a84-kube-api-access-74p7c\") pod \"klusterlet-addon-workmgr-6544fc656d-lqwlr\" (UID: \"869b9f48-371a-4d51-9737-08235e444a84\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6544fc656d-lqwlr" Jul 09 16:16:27.111050 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:27.110481 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-vdbkl"] Jul 09 16:16:27.113637 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:27.113568 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-558d6fd964-d9xbs"] Jul 09 16:16:27.116050 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:16:27.116004 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod022557ee_2bed_48ef_a717_3bf8a3c53dc5.slice/crio-68c5828f41f3518853bf3db2b0e81b4b92ee60f13fe6fd66a95575eed5bcdfa2 WatchSource:0}: Error finding container 68c5828f41f3518853bf3db2b0e81b4b92ee60f13fe6fd66a95575eed5bcdfa2: Status 404 returned error can't find the container with id 68c5828f41f3518853bf3db2b0e81b4b92ee60f13fe6fd66a95575eed5bcdfa2 Jul 09 16:16:27.117316 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:27.117294 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtm66\" (UniqueName: \"kubernetes.io/projected/93836e42-c8eb-4b8e-b980-9c0ee5d46492-kube-api-access-vtm66\") pod \"managed-serviceaccount-addon-agent-557897bb-cds89\" (UID: \"93836e42-c8eb-4b8e-b980-9c0ee5d46492\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-557897bb-cds89" Jul 09 16:16:27.118223 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:16:27.118150 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1032414d_f6cd_4936_a52f_4f57feddba8f.slice/crio-41f13333aee138496e3ea6f8083c2460effbf53d79c59fcf9aa5d5a1b9f64c9e WatchSource:0}: Error finding container 41f13333aee138496e3ea6f8083c2460effbf53d79c59fcf9aa5d5a1b9f64c9e: Status 404 returned error can't find the container with id 41f13333aee138496e3ea6f8083c2460effbf53d79c59fcf9aa5d5a1b9f64c9e Jul 09 16:16:27.143715 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:27.143695 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-84d7f6c55d-59vjr"] Jul 09 16:16:27.151938 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:16:27.151905 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe900560_5002_4708_98e4_237e41014fbf.slice/crio-6f58b11813233a430ca4e2199ede95a653f429c576b044cf27c91f2477681d0d WatchSource:0}: Error finding container 6f58b11813233a430ca4e2199ede95a653f429c576b044cf27c91f2477681d0d: Status 404 returned error can't find the container with id 6f58b11813233a430ca4e2199ede95a653f429c576b044cf27c91f2477681d0d Jul 09 16:16:27.167156 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:27.166373 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-q8zll"] Jul 09 16:16:27.171709 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:16:27.171684 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1cf555d_4290_4caa_a5f6_eaca63d05a1d.slice/crio-c92600c0795bf46a63d352f42a846d1198f831995aefba94a5252e7d0fa80fee WatchSource:0}: Error finding container c92600c0795bf46a63d352f42a846d1198f831995aefba94a5252e7d0fa80fee: Status 404 returned error can't find the container with id c92600c0795bf46a63d352f42a846d1198f831995aefba94a5252e7d0fa80fee Jul 09 16:16:27.182043 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:27.182016 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5ca1fe9-b3d3-45e9-9396-9515ac281088-additional-service-ca\") pod \"cluster-proxy-proxy-agent-68486b77b8-kd7p4\" (UID: \"d5ca1fe9-b3d3-45e9-9396-9515ac281088\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68486b77b8-kd7p4" Jul 09 16:16:27.182147 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:27.182053 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/d5ca1fe9-b3d3-45e9-9396-9515ac281088-hub\") pod \"cluster-proxy-proxy-agent-68486b77b8-kd7p4\" (UID: \"d5ca1fe9-b3d3-45e9-9396-9515ac281088\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68486b77b8-kd7p4" Jul 09 16:16:27.182147 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:27.182081 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f8eb4809-6734-4b9d-b89f-396679014b4a-nginx-conf\") pod \"networking-console-plugin-6cf856758c-2hrtq\" (UID: \"f8eb4809-6734-4b9d-b89f-396679014b4a\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-2hrtq" Jul 09 16:16:27.182147 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:27.182101 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbxx6\" (UniqueName: \"kubernetes.io/projected/d5ca1fe9-b3d3-45e9-9396-9515ac281088-kube-api-access-bbxx6\") pod \"cluster-proxy-proxy-agent-68486b77b8-kd7p4\" (UID: \"d5ca1fe9-b3d3-45e9-9396-9515ac281088\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68486b77b8-kd7p4" Jul 09 16:16:27.182266 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:27.182177 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d5ca1fe9-b3d3-45e9-9396-9515ac281088-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-68486b77b8-kd7p4\" (UID: \"d5ca1fe9-b3d3-45e9-9396-9515ac281088\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68486b77b8-kd7p4" Jul 09 16:16:27.182342 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:27.182322 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/d5ca1fe9-b3d3-45e9-9396-9515ac281088-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-68486b77b8-kd7p4\" (UID: \"d5ca1fe9-b3d3-45e9-9396-9515ac281088\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68486b77b8-kd7p4" Jul 09 16:16:27.182392 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:27.182368 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f8eb4809-6734-4b9d-b89f-396679014b4a-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-2hrtq\" (UID: \"f8eb4809-6734-4b9d-b89f-396679014b4a\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-2hrtq" Jul 09 16:16:27.182457 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:27.182438 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/d5ca1fe9-b3d3-45e9-9396-9515ac281088-ca\") pod \"cluster-proxy-proxy-agent-68486b77b8-kd7p4\" (UID: \"d5ca1fe9-b3d3-45e9-9396-9515ac281088\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68486b77b8-kd7p4" Jul 09 16:16:27.182797 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:27.182746 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5ca1fe9-b3d3-45e9-9396-9515ac281088-additional-service-ca\") pod \"cluster-proxy-proxy-agent-68486b77b8-kd7p4\" (UID: \"d5ca1fe9-b3d3-45e9-9396-9515ac281088\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68486b77b8-kd7p4" Jul 09 16:16:27.185338 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:27.185314 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d5ca1fe9-b3d3-45e9-9396-9515ac281088-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-68486b77b8-kd7p4\" (UID: \"d5ca1fe9-b3d3-45e9-9396-9515ac281088\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68486b77b8-kd7p4" Jul 09 16:16:27.185446 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:27.185427 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/d5ca1fe9-b3d3-45e9-9396-9515ac281088-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-68486b77b8-kd7p4\" (UID: \"d5ca1fe9-b3d3-45e9-9396-9515ac281088\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68486b77b8-kd7p4" Jul 09 16:16:27.185483 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:27.185437 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/d5ca1fe9-b3d3-45e9-9396-9515ac281088-hub\") pod \"cluster-proxy-proxy-agent-68486b77b8-kd7p4\" (UID: \"d5ca1fe9-b3d3-45e9-9396-9515ac281088\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68486b77b8-kd7p4" Jul 09 16:16:27.185965 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:27.185944 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/d5ca1fe9-b3d3-45e9-9396-9515ac281088-ca\") pod \"cluster-proxy-proxy-agent-68486b77b8-kd7p4\" (UID: \"d5ca1fe9-b3d3-45e9-9396-9515ac281088\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68486b77b8-kd7p4" Jul 09 16:16:27.201660 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:27.201629 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbxx6\" (UniqueName: \"kubernetes.io/projected/d5ca1fe9-b3d3-45e9-9396-9515ac281088-kube-api-access-bbxx6\") pod \"cluster-proxy-proxy-agent-68486b77b8-kd7p4\" (UID: \"d5ca1fe9-b3d3-45e9-9396-9515ac281088\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68486b77b8-kd7p4" Jul 09 16:16:27.207497 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:27.207470 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-557897bb-cds89" Jul 09 16:16:27.219204 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:27.219175 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6544fc656d-lqwlr" Jul 09 16:16:27.247760 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:27.247725 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68486b77b8-kd7p4" Jul 09 16:16:27.284110 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:27.283492 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f8eb4809-6734-4b9d-b89f-396679014b4a-nginx-conf\") pod \"networking-console-plugin-6cf856758c-2hrtq\" (UID: \"f8eb4809-6734-4b9d-b89f-396679014b4a\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-2hrtq" Jul 09 16:16:27.284110 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:27.283586 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f8eb4809-6734-4b9d-b89f-396679014b4a-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-2hrtq\" (UID: \"f8eb4809-6734-4b9d-b89f-396679014b4a\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-2hrtq" Jul 09 16:16:27.284688 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:27.284638 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f8eb4809-6734-4b9d-b89f-396679014b4a-nginx-conf\") pod \"networking-console-plugin-6cf856758c-2hrtq\" (UID: \"f8eb4809-6734-4b9d-b89f-396679014b4a\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-2hrtq" Jul 09 16:16:27.287443 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:27.287326 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f8eb4809-6734-4b9d-b89f-396679014b4a-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-2hrtq\" (UID: \"f8eb4809-6734-4b9d-b89f-396679014b4a\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-2hrtq" Jul 09 16:16:27.356068 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:27.355789 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-2hrtq" Jul 09 16:16:27.380535 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:27.380441 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-557897bb-cds89"] Jul 09 16:16:27.409757 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:27.408911 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6544fc656d-lqwlr"] Jul 09 16:16:27.411217 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:16:27.411186 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod869b9f48_371a_4d51_9737_08235e444a84.slice/crio-1c2fa8cced59e2a77b3c3e75e80e9af36aeb65707c243e5fb46bbe6dca54edf4 WatchSource:0}: Error finding container 1c2fa8cced59e2a77b3c3e75e80e9af36aeb65707c243e5fb46bbe6dca54edf4: Status 404 returned error can't find the container with id 1c2fa8cced59e2a77b3c3e75e80e9af36aeb65707c243e5fb46bbe6dca54edf4 Jul 09 16:16:27.441935 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:27.441906 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68486b77b8-kd7p4"] Jul 09 16:16:27.445758 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:16:27.445720 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd5ca1fe9_b3d3_45e9_9396_9515ac281088.slice/crio-57f82f1c216a773b780798201ac20c083c51ef42eb6b516e6fce3856d1b17f7c WatchSource:0}: Error finding container 57f82f1c216a773b780798201ac20c083c51ef42eb6b516e6fce3856d1b17f7c: Status 404 returned error can't find the container with id 57f82f1c216a773b780798201ac20c083c51ef42eb6b516e6fce3856d1b17f7c Jul 09 16:16:27.524142 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:27.524108 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-2hrtq"] Jul 09 16:16:27.527814 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:16:27.527786 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8eb4809_6734_4b9d_b89f_396679014b4a.slice/crio-102b94c4d1d49a02d5c143cdf0fab3d6d9df1c091ac22b852dac172520c645aa WatchSource:0}: Error finding container 102b94c4d1d49a02d5c143cdf0fab3d6d9df1c091ac22b852dac172520c645aa: Status 404 returned error can't find the container with id 102b94c4d1d49a02d5c143cdf0fab3d6d9df1c091ac22b852dac172520c645aa Jul 09 16:16:27.737968 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:27.737864 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6544fc656d-lqwlr" event={"ID":"869b9f48-371a-4d51-9737-08235e444a84","Type":"ContainerStarted","Data":"1c2fa8cced59e2a77b3c3e75e80e9af36aeb65707c243e5fb46bbe6dca54edf4"} Jul 09 16:16:27.739499 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:27.739462 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-vdbkl" event={"ID":"022557ee-2bed-48ef-a717-3bf8a3c53dc5","Type":"ContainerStarted","Data":"68c5828f41f3518853bf3db2b0e81b4b92ee60f13fe6fd66a95575eed5bcdfa2"} Jul 09 16:16:27.740658 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:27.740605 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-557897bb-cds89" event={"ID":"93836e42-c8eb-4b8e-b980-9c0ee5d46492","Type":"ContainerStarted","Data":"26550d0ff4efa72230d5de1d6bc5ff868fafa5db0378ca178399a58e250b7a70"} Jul 09 16:16:27.742280 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:27.742238 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-558d6fd964-d9xbs" event={"ID":"1032414d-f6cd-4936-a52f-4f57feddba8f","Type":"ContainerStarted","Data":"6fc391579119b7eca8c65444ebe216dc958f05204a2bbf5ad73eab9b62462e72"} Jul 09 16:16:27.742367 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:27.742280 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-558d6fd964-d9xbs" event={"ID":"1032414d-f6cd-4936-a52f-4f57feddba8f","Type":"ContainerStarted","Data":"41f13333aee138496e3ea6f8083c2460effbf53d79c59fcf9aa5d5a1b9f64c9e"} Jul 09 16:16:27.742590 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:27.742548 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-558d6fd964-d9xbs" Jul 09 16:16:27.744741 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:27.744692 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-2hrtq" event={"ID":"f8eb4809-6734-4b9d-b89f-396679014b4a","Type":"ContainerStarted","Data":"102b94c4d1d49a02d5c143cdf0fab3d6d9df1c091ac22b852dac172520c645aa"} Jul 09 16:16:27.746690 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:27.746652 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-84d7f6c55d-59vjr" event={"ID":"fe900560-5002-4708-98e4-237e41014fbf","Type":"ContainerStarted","Data":"8cfe9ad7b2b6203ad93daf065e4036300527dc02c2f8b611add97a4ee537d5c9"} Jul 09 16:16:27.746791 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:27.746693 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-84d7f6c55d-59vjr" event={"ID":"fe900560-5002-4708-98e4-237e41014fbf","Type":"ContainerStarted","Data":"6f58b11813233a430ca4e2199ede95a653f429c576b044cf27c91f2477681d0d"} Jul 09 16:16:27.749128 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:27.749100 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rkqp6" event={"ID":"031a3833-1870-44a3-abfb-9a840df32ac3","Type":"ContainerStarted","Data":"262213facb61b54bcb724771afc7000fcbb3f4861bef633489cc24ff760c70ef"} Jul 09 16:16:27.750320 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:27.750299 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68486b77b8-kd7p4" event={"ID":"d5ca1fe9-b3d3-45e9-9396-9515ac281088","Type":"ContainerStarted","Data":"57f82f1c216a773b780798201ac20c083c51ef42eb6b516e6fce3856d1b17f7c"} Jul 09 16:16:27.751668 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:27.751619 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-q8zll" event={"ID":"e1cf555d-4290-4caa-a5f6-eaca63d05a1d","Type":"ContainerStarted","Data":"c92600c0795bf46a63d352f42a846d1198f831995aefba94a5252e7d0fa80fee"} Jul 09 16:16:27.753434 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:27.753358 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-f6wzq" event={"ID":"3bd017f3-bffb-4016-bdac-48c6fe9fa352","Type":"ContainerStarted","Data":"3401b17dc2f2175fab14eabf9018b9ff40a0a17a1df3428ef3183d9e18ba4349"} Jul 09 16:16:27.766245 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:27.766203 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-558d6fd964-d9xbs" podStartSLOduration=59.766188965 podStartE2EDuration="59.766188965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:15:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:16:27.764808358 +0000 UTC m=+72.957914741" watchObservedRunningTime="2026-07-09 16:16:27.766188965 +0000 UTC m=+72.959295346" Jul 09 16:16:27.782313 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:27.782256 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-f6wzq" podStartSLOduration=66.522594915 podStartE2EDuration="1m11.782240727s" totalImagesPullingTime="5.259645812s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 16:15:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:16:27.78029724 +0000 UTC m=+72.973403621" watchObservedRunningTime="2026-07-09 16:16:27.782240727 +0000 UTC m=+72.975347110" Jul 09 16:16:27.805288 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:27.805226 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-84d7f6c55d-59vjr" podStartSLOduration=44.805210217 podStartE2EDuration="44.805210217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:15:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:16:27.803442056 +0000 UTC m=+72.996548462" watchObservedRunningTime="2026-07-09 16:16:27.805210217 +0000 UTC m=+72.998316599" Jul 09 16:16:27.843998 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:27.843915 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-84d7f6c55d-59vjr" Jul 09 16:16:27.847240 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:27.847184 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-84d7f6c55d-59vjr" Jul 09 16:16:28.766454 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:28.766388 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-84d7f6c55d-59vjr" Jul 09 16:16:28.767686 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:28.767166 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-84d7f6c55d-59vjr" Jul 09 16:16:37.810355 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:37.810322 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-2hrtq" event={"ID":"f8eb4809-6734-4b9d-b89f-396679014b4a","Type":"ContainerStarted","Data":"c7086b0505fe18c9373c8fed500c2466780f108d5f6aaca460a2bb8f95ae4fa1"} Jul 09 16:16:37.812748 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:37.812720 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rkqp6" event={"ID":"031a3833-1870-44a3-abfb-9a840df32ac3","Type":"ContainerStarted","Data":"9ffb85326d6da09e406bb14029f0eeb16b0d53472b504417d9bf9730c1f0bf68"} Jul 09 16:16:37.815131 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:37.815104 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68486b77b8-kd7p4" event={"ID":"d5ca1fe9-b3d3-45e9-9396-9515ac281088","Type":"ContainerStarted","Data":"dbf1e3b74b4e0d758e8643f8f5a857a646cf712f7979dc33a2e8162c3f96322a"} Jul 09 16:16:37.815245 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:37.815147 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68486b77b8-kd7p4" event={"ID":"d5ca1fe9-b3d3-45e9-9396-9515ac281088","Type":"ContainerStarted","Data":"59914c01307588655433cba2094cbab55b3233a2ee9c8534529de874a1a8dfce"} Jul 09 16:16:37.815245 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:37.815163 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68486b77b8-kd7p4" event={"ID":"d5ca1fe9-b3d3-45e9-9396-9515ac281088","Type":"ContainerStarted","Data":"866caafc45be095b46bb12f24736d0d54a1250819974b59f84bced46db4c675c"} Jul 09 16:16:37.816854 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:37.816833 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-q8zll" event={"ID":"e1cf555d-4290-4caa-a5f6-eaca63d05a1d","Type":"ContainerStarted","Data":"427e868426bd5e219697b35e72c869ff2236d26225e227f01ef665c7c46b10b2"} Jul 09 16:16:37.816974 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:37.816869 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-q8zll" event={"ID":"e1cf555d-4290-4caa-a5f6-eaca63d05a1d","Type":"ContainerStarted","Data":"062e6772a76137a26d3d6995a7b034c28fdbc130988e9702a38302fa8327be66"} Jul 09 16:16:37.816974 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:37.816916 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-q8zll" Jul 09 16:16:37.818482 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:37.818463 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6544fc656d-lqwlr" event={"ID":"869b9f48-371a-4d51-9737-08235e444a84","Type":"ContainerStarted","Data":"8e764ff6c79b5c158052eada79efc492208eb0c7fd2a1de21ddc3d4784645ded"} Jul 09 16:16:37.818576 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:37.818503 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6544fc656d-lqwlr" Jul 09 16:16:37.820109 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:37.820083 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-vdbkl" event={"ID":"022557ee-2bed-48ef-a717-3bf8a3c53dc5","Type":"ContainerStarted","Data":"0589215de6f04bfab753fb1e15b3f30ecbef794b1c99d47e7fdb4b6ac15f7583"} Jul 09 16:16:37.820484 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:37.820465 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6544fc656d-lqwlr" Jul 09 16:16:37.821668 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:37.821649 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-557897bb-cds89" event={"ID":"93836e42-c8eb-4b8e-b980-9c0ee5d46492","Type":"ContainerStarted","Data":"acbd8d89a9743f66185a483a75112219de8e0b4f3e06f96be28630354df47ad9"} Jul 09 16:16:37.827440 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:37.827400 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6cf856758c-2hrtq" podStartSLOduration=2.360630853 podStartE2EDuration="11.827388444s" totalImagesPullingTime="9.466757591s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:16:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:16:37.825647892 +0000 UTC m=+83.018754277" watchObservedRunningTime="2026-07-09 16:16:37.827388444 +0000 UTC m=+83.020494825" Jul 09 16:16:37.844440 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:37.844396 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-q8zll" podStartSLOduration=34.013230407 podStartE2EDuration="43.844378355s" totalImagesPullingTime="9.831147948s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:15:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:16:37.843319493 +0000 UTC m=+83.036425876" watchObservedRunningTime="2026-07-09 16:16:37.844378355 +0000 UTC m=+83.037484739" Jul 09 16:16:37.863333 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:37.863271 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68486b77b8-kd7p4" podStartSLOduration=2.283517616 podStartE2EDuration="11.863247653s" totalImagesPullingTime="9.579730037s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:16:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:16:37.86246646 +0000 UTC m=+83.055572867" watchObservedRunningTime="2026-07-09 16:16:37.863247653 +0000 UTC m=+83.056354037" Jul 09 16:16:37.882075 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:37.882013 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-557897bb-cds89" podStartSLOduration=2.255684757 podStartE2EDuration="11.8819902s" totalImagesPullingTime="9.626305443s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:16:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:16:37.879669939 +0000 UTC m=+83.072776302" watchObservedRunningTime="2026-07-09 16:16:37.8819902 +0000 UTC m=+83.075096582" Jul 09 16:16:37.933351 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:37.933295 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6544fc656d-lqwlr" podStartSLOduration=2.316673669 podStartE2EDuration="11.933279028s" totalImagesPullingTime="9.616605359s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:16:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:16:37.932109371 +0000 UTC m=+83.125215752" watchObservedRunningTime="2026-07-09 16:16:37.933279028 +0000 UTC m=+83.126385405" Jul 09 16:16:37.934615 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:37.934576 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-rkqp6" podStartSLOduration=18.218536666 podStartE2EDuration="34.934561725s" totalImagesPullingTime="16.716025059s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 16:16:03 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:16:37.908495785 +0000 UTC m=+83.101602166" watchObservedRunningTime="2026-07-09 16:16:37.934561725 +0000 UTC m=+83.127668102" Jul 09 16:16:37.955863 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:37.955809 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-vdbkl" podStartSLOduration=45.112611566 podStartE2EDuration="54.955791563s" totalImagesPullingTime="9.843179997s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:15:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:16:37.954837526 +0000 UTC m=+83.147943907" watchObservedRunningTime="2026-07-09 16:16:37.955791563 +0000 UTC m=+83.148897952" Jul 09 16:16:40.384148 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:40.384108 2569 scope.go:117] "RemoveContainer" containerID="4b3ee3e67675af9b52c91e3edfd3e7675ee25d214a69cc540a6c451c8b9385a2" Jul 09 16:16:40.384601 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:16:40.384337 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7866497494-nzrk9_openshift-console-operator(a284ea64-610c-40f7-837b-fa0975c0a290)\"" pod="openshift-console-operator/console-operator-7866497494-nzrk9" podUID="a284ea64-610c-40f7-837b-fa0975c0a290" Jul 09 16:16:46.051589 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:46.051548 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-c9pj8"] Jul 09 16:16:46.058629 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:46.058599 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-c9pj8" Jul 09 16:16:46.061614 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:46.061589 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 09 16:16:46.062308 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:46.062279 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 09 16:16:46.062944 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:46.062922 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 09 16:16:46.063021 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:46.062960 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 09 16:16:46.063021 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:46.062980 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-zhzht\"" Jul 09 16:16:46.148034 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:46.147999 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/792741f3-785c-420e-98c5-01167e3649ad-sys\") pod \"node-exporter-c9pj8\" (UID: \"792741f3-785c-420e-98c5-01167e3649ad\") " pod="openshift-monitoring/node-exporter-c9pj8" Jul 09 16:16:46.148034 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:46.148035 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/792741f3-785c-420e-98c5-01167e3649ad-node-exporter-tls\") pod \"node-exporter-c9pj8\" (UID: \"792741f3-785c-420e-98c5-01167e3649ad\") " pod="openshift-monitoring/node-exporter-c9pj8" Jul 09 16:16:46.148286 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:46.148058 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/792741f3-785c-420e-98c5-01167e3649ad-metrics-client-ca\") pod \"node-exporter-c9pj8\" (UID: \"792741f3-785c-420e-98c5-01167e3649ad\") " pod="openshift-monitoring/node-exporter-c9pj8" Jul 09 16:16:46.148286 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:46.148075 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/792741f3-785c-420e-98c5-01167e3649ad-root\") pod \"node-exporter-c9pj8\" (UID: \"792741f3-785c-420e-98c5-01167e3649ad\") " pod="openshift-monitoring/node-exporter-c9pj8" Jul 09 16:16:46.148286 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:46.148200 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/792741f3-785c-420e-98c5-01167e3649ad-node-exporter-accelerators-collector-config\") pod \"node-exporter-c9pj8\" (UID: \"792741f3-785c-420e-98c5-01167e3649ad\") " pod="openshift-monitoring/node-exporter-c9pj8" Jul 09 16:16:46.148286 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:46.148260 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/792741f3-785c-420e-98c5-01167e3649ad-node-exporter-textfile\") pod \"node-exporter-c9pj8\" (UID: \"792741f3-785c-420e-98c5-01167e3649ad\") " pod="openshift-monitoring/node-exporter-c9pj8" Jul 09 16:16:46.148484 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:46.148300 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/792741f3-785c-420e-98c5-01167e3649ad-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c9pj8\" (UID: \"792741f3-785c-420e-98c5-01167e3649ad\") " pod="openshift-monitoring/node-exporter-c9pj8" Jul 09 16:16:46.148484 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:46.148328 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxcpq\" (UniqueName: \"kubernetes.io/projected/792741f3-785c-420e-98c5-01167e3649ad-kube-api-access-kxcpq\") pod \"node-exporter-c9pj8\" (UID: \"792741f3-785c-420e-98c5-01167e3649ad\") " pod="openshift-monitoring/node-exporter-c9pj8" Jul 09 16:16:46.148484 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:46.148366 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/792741f3-785c-420e-98c5-01167e3649ad-node-exporter-wtmp\") pod \"node-exporter-c9pj8\" (UID: \"792741f3-785c-420e-98c5-01167e3649ad\") " pod="openshift-monitoring/node-exporter-c9pj8" Jul 09 16:16:46.249796 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:46.249761 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/792741f3-785c-420e-98c5-01167e3649ad-node-exporter-tls\") pod \"node-exporter-c9pj8\" (UID: \"792741f3-785c-420e-98c5-01167e3649ad\") " pod="openshift-monitoring/node-exporter-c9pj8" Jul 09 16:16:46.250028 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:46.249808 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/792741f3-785c-420e-98c5-01167e3649ad-metrics-client-ca\") pod \"node-exporter-c9pj8\" (UID: \"792741f3-785c-420e-98c5-01167e3649ad\") " pod="openshift-monitoring/node-exporter-c9pj8" Jul 09 16:16:46.250028 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:46.249835 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/792741f3-785c-420e-98c5-01167e3649ad-root\") pod \"node-exporter-c9pj8\" (UID: \"792741f3-785c-420e-98c5-01167e3649ad\") " pod="openshift-monitoring/node-exporter-c9pj8" Jul 09 16:16:46.250028 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:46.249919 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/792741f3-785c-420e-98c5-01167e3649ad-node-exporter-accelerators-collector-config\") pod \"node-exporter-c9pj8\" (UID: \"792741f3-785c-420e-98c5-01167e3649ad\") " pod="openshift-monitoring/node-exporter-c9pj8" Jul 09 16:16:46.250028 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:16:46.249956 2569 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 09 16:16:46.250028 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:46.249967 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/792741f3-785c-420e-98c5-01167e3649ad-node-exporter-textfile\") pod \"node-exporter-c9pj8\" (UID: \"792741f3-785c-420e-98c5-01167e3649ad\") " pod="openshift-monitoring/node-exporter-c9pj8" Jul 09 16:16:46.250028 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:46.249980 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/792741f3-785c-420e-98c5-01167e3649ad-root\") pod \"node-exporter-c9pj8\" (UID: \"792741f3-785c-420e-98c5-01167e3649ad\") " pod="openshift-monitoring/node-exporter-c9pj8" Jul 09 16:16:46.250028 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:46.249989 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/792741f3-785c-420e-98c5-01167e3649ad-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c9pj8\" (UID: \"792741f3-785c-420e-98c5-01167e3649ad\") " pod="openshift-monitoring/node-exporter-c9pj8" Jul 09 16:16:46.250028 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:16:46.250032 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/792741f3-785c-420e-98c5-01167e3649ad-node-exporter-tls podName:792741f3-785c-420e-98c5-01167e3649ad nodeName:}" failed. No retries permitted until 2026-07-09 16:16:46.750009975 +0000 UTC m=+91.943116360 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/792741f3-785c-420e-98c5-01167e3649ad-node-exporter-tls") pod "node-exporter-c9pj8" (UID: "792741f3-785c-420e-98c5-01167e3649ad") : secret "node-exporter-tls" not found Jul 09 16:16:46.250513 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:46.250116 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxcpq\" (UniqueName: \"kubernetes.io/projected/792741f3-785c-420e-98c5-01167e3649ad-kube-api-access-kxcpq\") pod \"node-exporter-c9pj8\" (UID: \"792741f3-785c-420e-98c5-01167e3649ad\") " pod="openshift-monitoring/node-exporter-c9pj8" Jul 09 16:16:46.250513 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:46.250158 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/792741f3-785c-420e-98c5-01167e3649ad-node-exporter-wtmp\") pod \"node-exporter-c9pj8\" (UID: \"792741f3-785c-420e-98c5-01167e3649ad\") " pod="openshift-monitoring/node-exporter-c9pj8" Jul 09 16:16:46.250513 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:46.250224 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/792741f3-785c-420e-98c5-01167e3649ad-sys\") pod \"node-exporter-c9pj8\" (UID: \"792741f3-785c-420e-98c5-01167e3649ad\") " pod="openshift-monitoring/node-exporter-c9pj8" Jul 09 16:16:46.250513 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:46.250302 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/792741f3-785c-420e-98c5-01167e3649ad-sys\") pod \"node-exporter-c9pj8\" (UID: \"792741f3-785c-420e-98c5-01167e3649ad\") " pod="openshift-monitoring/node-exporter-c9pj8" Jul 09 16:16:46.250513 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:46.250403 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/792741f3-785c-420e-98c5-01167e3649ad-node-exporter-wtmp\") pod \"node-exporter-c9pj8\" (UID: \"792741f3-785c-420e-98c5-01167e3649ad\") " pod="openshift-monitoring/node-exporter-c9pj8" Jul 09 16:16:46.250513 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:46.250426 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/792741f3-785c-420e-98c5-01167e3649ad-node-exporter-textfile\") pod \"node-exporter-c9pj8\" (UID: \"792741f3-785c-420e-98c5-01167e3649ad\") " pod="openshift-monitoring/node-exporter-c9pj8" Jul 09 16:16:46.250745 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:46.250608 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/792741f3-785c-420e-98c5-01167e3649ad-metrics-client-ca\") pod \"node-exporter-c9pj8\" (UID: \"792741f3-785c-420e-98c5-01167e3649ad\") " pod="openshift-monitoring/node-exporter-c9pj8" Jul 09 16:16:46.250745 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:46.250689 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/792741f3-785c-420e-98c5-01167e3649ad-node-exporter-accelerators-collector-config\") pod \"node-exporter-c9pj8\" (UID: \"792741f3-785c-420e-98c5-01167e3649ad\") " pod="openshift-monitoring/node-exporter-c9pj8" Jul 09 16:16:46.252399 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:46.252376 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/792741f3-785c-420e-98c5-01167e3649ad-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c9pj8\" (UID: \"792741f3-785c-420e-98c5-01167e3649ad\") " pod="openshift-monitoring/node-exporter-c9pj8" Jul 09 16:16:46.259316 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:46.259289 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxcpq\" (UniqueName: \"kubernetes.io/projected/792741f3-785c-420e-98c5-01167e3649ad-kube-api-access-kxcpq\") pod \"node-exporter-c9pj8\" (UID: \"792741f3-785c-420e-98c5-01167e3649ad\") " pod="openshift-monitoring/node-exporter-c9pj8" Jul 09 16:16:46.755103 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:46.755065 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/792741f3-785c-420e-98c5-01167e3649ad-node-exporter-tls\") pod \"node-exporter-c9pj8\" (UID: \"792741f3-785c-420e-98c5-01167e3649ad\") " pod="openshift-monitoring/node-exporter-c9pj8" Jul 09 16:16:46.757490 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:46.757464 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/792741f3-785c-420e-98c5-01167e3649ad-node-exporter-tls\") pod \"node-exporter-c9pj8\" (UID: \"792741f3-785c-420e-98c5-01167e3649ad\") " pod="openshift-monitoring/node-exporter-c9pj8" Jul 09 16:16:46.816554 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:46.816518 2569 patch_prober.go:28] interesting pod/image-registry-558d6fd964-d9xbs container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 09 16:16:46.816745 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:46.816572 2569 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-558d6fd964-d9xbs" podUID="1032414d-f6cd-4936-a52f-4f57feddba8f" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 16:16:46.971641 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:46.971610 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-c9pj8" Jul 09 16:16:46.985526 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:16:46.985483 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod792741f3_785c_420e_98c5_01167e3649ad.slice/crio-8c378f066b3551febcdd39c70e4dbe567fe45ab4ce26166e094adba901ecdeef WatchSource:0}: Error finding container 8c378f066b3551febcdd39c70e4dbe567fe45ab4ce26166e094adba901ecdeef: Status 404 returned error can't find the container with id 8c378f066b3551febcdd39c70e4dbe567fe45ab4ce26166e094adba901ecdeef Jul 09 16:16:47.827707 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:47.827658 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-q8zll" Jul 09 16:16:47.864944 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:47.863733 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c9pj8" event={"ID":"792741f3-785c-420e-98c5-01167e3649ad","Type":"ContainerStarted","Data":"8c378f066b3551febcdd39c70e4dbe567fe45ab4ce26166e094adba901ecdeef"} Jul 09 16:16:48.770750 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:48.770722 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-558d6fd964-d9xbs" Jul 09 16:16:48.870012 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:48.869974 2569 generic.go:356] "Generic (PLEG): container finished" podID="792741f3-785c-420e-98c5-01167e3649ad" containerID="c75c46f12e7759b72e8f24907377852d1263b04e022f01d8eabcffca1c31aed1" exitCode=0 Jul 09 16:16:48.870488 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:48.870024 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c9pj8" event={"ID":"792741f3-785c-420e-98c5-01167e3649ad","Type":"ContainerDied","Data":"c75c46f12e7759b72e8f24907377852d1263b04e022f01d8eabcffca1c31aed1"} Jul 09 16:16:49.196524 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:49.196489 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-558d6fd964-d9xbs"] Jul 09 16:16:49.875609 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:49.875576 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c9pj8" event={"ID":"792741f3-785c-420e-98c5-01167e3649ad","Type":"ContainerStarted","Data":"17526ac02ead59598422bee6871be402bcb158f6bbd3a029e949efe1210f4744"} Jul 09 16:16:49.876021 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:49.875627 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c9pj8" event={"ID":"792741f3-785c-420e-98c5-01167e3649ad","Type":"ContainerStarted","Data":"c8bcd1a3cc9871c8380240029bfeb3afd32fb87f66b68301d5ee48ab0180372e"} Jul 09 16:16:49.897730 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:49.897680 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-c9pj8" podStartSLOduration=2.8045034639999997 podStartE2EDuration="3.897663448s" totalImagesPullingTime="1.028860388s" totalInitContainerRuntime="64.299596ms" isStatefulPod=true podCreationTimestamp="2026-07-09 16:16:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:16:49.896929322 +0000 UTC m=+95.090035702" watchObservedRunningTime="2026-07-09 16:16:49.897663448 +0000 UTC m=+95.090769856" Jul 09 16:16:52.384166 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:52.384128 2569 scope.go:117] "RemoveContainer" containerID="4b3ee3e67675af9b52c91e3edfd3e7675ee25d214a69cc540a6c451c8b9385a2" Jul 09 16:16:52.713105 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:52.713014 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-kn7sp" Jul 09 16:16:52.889922 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:52.889866 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-nzrk9_a284ea64-610c-40f7-837b-fa0975c0a290/console-operator/2.log" Jul 09 16:16:52.890113 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:52.889999 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-nzrk9" event={"ID":"a284ea64-610c-40f7-837b-fa0975c0a290","Type":"ContainerStarted","Data":"9bcdb5e91f65d61ab8cfd8fbb1e7e81908d76358453db9339125081eef4b8864"} Jul 09 16:16:52.890264 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:52.890232 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-nzrk9" Jul 09 16:16:52.910905 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:52.910837 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-7866497494-nzrk9" podStartSLOduration=62.644440529 podStartE2EDuration="1m9.91082084s" totalImagesPullingTime="7.266380311s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:15:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:16:52.909811421 +0000 UTC m=+98.102917789" watchObservedRunningTime="2026-07-09 16:16:52.91082084 +0000 UTC m=+98.103927258" Jul 09 16:16:53.633894 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:16:53.633838 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7866497494-nzrk9" Jul 09 16:17:08.955995 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:17:08.955962 2569 generic.go:356] "Generic (PLEG): container finished" podID="79ec4c79-9eb6-427a-b2f3-e66a6cd57dcb" containerID="8f881a8a49d27bed47d97f70fb59fa88bdfcb263bf280abfe7f8b5616c712359" exitCode=0 Jul 09 16:17:08.956452 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:17:08.956037 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qfsjp" event={"ID":"79ec4c79-9eb6-427a-b2f3-e66a6cd57dcb","Type":"ContainerDied","Data":"8f881a8a49d27bed47d97f70fb59fa88bdfcb263bf280abfe7f8b5616c712359"} Jul 09 16:17:08.956452 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:17:08.956428 2569 scope.go:117] "RemoveContainer" containerID="8f881a8a49d27bed47d97f70fb59fa88bdfcb263bf280abfe7f8b5616c712359" Jul 09 16:17:09.962044 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:17:09.962014 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qfsjp" event={"ID":"79ec4c79-9eb6-427a-b2f3-e66a6cd57dcb","Type":"ContainerStarted","Data":"372eb09c0111fa0f97aca738dd1c8e0ac44d126fdb76458da16ca02da82360d1"} Jul 09 16:17:14.218118 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:17:14.218073 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-558d6fd964-d9xbs" podUID="1032414d-f6cd-4936-a52f-4f57feddba8f" containerName="registry" containerID="cri-o://6fc391579119b7eca8c65444ebe216dc958f05204a2bbf5ad73eab9b62462e72" gracePeriod=30 Jul 09 16:17:14.478912 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:17:14.478836 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-558d6fd964-d9xbs" Jul 09 16:17:14.606492 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:17:14.606448 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1032414d-f6cd-4936-a52f-4f57feddba8f-bound-sa-token\") pod \"1032414d-f6cd-4936-a52f-4f57feddba8f\" (UID: \"1032414d-f6cd-4936-a52f-4f57feddba8f\") " Jul 09 16:17:14.606665 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:17:14.606514 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1032414d-f6cd-4936-a52f-4f57feddba8f-image-registry-private-configuration\") pod \"1032414d-f6cd-4936-a52f-4f57feddba8f\" (UID: \"1032414d-f6cd-4936-a52f-4f57feddba8f\") " Jul 09 16:17:14.606665 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:17:14.606535 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1032414d-f6cd-4936-a52f-4f57feddba8f-installation-pull-secrets\") pod \"1032414d-f6cd-4936-a52f-4f57feddba8f\" (UID: \"1032414d-f6cd-4936-a52f-4f57feddba8f\") " Jul 09 16:17:14.606665 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:17:14.606554 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1032414d-f6cd-4936-a52f-4f57feddba8f-trusted-ca\") pod \"1032414d-f6cd-4936-a52f-4f57feddba8f\" (UID: \"1032414d-f6cd-4936-a52f-4f57feddba8f\") " Jul 09 16:17:14.606665 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:17:14.606574 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1032414d-f6cd-4936-a52f-4f57feddba8f-registry-certificates\") pod \"1032414d-f6cd-4936-a52f-4f57feddba8f\" (UID: \"1032414d-f6cd-4936-a52f-4f57feddba8f\") " Jul 09 16:17:14.606665 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:17:14.606601 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6rpw2\" (UniqueName: \"kubernetes.io/projected/1032414d-f6cd-4936-a52f-4f57feddba8f-kube-api-access-6rpw2\") pod \"1032414d-f6cd-4936-a52f-4f57feddba8f\" (UID: \"1032414d-f6cd-4936-a52f-4f57feddba8f\") " Jul 09 16:17:14.606665 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:17:14.606633 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1032414d-f6cd-4936-a52f-4f57feddba8f-ca-trust-extracted\") pod \"1032414d-f6cd-4936-a52f-4f57feddba8f\" (UID: \"1032414d-f6cd-4936-a52f-4f57feddba8f\") " Jul 09 16:17:14.606994 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:17:14.606677 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1032414d-f6cd-4936-a52f-4f57feddba8f-registry-tls\") pod \"1032414d-f6cd-4936-a52f-4f57feddba8f\" (UID: \"1032414d-f6cd-4936-a52f-4f57feddba8f\") " Jul 09 16:17:14.607121 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:17:14.607050 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1032414d-f6cd-4936-a52f-4f57feddba8f-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "1032414d-f6cd-4936-a52f-4f57feddba8f" (UID: "1032414d-f6cd-4936-a52f-4f57feddba8f"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 16:17:14.607121 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:17:14.607101 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1032414d-f6cd-4936-a52f-4f57feddba8f-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "1032414d-f6cd-4936-a52f-4f57feddba8f" (UID: "1032414d-f6cd-4936-a52f-4f57feddba8f"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 16:17:14.609097 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:17:14.609052 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1032414d-f6cd-4936-a52f-4f57feddba8f-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "1032414d-f6cd-4936-a52f-4f57feddba8f" (UID: "1032414d-f6cd-4936-a52f-4f57feddba8f"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 16:17:14.609097 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:17:14.609080 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1032414d-f6cd-4936-a52f-4f57feddba8f-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "1032414d-f6cd-4936-a52f-4f57feddba8f" (UID: "1032414d-f6cd-4936-a52f-4f57feddba8f"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 16:17:14.609308 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:17:14.609280 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1032414d-f6cd-4936-a52f-4f57feddba8f-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "1032414d-f6cd-4936-a52f-4f57feddba8f" (UID: "1032414d-f6cd-4936-a52f-4f57feddba8f"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 16:17:14.609518 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:17:14.609488 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1032414d-f6cd-4936-a52f-4f57feddba8f-kube-api-access-6rpw2" (OuterVolumeSpecName: "kube-api-access-6rpw2") pod "1032414d-f6cd-4936-a52f-4f57feddba8f" (UID: "1032414d-f6cd-4936-a52f-4f57feddba8f"). InnerVolumeSpecName "kube-api-access-6rpw2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 16:17:14.609600 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:17:14.609537 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1032414d-f6cd-4936-a52f-4f57feddba8f-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "1032414d-f6cd-4936-a52f-4f57feddba8f" (UID: "1032414d-f6cd-4936-a52f-4f57feddba8f"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 16:17:14.615383 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:17:14.615357 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1032414d-f6cd-4936-a52f-4f57feddba8f-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "1032414d-f6cd-4936-a52f-4f57feddba8f" (UID: "1032414d-f6cd-4936-a52f-4f57feddba8f"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:17:14.707963 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:17:14.707929 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rpw2\" (UniqueName: \"kubernetes.io/projected/1032414d-f6cd-4936-a52f-4f57feddba8f-kube-api-access-6rpw2\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:17:14.707963 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:17:14.707957 2569 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1032414d-f6cd-4936-a52f-4f57feddba8f-ca-trust-extracted\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:17:14.707963 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:17:14.707967 2569 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1032414d-f6cd-4936-a52f-4f57feddba8f-registry-tls\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:17:14.708182 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:17:14.707978 2569 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1032414d-f6cd-4936-a52f-4f57feddba8f-bound-sa-token\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:17:14.708182 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:17:14.707987 2569 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1032414d-f6cd-4936-a52f-4f57feddba8f-image-registry-private-configuration\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:17:14.708182 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:17:14.707997 2569 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1032414d-f6cd-4936-a52f-4f57feddba8f-installation-pull-secrets\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:17:14.708182 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:17:14.708005 2569 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1032414d-f6cd-4936-a52f-4f57feddba8f-trusted-ca\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:17:14.708182 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:17:14.708013 2569 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1032414d-f6cd-4936-a52f-4f57feddba8f-registry-certificates\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:17:14.982523 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:17:14.982490 2569 generic.go:356] "Generic (PLEG): container finished" podID="1032414d-f6cd-4936-a52f-4f57feddba8f" containerID="6fc391579119b7eca8c65444ebe216dc958f05204a2bbf5ad73eab9b62462e72" exitCode=0 Jul 09 16:17:14.982724 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:17:14.982533 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-558d6fd964-d9xbs" Jul 09 16:17:14.982724 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:17:14.982576 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-558d6fd964-d9xbs" event={"ID":"1032414d-f6cd-4936-a52f-4f57feddba8f","Type":"ContainerDied","Data":"6fc391579119b7eca8c65444ebe216dc958f05204a2bbf5ad73eab9b62462e72"} Jul 09 16:17:14.982724 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:17:14.982633 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-558d6fd964-d9xbs" event={"ID":"1032414d-f6cd-4936-a52f-4f57feddba8f","Type":"ContainerDied","Data":"41f13333aee138496e3ea6f8083c2460effbf53d79c59fcf9aa5d5a1b9f64c9e"} Jul 09 16:17:14.982724 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:17:14.982660 2569 scope.go:117] "RemoveContainer" containerID="6fc391579119b7eca8c65444ebe216dc958f05204a2bbf5ad73eab9b62462e72" Jul 09 16:17:14.991678 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:17:14.991659 2569 scope.go:117] "RemoveContainer" containerID="6fc391579119b7eca8c65444ebe216dc958f05204a2bbf5ad73eab9b62462e72" Jul 09 16:17:14.991986 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:17:14.991966 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fc391579119b7eca8c65444ebe216dc958f05204a2bbf5ad73eab9b62462e72\": container with ID starting with 6fc391579119b7eca8c65444ebe216dc958f05204a2bbf5ad73eab9b62462e72 not found: ID does not exist" containerID="6fc391579119b7eca8c65444ebe216dc958f05204a2bbf5ad73eab9b62462e72" Jul 09 16:17:14.992045 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:17:14.991994 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fc391579119b7eca8c65444ebe216dc958f05204a2bbf5ad73eab9b62462e72"} err="failed to get container status \"6fc391579119b7eca8c65444ebe216dc958f05204a2bbf5ad73eab9b62462e72\": rpc error: code = NotFound desc = could not find container \"6fc391579119b7eca8c65444ebe216dc958f05204a2bbf5ad73eab9b62462e72\": container with ID starting with 6fc391579119b7eca8c65444ebe216dc958f05204a2bbf5ad73eab9b62462e72 not found: ID does not exist" Jul 09 16:17:15.004478 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:17:15.004447 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-558d6fd964-d9xbs"] Jul 09 16:17:15.008292 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:17:15.008266 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-558d6fd964-d9xbs"] Jul 09 16:17:15.388785 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:17:15.388751 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1032414d-f6cd-4936-a52f-4f57feddba8f" path="/var/lib/kubelet/pods/1032414d-f6cd-4936-a52f-4f57feddba8f/volumes" Jul 09 16:17:19.001084 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:17:19.001040 2569 generic.go:356] "Generic (PLEG): container finished" podID="bb610783-4a86-4291-a9f0-42deb83b322d" containerID="182e8256b448b6b5aacaaf85377f7879b8a3446b5cedcef6361768242b9c0043" exitCode=0 Jul 09 16:17:19.001857 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:17:19.001722 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-6jksf" event={"ID":"bb610783-4a86-4291-a9f0-42deb83b322d","Type":"ContainerDied","Data":"182e8256b448b6b5aacaaf85377f7879b8a3446b5cedcef6361768242b9c0043"} Jul 09 16:17:19.002580 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:17:19.002564 2569 scope.go:117] "RemoveContainer" containerID="182e8256b448b6b5aacaaf85377f7879b8a3446b5cedcef6361768242b9c0043" Jul 09 16:17:20.007411 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:17:20.007376 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-6jksf" event={"ID":"bb610783-4a86-4291-a9f0-42deb83b322d","Type":"ContainerStarted","Data":"183639a31ab5555da71b5f9b985e614f5319de4c4c4768339083cdf6d8918bbf"} Jul 09 16:17:29.043538 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:17:29.043501 2569 generic.go:356] "Generic (PLEG): container finished" podID="f54c8925-c09f-46d8-84fb-9f3af042dc41" containerID="3562980d29b75d34d213ad74c95f5a34b87cc98ce953262972e5af5be3a4d338" exitCode=0 Jul 09 16:17:29.044037 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:17:29.043574 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-xjcmf" event={"ID":"f54c8925-c09f-46d8-84fb-9f3af042dc41","Type":"ContainerDied","Data":"3562980d29b75d34d213ad74c95f5a34b87cc98ce953262972e5af5be3a4d338"} Jul 09 16:17:29.044135 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:17:29.044049 2569 scope.go:117] "RemoveContainer" containerID="3562980d29b75d34d213ad74c95f5a34b87cc98ce953262972e5af5be3a4d338" Jul 09 16:17:30.049363 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:17:30.049330 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-xjcmf" event={"ID":"f54c8925-c09f-46d8-84fb-9f3af042dc41","Type":"ContainerStarted","Data":"52404e12a40b68e340963d67e1d278fc9ff7f06d34e8eaa48e1ad82039baa114"} Jul 09 16:20:15.234114 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:20:15.234085 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-nzrk9_a284ea64-610c-40f7-837b-fa0975c0a290/console-operator/2.log" Jul 09 16:20:15.237154 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:20:15.237118 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-nzrk9_a284ea64-610c-40f7-837b-fa0975c0a290/console-operator/2.log" Jul 09 16:20:15.242183 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:20:15.242158 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-49sqd_17313664-7a0f-472b-a597-39f8c5412bc3/ovn-acl-logging/0.log" Jul 09 16:20:15.245639 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:20:15.245605 2569 kubelet.go:1648] "Image garbage collection succeeded" Jul 09 16:20:15.245807 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:20:15.245694 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-49sqd_17313664-7a0f-472b-a597-39f8c5412bc3/ovn-acl-logging/0.log" Jul 09 16:21:13.355476 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:21:13.355394 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7df8c55954-fxg9l"] Jul 09 16:21:13.355866 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:21:13.355688 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1032414d-f6cd-4936-a52f-4f57feddba8f" containerName="registry" Jul 09 16:21:13.355866 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:21:13.355699 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="1032414d-f6cd-4936-a52f-4f57feddba8f" containerName="registry" Jul 09 16:21:13.355866 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:21:13.355761 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="1032414d-f6cd-4936-a52f-4f57feddba8f" containerName="registry" Jul 09 16:21:13.358710 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:21:13.358693 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7df8c55954-fxg9l" Jul 09 16:21:13.363186 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:21:13.363158 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 09 16:21:13.363341 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:21:13.363268 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-gdnnq\"" Jul 09 16:21:13.363477 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:21:13.363461 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 09 16:21:13.364187 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:21:13.364166 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 09 16:21:13.364323 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:21:13.364236 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 09 16:21:13.383347 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:21:13.383307 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7df8c55954-fxg9l"] Jul 09 16:21:13.407049 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:21:13.407010 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/880f381d-9f12-4935-93d6-46862471d4bc-tmp\") pod \"opendatahub-operator-controller-manager-7df8c55954-fxg9l\" (UID: \"880f381d-9f12-4935-93d6-46862471d4bc\") " pod="opendatahub/opendatahub-operator-controller-manager-7df8c55954-fxg9l" Jul 09 16:21:13.407218 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:21:13.407056 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/880f381d-9f12-4935-93d6-46862471d4bc-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7df8c55954-fxg9l\" (UID: \"880f381d-9f12-4935-93d6-46862471d4bc\") " pod="opendatahub/opendatahub-operator-controller-manager-7df8c55954-fxg9l" Jul 09 16:21:13.407218 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:21:13.407096 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/880f381d-9f12-4935-93d6-46862471d4bc-manifests\") pod \"opendatahub-operator-controller-manager-7df8c55954-fxg9l\" (UID: \"880f381d-9f12-4935-93d6-46862471d4bc\") " pod="opendatahub/opendatahub-operator-controller-manager-7df8c55954-fxg9l" Jul 09 16:21:13.407218 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:21:13.407157 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/880f381d-9f12-4935-93d6-46862471d4bc-webhook-cert\") pod \"opendatahub-operator-controller-manager-7df8c55954-fxg9l\" (UID: \"880f381d-9f12-4935-93d6-46862471d4bc\") " pod="opendatahub/opendatahub-operator-controller-manager-7df8c55954-fxg9l" Jul 09 16:21:13.407330 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:21:13.407231 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjs45\" (UniqueName: \"kubernetes.io/projected/880f381d-9f12-4935-93d6-46862471d4bc-kube-api-access-jjs45\") pod \"opendatahub-operator-controller-manager-7df8c55954-fxg9l\" (UID: \"880f381d-9f12-4935-93d6-46862471d4bc\") " pod="opendatahub/opendatahub-operator-controller-manager-7df8c55954-fxg9l" Jul 09 16:21:13.508348 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:21:13.508311 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/880f381d-9f12-4935-93d6-46862471d4bc-webhook-cert\") pod \"opendatahub-operator-controller-manager-7df8c55954-fxg9l\" (UID: \"880f381d-9f12-4935-93d6-46862471d4bc\") " pod="opendatahub/opendatahub-operator-controller-manager-7df8c55954-fxg9l" Jul 09 16:21:13.508549 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:21:13.508367 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjs45\" (UniqueName: \"kubernetes.io/projected/880f381d-9f12-4935-93d6-46862471d4bc-kube-api-access-jjs45\") pod \"opendatahub-operator-controller-manager-7df8c55954-fxg9l\" (UID: \"880f381d-9f12-4935-93d6-46862471d4bc\") " pod="opendatahub/opendatahub-operator-controller-manager-7df8c55954-fxg9l" Jul 09 16:21:13.508549 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:21:13.508395 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/880f381d-9f12-4935-93d6-46862471d4bc-tmp\") pod \"opendatahub-operator-controller-manager-7df8c55954-fxg9l\" (UID: \"880f381d-9f12-4935-93d6-46862471d4bc\") " pod="opendatahub/opendatahub-operator-controller-manager-7df8c55954-fxg9l" Jul 09 16:21:13.508549 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:21:13.508420 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/880f381d-9f12-4935-93d6-46862471d4bc-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7df8c55954-fxg9l\" (UID: \"880f381d-9f12-4935-93d6-46862471d4bc\") " pod="opendatahub/opendatahub-operator-controller-manager-7df8c55954-fxg9l" Jul 09 16:21:13.508549 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:21:13.508450 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/880f381d-9f12-4935-93d6-46862471d4bc-manifests\") pod \"opendatahub-operator-controller-manager-7df8c55954-fxg9l\" (UID: \"880f381d-9f12-4935-93d6-46862471d4bc\") " pod="opendatahub/opendatahub-operator-controller-manager-7df8c55954-fxg9l" Jul 09 16:21:13.508848 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:21:13.508821 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/880f381d-9f12-4935-93d6-46862471d4bc-manifests\") pod \"opendatahub-operator-controller-manager-7df8c55954-fxg9l\" (UID: \"880f381d-9f12-4935-93d6-46862471d4bc\") " pod="opendatahub/opendatahub-operator-controller-manager-7df8c55954-fxg9l" Jul 09 16:21:13.510720 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:21:13.510697 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/880f381d-9f12-4935-93d6-46862471d4bc-tmp\") pod \"opendatahub-operator-controller-manager-7df8c55954-fxg9l\" (UID: \"880f381d-9f12-4935-93d6-46862471d4bc\") " pod="opendatahub/opendatahub-operator-controller-manager-7df8c55954-fxg9l" Jul 09 16:21:13.511018 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:21:13.511001 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/880f381d-9f12-4935-93d6-46862471d4bc-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7df8c55954-fxg9l\" (UID: \"880f381d-9f12-4935-93d6-46862471d4bc\") " pod="opendatahub/opendatahub-operator-controller-manager-7df8c55954-fxg9l" Jul 09 16:21:13.511062 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:21:13.511042 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/880f381d-9f12-4935-93d6-46862471d4bc-webhook-cert\") pod \"opendatahub-operator-controller-manager-7df8c55954-fxg9l\" (UID: \"880f381d-9f12-4935-93d6-46862471d4bc\") " pod="opendatahub/opendatahub-operator-controller-manager-7df8c55954-fxg9l" Jul 09 16:21:13.520962 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:21:13.520942 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjs45\" (UniqueName: \"kubernetes.io/projected/880f381d-9f12-4935-93d6-46862471d4bc-kube-api-access-jjs45\") pod \"opendatahub-operator-controller-manager-7df8c55954-fxg9l\" (UID: \"880f381d-9f12-4935-93d6-46862471d4bc\") " pod="opendatahub/opendatahub-operator-controller-manager-7df8c55954-fxg9l" Jul 09 16:21:13.670040 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:21:13.669953 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7df8c55954-fxg9l" Jul 09 16:21:13.827590 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:21:13.827554 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7df8c55954-fxg9l"] Jul 09 16:21:13.831618 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:21:13.831595 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod880f381d_9f12_4935_93d6_46862471d4bc.slice/crio-d48de583aff632ce93c5b5bd7cc6defaa96972a23162780b1a22e6037479f522 WatchSource:0}: Error finding container d48de583aff632ce93c5b5bd7cc6defaa96972a23162780b1a22e6037479f522: Status 404 returned error can't find the container with id d48de583aff632ce93c5b5bd7cc6defaa96972a23162780b1a22e6037479f522 Jul 09 16:21:13.833346 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:21:13.833330 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 16:21:13.911435 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:21:13.911392 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7df8c55954-fxg9l" event={"ID":"880f381d-9f12-4935-93d6-46862471d4bc","Type":"ContainerStarted","Data":"d48de583aff632ce93c5b5bd7cc6defaa96972a23162780b1a22e6037479f522"} Jul 09 16:21:17.931892 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:21:17.931843 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7df8c55954-fxg9l" event={"ID":"880f381d-9f12-4935-93d6-46862471d4bc","Type":"ContainerStarted","Data":"2c9d4a6a1c2b4b67f8c472b45d3c3a4b8ab264e4240800c46376971c46fd1611"} Jul 09 16:21:18.937603 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:21:18.937556 2569 generic.go:356] "Generic (PLEG): container finished" podID="880f381d-9f12-4935-93d6-46862471d4bc" containerID="2c9d4a6a1c2b4b67f8c472b45d3c3a4b8ab264e4240800c46376971c46fd1611" exitCode=0 Jul 09 16:21:18.938067 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:21:18.937648 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7df8c55954-fxg9l" event={"ID":"880f381d-9f12-4935-93d6-46862471d4bc","Type":"ContainerDied","Data":"2c9d4a6a1c2b4b67f8c472b45d3c3a4b8ab264e4240800c46376971c46fd1611"} Jul 09 16:21:19.945182 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:21:19.945139 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7df8c55954-fxg9l" event={"ID":"880f381d-9f12-4935-93d6-46862471d4bc","Type":"ContainerStarted","Data":"e484c84c69753a3f452d669246e12f631aa20c737fb9dd291987ac93676dec8b"} Jul 09 16:21:19.945680 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:21:19.945272 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7df8c55954-fxg9l" Jul 09 16:21:19.969863 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:21:19.969806 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7df8c55954-fxg9l" podStartSLOduration=2.177796168 podStartE2EDuration="6.969786254s" totalImagesPullingTime="3.635642936s" totalInitContainerRuntime="1.15634715s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:21:13 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:21:19.968245801 +0000 UTC m=+365.161352182" watchObservedRunningTime="2026-07-09 16:21:19.969786254 +0000 UTC m=+365.162892635" Jul 09 16:21:30.951665 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:21:30.951633 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7df8c55954-fxg9l" Jul 09 16:21:42.727826 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:21:42.727785 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5b747dc9cb-w7zdv"] Jul 09 16:21:42.731172 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:21:42.731152 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5b747dc9cb-w7zdv" Jul 09 16:21:42.735839 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:21:42.735820 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 09 16:21:42.736134 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:21:42.736116 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 09 16:21:42.736767 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:21:42.736749 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 09 16:21:42.737090 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:21:42.737070 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-sfm9g\"" Jul 09 16:21:42.740052 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:21:42.740030 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 09 16:21:42.742357 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:21:42.742339 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 09 16:21:42.757971 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:21:42.754977 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5b747dc9cb-w7zdv"] Jul 09 16:21:42.864738 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:21:42.864702 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/92c8265d-4487-47f0-a924-e3990dcfbd87-metrics-cert\") pod \"lws-controller-manager-5b747dc9cb-w7zdv\" (UID: \"92c8265d-4487-47f0-a924-e3990dcfbd87\") " pod="openshift-lws-operator/lws-controller-manager-5b747dc9cb-w7zdv" Jul 09 16:21:42.864738 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:21:42.864744 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/92c8265d-4487-47f0-a924-e3990dcfbd87-manager-config\") pod \"lws-controller-manager-5b747dc9cb-w7zdv\" (UID: \"92c8265d-4487-47f0-a924-e3990dcfbd87\") " pod="openshift-lws-operator/lws-controller-manager-5b747dc9cb-w7zdv" Jul 09 16:21:42.864994 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:21:42.864772 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/92c8265d-4487-47f0-a924-e3990dcfbd87-cert\") pod \"lws-controller-manager-5b747dc9cb-w7zdv\" (UID: \"92c8265d-4487-47f0-a924-e3990dcfbd87\") " pod="openshift-lws-operator/lws-controller-manager-5b747dc9cb-w7zdv" Jul 09 16:21:42.864994 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:21:42.864823 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56kzz\" (UniqueName: \"kubernetes.io/projected/92c8265d-4487-47f0-a924-e3990dcfbd87-kube-api-access-56kzz\") pod \"lws-controller-manager-5b747dc9cb-w7zdv\" (UID: \"92c8265d-4487-47f0-a924-e3990dcfbd87\") " pod="openshift-lws-operator/lws-controller-manager-5b747dc9cb-w7zdv" Jul 09 16:21:42.965997 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:21:42.965965 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/92c8265d-4487-47f0-a924-e3990dcfbd87-metrics-cert\") pod \"lws-controller-manager-5b747dc9cb-w7zdv\" (UID: \"92c8265d-4487-47f0-a924-e3990dcfbd87\") " pod="openshift-lws-operator/lws-controller-manager-5b747dc9cb-w7zdv" Jul 09 16:21:42.965997 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:21:42.966003 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/92c8265d-4487-47f0-a924-e3990dcfbd87-manager-config\") pod \"lws-controller-manager-5b747dc9cb-w7zdv\" (UID: \"92c8265d-4487-47f0-a924-e3990dcfbd87\") " pod="openshift-lws-operator/lws-controller-manager-5b747dc9cb-w7zdv" Jul 09 16:21:42.966222 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:21:42.966032 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/92c8265d-4487-47f0-a924-e3990dcfbd87-cert\") pod \"lws-controller-manager-5b747dc9cb-w7zdv\" (UID: \"92c8265d-4487-47f0-a924-e3990dcfbd87\") " pod="openshift-lws-operator/lws-controller-manager-5b747dc9cb-w7zdv" Jul 09 16:21:42.966222 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:21:42.966076 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56kzz\" (UniqueName: \"kubernetes.io/projected/92c8265d-4487-47f0-a924-e3990dcfbd87-kube-api-access-56kzz\") pod \"lws-controller-manager-5b747dc9cb-w7zdv\" (UID: \"92c8265d-4487-47f0-a924-e3990dcfbd87\") " pod="openshift-lws-operator/lws-controller-manager-5b747dc9cb-w7zdv" Jul 09 16:21:42.966918 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:21:42.966870 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/92c8265d-4487-47f0-a924-e3990dcfbd87-manager-config\") pod \"lws-controller-manager-5b747dc9cb-w7zdv\" (UID: \"92c8265d-4487-47f0-a924-e3990dcfbd87\") " pod="openshift-lws-operator/lws-controller-manager-5b747dc9cb-w7zdv" Jul 09 16:21:42.968599 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:21:42.968505 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/92c8265d-4487-47f0-a924-e3990dcfbd87-metrics-cert\") pod \"lws-controller-manager-5b747dc9cb-w7zdv\" (UID: \"92c8265d-4487-47f0-a924-e3990dcfbd87\") " pod="openshift-lws-operator/lws-controller-manager-5b747dc9cb-w7zdv" Jul 09 16:21:42.969358 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:21:42.969324 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/92c8265d-4487-47f0-a924-e3990dcfbd87-cert\") pod \"lws-controller-manager-5b747dc9cb-w7zdv\" (UID: \"92c8265d-4487-47f0-a924-e3990dcfbd87\") " pod="openshift-lws-operator/lws-controller-manager-5b747dc9cb-w7zdv" Jul 09 16:21:42.975705 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:21:42.975678 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56kzz\" (UniqueName: \"kubernetes.io/projected/92c8265d-4487-47f0-a924-e3990dcfbd87-kube-api-access-56kzz\") pod \"lws-controller-manager-5b747dc9cb-w7zdv\" (UID: \"92c8265d-4487-47f0-a924-e3990dcfbd87\") " pod="openshift-lws-operator/lws-controller-manager-5b747dc9cb-w7zdv" Jul 09 16:21:43.040260 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:21:43.040163 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5b747dc9cb-w7zdv" Jul 09 16:21:43.177859 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:21:43.177806 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5b747dc9cb-w7zdv"] Jul 09 16:21:43.181561 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:21:43.181525 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92c8265d_4487_47f0_a924_e3990dcfbd87.slice/crio-ed95b32b1dd9e8fb506da68215e8935b08c5461de4127e4e114cfe36632848cf WatchSource:0}: Error finding container ed95b32b1dd9e8fb506da68215e8935b08c5461de4127e4e114cfe36632848cf: Status 404 returned error can't find the container with id ed95b32b1dd9e8fb506da68215e8935b08c5461de4127e4e114cfe36632848cf Jul 09 16:21:44.053951 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:21:44.053905 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5b747dc9cb-w7zdv" event={"ID":"92c8265d-4487-47f0-a924-e3990dcfbd87","Type":"ContainerStarted","Data":"ed95b32b1dd9e8fb506da68215e8935b08c5461de4127e4e114cfe36632848cf"} Jul 09 16:21:46.065055 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:21:46.064953 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5b747dc9cb-w7zdv" event={"ID":"92c8265d-4487-47f0-a924-e3990dcfbd87","Type":"ContainerStarted","Data":"5bc683a9a1f7274a4a9a26de83ec027ff2520d2c139072b026e650948a028a1f"} Jul 09 16:21:46.065529 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:21:46.065110 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5b747dc9cb-w7zdv" Jul 09 16:21:46.146419 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:21:46.146367 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5b747dc9cb-w7zdv" podStartSLOduration=1.562749527 podStartE2EDuration="4.1463479s" totalImagesPullingTime="2.583598373s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:21:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:21:46.139255892 +0000 UTC m=+391.332362272" watchObservedRunningTime="2026-07-09 16:21:46.1463479 +0000 UTC m=+391.339454280" Jul 09 16:21:57.071820 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:21:57.071784 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5b747dc9cb-w7zdv" Jul 09 16:22:34.873463 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:22:34.873428 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-qf8hw"] Jul 09 16:22:34.881147 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:22:34.881123 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-qf8hw" Jul 09 16:22:34.883909 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:22:34.883868 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 09 16:22:34.884136 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:22:34.884112 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 09 16:22:34.884278 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:22:34.884011 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-jfqqd\"" Jul 09 16:22:34.885425 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:22:34.885401 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-qf8hw"] Jul 09 16:22:34.987787 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:22:34.987743 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq66z\" (UniqueName: \"kubernetes.io/projected/898fb686-054e-4ac2-bc58-93b3a5b1fc4c-kube-api-access-zq66z\") pod \"kuadrant-operator-catalog-qf8hw\" (UID: \"898fb686-054e-4ac2-bc58-93b3a5b1fc4c\") " pod="kuadrant-system/kuadrant-operator-catalog-qf8hw" Jul 09 16:22:35.089097 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:22:35.089060 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq66z\" (UniqueName: \"kubernetes.io/projected/898fb686-054e-4ac2-bc58-93b3a5b1fc4c-kube-api-access-zq66z\") pod \"kuadrant-operator-catalog-qf8hw\" (UID: \"898fb686-054e-4ac2-bc58-93b3a5b1fc4c\") " pod="kuadrant-system/kuadrant-operator-catalog-qf8hw" Jul 09 16:22:35.098187 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:22:35.098156 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq66z\" (UniqueName: \"kubernetes.io/projected/898fb686-054e-4ac2-bc58-93b3a5b1fc4c-kube-api-access-zq66z\") pod \"kuadrant-operator-catalog-qf8hw\" (UID: \"898fb686-054e-4ac2-bc58-93b3a5b1fc4c\") " pod="kuadrant-system/kuadrant-operator-catalog-qf8hw" Jul 09 16:22:35.193018 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:22:35.192920 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-qf8hw" Jul 09 16:22:35.243514 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:22:35.243482 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-qf8hw"] Jul 09 16:22:35.329276 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:22:35.329251 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-qf8hw"] Jul 09 16:22:35.331771 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:22:35.331739 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod898fb686_054e_4ac2_bc58_93b3a5b1fc4c.slice/crio-f3bca7142dad29d3105c59863a5add8562639f636436aec09f5bd5f7f4d0eea8 WatchSource:0}: Error finding container f3bca7142dad29d3105c59863a5add8562639f636436aec09f5bd5f7f4d0eea8: Status 404 returned error can't find the container with id f3bca7142dad29d3105c59863a5add8562639f636436aec09f5bd5f7f4d0eea8 Jul 09 16:22:35.451787 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:22:35.451704 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-xkzkb"] Jul 09 16:22:35.456505 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:22:35.456484 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-xkzkb" Jul 09 16:22:35.462759 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:22:35.462733 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-xkzkb"] Jul 09 16:22:35.593028 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:22:35.592977 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q4hs\" (UniqueName: \"kubernetes.io/projected/0847f205-b153-4da8-8434-ec8556afa22c-kube-api-access-2q4hs\") pod \"kuadrant-operator-catalog-xkzkb\" (UID: \"0847f205-b153-4da8-8434-ec8556afa22c\") " pod="kuadrant-system/kuadrant-operator-catalog-xkzkb" Jul 09 16:22:35.694225 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:22:35.694182 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q4hs\" (UniqueName: \"kubernetes.io/projected/0847f205-b153-4da8-8434-ec8556afa22c-kube-api-access-2q4hs\") pod \"kuadrant-operator-catalog-xkzkb\" (UID: \"0847f205-b153-4da8-8434-ec8556afa22c\") " pod="kuadrant-system/kuadrant-operator-catalog-xkzkb" Jul 09 16:22:35.702760 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:22:35.702694 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q4hs\" (UniqueName: \"kubernetes.io/projected/0847f205-b153-4da8-8434-ec8556afa22c-kube-api-access-2q4hs\") pod \"kuadrant-operator-catalog-xkzkb\" (UID: \"0847f205-b153-4da8-8434-ec8556afa22c\") " pod="kuadrant-system/kuadrant-operator-catalog-xkzkb" Jul 09 16:22:35.767727 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:22:35.767687 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-xkzkb" Jul 09 16:22:35.910480 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:22:35.910433 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-xkzkb"] Jul 09 16:22:35.923747 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:22:35.923718 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0847f205_b153_4da8_8434_ec8556afa22c.slice/crio-73f00dbecb761ff2bac384369495a109b926a72b9b9da61cab8f3eba8d5b379f WatchSource:0}: Error finding container 73f00dbecb761ff2bac384369495a109b926a72b9b9da61cab8f3eba8d5b379f: Status 404 returned error can't find the container with id 73f00dbecb761ff2bac384369495a109b926a72b9b9da61cab8f3eba8d5b379f Jul 09 16:22:36.291944 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:22:36.291905 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-xkzkb" event={"ID":"0847f205-b153-4da8-8434-ec8556afa22c","Type":"ContainerStarted","Data":"73f00dbecb761ff2bac384369495a109b926a72b9b9da61cab8f3eba8d5b379f"} Jul 09 16:22:36.293106 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:22:36.293077 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-qf8hw" event={"ID":"898fb686-054e-4ac2-bc58-93b3a5b1fc4c","Type":"ContainerStarted","Data":"f3bca7142dad29d3105c59863a5add8562639f636436aec09f5bd5f7f4d0eea8"} Jul 09 16:22:38.305169 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:22:38.305134 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-xkzkb" event={"ID":"0847f205-b153-4da8-8434-ec8556afa22c","Type":"ContainerStarted","Data":"4b8ddbcae8cdf9c1544cbcd1d1e6a936d4979cd8465bf27a41f5035582fafae7"} Jul 09 16:22:38.306732 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:22:38.306701 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-qf8hw" event={"ID":"898fb686-054e-4ac2-bc58-93b3a5b1fc4c","Type":"ContainerStarted","Data":"2403ff971bfe528ff47a3fc13a622fc6bbacc19bc409ce1317108052ca0a8c8f"} Jul 09 16:22:38.306891 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:22:38.306726 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-qf8hw" podUID="898fb686-054e-4ac2-bc58-93b3a5b1fc4c" containerName="registry-server" containerID="cri-o://2403ff971bfe528ff47a3fc13a622fc6bbacc19bc409ce1317108052ca0a8c8f" gracePeriod=2 Jul 09 16:22:38.325164 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:22:38.325119 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-xkzkb" podStartSLOduration=1.762957477 podStartE2EDuration="3.325104365s" totalImagesPullingTime="1.562146888s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:22:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:22:38.323050633 +0000 UTC m=+443.516157014" watchObservedRunningTime="2026-07-09 16:22:38.325104365 +0000 UTC m=+443.518210774" Jul 09 16:22:38.339716 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:22:38.339669 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-qf8hw" podStartSLOduration=2.1857214 podStartE2EDuration="4.339653799s" totalImagesPullingTime="2.153932399s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:22:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:22:38.337995334 +0000 UTC m=+443.531101715" watchObservedRunningTime="2026-07-09 16:22:38.339653799 +0000 UTC m=+443.532760180" Jul 09 16:22:38.556136 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:22:38.556064 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-qf8hw" Jul 09 16:22:38.621426 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:22:38.621388 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zq66z\" (UniqueName: \"kubernetes.io/projected/898fb686-054e-4ac2-bc58-93b3a5b1fc4c-kube-api-access-zq66z\") pod \"898fb686-054e-4ac2-bc58-93b3a5b1fc4c\" (UID: \"898fb686-054e-4ac2-bc58-93b3a5b1fc4c\") " Jul 09 16:22:38.623582 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:22:38.623553 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/898fb686-054e-4ac2-bc58-93b3a5b1fc4c-kube-api-access-zq66z" (OuterVolumeSpecName: "kube-api-access-zq66z") pod "898fb686-054e-4ac2-bc58-93b3a5b1fc4c" (UID: "898fb686-054e-4ac2-bc58-93b3a5b1fc4c"). InnerVolumeSpecName "kube-api-access-zq66z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 16:22:38.722464 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:22:38.722417 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zq66z\" (UniqueName: \"kubernetes.io/projected/898fb686-054e-4ac2-bc58-93b3a5b1fc4c-kube-api-access-zq66z\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:22:39.312297 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:22:39.312261 2569 generic.go:356] "Generic (PLEG): container finished" podID="898fb686-054e-4ac2-bc58-93b3a5b1fc4c" containerID="2403ff971bfe528ff47a3fc13a622fc6bbacc19bc409ce1317108052ca0a8c8f" exitCode=0 Jul 09 16:22:39.312756 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:22:39.312304 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-qf8hw" Jul 09 16:22:39.312756 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:22:39.312348 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-qf8hw" event={"ID":"898fb686-054e-4ac2-bc58-93b3a5b1fc4c","Type":"ContainerDied","Data":"2403ff971bfe528ff47a3fc13a622fc6bbacc19bc409ce1317108052ca0a8c8f"} Jul 09 16:22:39.312756 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:22:39.312442 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-qf8hw" event={"ID":"898fb686-054e-4ac2-bc58-93b3a5b1fc4c","Type":"ContainerDied","Data":"f3bca7142dad29d3105c59863a5add8562639f636436aec09f5bd5f7f4d0eea8"} Jul 09 16:22:39.312756 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:22:39.312467 2569 scope.go:117] "RemoveContainer" containerID="2403ff971bfe528ff47a3fc13a622fc6bbacc19bc409ce1317108052ca0a8c8f" Jul 09 16:22:39.321373 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:22:39.321352 2569 scope.go:117] "RemoveContainer" containerID="2403ff971bfe528ff47a3fc13a622fc6bbacc19bc409ce1317108052ca0a8c8f" Jul 09 16:22:39.321646 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:22:39.321627 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2403ff971bfe528ff47a3fc13a622fc6bbacc19bc409ce1317108052ca0a8c8f\": container with ID starting with 2403ff971bfe528ff47a3fc13a622fc6bbacc19bc409ce1317108052ca0a8c8f not found: ID does not exist" containerID="2403ff971bfe528ff47a3fc13a622fc6bbacc19bc409ce1317108052ca0a8c8f" Jul 09 16:22:39.321710 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:22:39.321653 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2403ff971bfe528ff47a3fc13a622fc6bbacc19bc409ce1317108052ca0a8c8f"} err="failed to get container status \"2403ff971bfe528ff47a3fc13a622fc6bbacc19bc409ce1317108052ca0a8c8f\": rpc error: code = NotFound desc = could not find container \"2403ff971bfe528ff47a3fc13a622fc6bbacc19bc409ce1317108052ca0a8c8f\": container with ID starting with 2403ff971bfe528ff47a3fc13a622fc6bbacc19bc409ce1317108052ca0a8c8f not found: ID does not exist" Jul 09 16:22:39.332513 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:22:39.332487 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-qf8hw"] Jul 09 16:22:39.334727 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:22:39.334701 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-qf8hw"] Jul 09 16:22:39.388682 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:22:39.388651 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="898fb686-054e-4ac2-bc58-93b3a5b1fc4c" path="/var/lib/kubelet/pods/898fb686-054e-4ac2-bc58-93b3a5b1fc4c/volumes" Jul 09 16:22:45.768294 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:22:45.768247 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-xkzkb" Jul 09 16:22:45.768732 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:22:45.768315 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-xkzkb" Jul 09 16:22:45.790755 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:22:45.790727 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-xkzkb" Jul 09 16:22:46.365742 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:22:46.365717 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-xkzkb" Jul 09 16:23:10.470475 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:10.470435 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-vdkcb"] Jul 09 16:23:10.471016 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:10.470754 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="898fb686-054e-4ac2-bc58-93b3a5b1fc4c" containerName="registry-server" Jul 09 16:23:10.471016 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:10.470765 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="898fb686-054e-4ac2-bc58-93b3a5b1fc4c" containerName="registry-server" Jul 09 16:23:10.471016 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:10.470823 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="898fb686-054e-4ac2-bc58-93b3a5b1fc4c" containerName="registry-server" Jul 09 16:23:10.473662 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:10.473643 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-vdkcb" Jul 09 16:23:10.476581 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:10.476558 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 09 16:23:10.476702 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:10.476603 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-tmbdj\"" Jul 09 16:23:10.485067 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:10.485040 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-vdkcb"] Jul 09 16:23:10.515591 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:10.515559 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgmgb\" (UniqueName: \"kubernetes.io/projected/e7fce909-26bb-4713-bac8-a2a046e4c5b0-kube-api-access-rgmgb\") pod \"dns-operator-controller-manager-864dd78c4c-vdkcb\" (UID: \"e7fce909-26bb-4713-bac8-a2a046e4c5b0\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-vdkcb" Jul 09 16:23:10.617034 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:10.616998 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgmgb\" (UniqueName: \"kubernetes.io/projected/e7fce909-26bb-4713-bac8-a2a046e4c5b0-kube-api-access-rgmgb\") pod \"dns-operator-controller-manager-864dd78c4c-vdkcb\" (UID: \"e7fce909-26bb-4713-bac8-a2a046e4c5b0\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-vdkcb" Jul 09 16:23:10.631158 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:10.631126 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgmgb\" (UniqueName: \"kubernetes.io/projected/e7fce909-26bb-4713-bac8-a2a046e4c5b0-kube-api-access-rgmgb\") pod \"dns-operator-controller-manager-864dd78c4c-vdkcb\" (UID: \"e7fce909-26bb-4713-bac8-a2a046e4c5b0\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-vdkcb" Jul 09 16:23:10.783657 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:10.783570 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-vdkcb" Jul 09 16:23:10.918580 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:10.918483 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-vdkcb"] Jul 09 16:23:10.921309 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:23:10.921272 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7fce909_26bb_4713_bac8_a2a046e4c5b0.slice/crio-88dce3b2f0a63a79ca87d05b3c9277c585614acd30778b1dedd6f953d0af226a WatchSource:0}: Error finding container 88dce3b2f0a63a79ca87d05b3c9277c585614acd30778b1dedd6f953d0af226a: Status 404 returned error can't find the container with id 88dce3b2f0a63a79ca87d05b3c9277c585614acd30778b1dedd6f953d0af226a Jul 09 16:23:11.462509 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:11.462476 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-vdkcb" event={"ID":"e7fce909-26bb-4713-bac8-a2a046e4c5b0","Type":"ContainerStarted","Data":"88dce3b2f0a63a79ca87d05b3c9277c585614acd30778b1dedd6f953d0af226a"} Jul 09 16:23:13.221635 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:13.221603 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-grv6b"] Jul 09 16:23:13.224748 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:13.224729 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-grv6b" Jul 09 16:23:13.228816 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:13.228787 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-fjmln\"" Jul 09 16:23:13.237207 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:13.237181 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-grv6b"] Jul 09 16:23:13.343108 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:13.343080 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhgxf\" (UniqueName: \"kubernetes.io/projected/ffc04543-a138-455c-be42-0497ca3b93e2-kube-api-access-qhgxf\") pod \"limitador-operator-controller-manager-859d7b78bd-grv6b\" (UID: \"ffc04543-a138-455c-be42-0497ca3b93e2\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-grv6b" Jul 09 16:23:13.444538 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:13.444504 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhgxf\" (UniqueName: \"kubernetes.io/projected/ffc04543-a138-455c-be42-0497ca3b93e2-kube-api-access-qhgxf\") pod \"limitador-operator-controller-manager-859d7b78bd-grv6b\" (UID: \"ffc04543-a138-455c-be42-0497ca3b93e2\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-grv6b" Jul 09 16:23:13.454603 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:13.454578 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhgxf\" (UniqueName: \"kubernetes.io/projected/ffc04543-a138-455c-be42-0497ca3b93e2-kube-api-access-qhgxf\") pod \"limitador-operator-controller-manager-859d7b78bd-grv6b\" (UID: \"ffc04543-a138-455c-be42-0497ca3b93e2\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-grv6b" Jul 09 16:23:13.475673 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:13.475580 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-vdkcb" event={"ID":"e7fce909-26bb-4713-bac8-a2a046e4c5b0","Type":"ContainerStarted","Data":"2a7ecb20890d573413ae45d146efdf22a15c771c8c86050d656ec1f52e801689"} Jul 09 16:23:13.476076 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:13.475708 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-vdkcb" Jul 09 16:23:13.503230 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:13.503174 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-vdkcb" podStartSLOduration=1.148901028 podStartE2EDuration="3.503156474s" totalImagesPullingTime="2.354255446s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:23:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:23:13.502278776 +0000 UTC m=+478.695385158" watchObservedRunningTime="2026-07-09 16:23:13.503156474 +0000 UTC m=+478.696262854" Jul 09 16:23:13.536604 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:13.536557 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-grv6b" Jul 09 16:23:13.681714 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:13.681685 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-grv6b"] Jul 09 16:23:13.683533 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:23:13.683500 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffc04543_a138_455c_be42_0497ca3b93e2.slice/crio-ace6712c2921c59c2ee79f7dc6a368777a603a1d39ac92b36efcdcc9713222a9 WatchSource:0}: Error finding container ace6712c2921c59c2ee79f7dc6a368777a603a1d39ac92b36efcdcc9713222a9: Status 404 returned error can't find the container with id ace6712c2921c59c2ee79f7dc6a368777a603a1d39ac92b36efcdcc9713222a9 Jul 09 16:23:14.480994 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:14.480962 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-grv6b" event={"ID":"ffc04543-a138-455c-be42-0497ca3b93e2","Type":"ContainerStarted","Data":"ace6712c2921c59c2ee79f7dc6a368777a603a1d39ac92b36efcdcc9713222a9"} Jul 09 16:23:17.496340 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:17.496298 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-grv6b" event={"ID":"ffc04543-a138-455c-be42-0497ca3b93e2","Type":"ContainerStarted","Data":"374d77bc6c1ae2bc7fb1392c42b549671861f3406f90d2bc8bfbc0b6415f146b"} Jul 09 16:23:17.496749 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:17.496417 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-grv6b" Jul 09 16:23:17.518822 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:17.518766 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-grv6b" podStartSLOduration=1.680287958 podStartE2EDuration="4.518748965s" totalImagesPullingTime="2.838461007s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:23:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:23:17.517480259 +0000 UTC m=+482.710586640" watchObservedRunningTime="2026-07-09 16:23:17.518748965 +0000 UTC m=+482.711855403" Jul 09 16:23:17.755471 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:17.755376 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ht6r7"] Jul 09 16:23:17.758932 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:17.758906 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ht6r7" Jul 09 16:23:17.762040 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:17.762021 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-p94jf\"" Jul 09 16:23:17.772215 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:17.772191 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ht6r7"] Jul 09 16:23:17.877226 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:17.877188 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d4dcd1d8-c53e-4296-9e8c-a0538895f0b1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-ht6r7\" (UID: \"d4dcd1d8-c53e-4296-9e8c-a0538895f0b1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ht6r7" Jul 09 16:23:17.877226 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:17.877229 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28gcz\" (UniqueName: \"kubernetes.io/projected/d4dcd1d8-c53e-4296-9e8c-a0538895f0b1-kube-api-access-28gcz\") pod \"kuadrant-operator-controller-manager-8684c9ff49-ht6r7\" (UID: \"d4dcd1d8-c53e-4296-9e8c-a0538895f0b1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ht6r7" Jul 09 16:23:17.978482 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:17.978446 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d4dcd1d8-c53e-4296-9e8c-a0538895f0b1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-ht6r7\" (UID: \"d4dcd1d8-c53e-4296-9e8c-a0538895f0b1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ht6r7" Jul 09 16:23:17.978482 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:17.978486 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28gcz\" (UniqueName: \"kubernetes.io/projected/d4dcd1d8-c53e-4296-9e8c-a0538895f0b1-kube-api-access-28gcz\") pod \"kuadrant-operator-controller-manager-8684c9ff49-ht6r7\" (UID: \"d4dcd1d8-c53e-4296-9e8c-a0538895f0b1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ht6r7" Jul 09 16:23:17.978855 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:17.978832 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d4dcd1d8-c53e-4296-9e8c-a0538895f0b1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-ht6r7\" (UID: \"d4dcd1d8-c53e-4296-9e8c-a0538895f0b1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ht6r7" Jul 09 16:23:17.987962 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:17.987938 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28gcz\" (UniqueName: \"kubernetes.io/projected/d4dcd1d8-c53e-4296-9e8c-a0538895f0b1-kube-api-access-28gcz\") pod \"kuadrant-operator-controller-manager-8684c9ff49-ht6r7\" (UID: \"d4dcd1d8-c53e-4296-9e8c-a0538895f0b1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ht6r7" Jul 09 16:23:18.070074 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:18.069991 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ht6r7" Jul 09 16:23:18.214461 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:18.214436 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ht6r7"] Jul 09 16:23:18.217204 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:23:18.217169 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4dcd1d8_c53e_4296_9e8c_a0538895f0b1.slice/crio-b2d05024242f4c5d48de22b82a4078de55c7a124eb5286ceb3ac5d1a7a314e8e WatchSource:0}: Error finding container b2d05024242f4c5d48de22b82a4078de55c7a124eb5286ceb3ac5d1a7a314e8e: Status 404 returned error can't find the container with id b2d05024242f4c5d48de22b82a4078de55c7a124eb5286ceb3ac5d1a7a314e8e Jul 09 16:23:18.502253 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:18.502211 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ht6r7" event={"ID":"d4dcd1d8-c53e-4296-9e8c-a0538895f0b1","Type":"ContainerStarted","Data":"b2d05024242f4c5d48de22b82a4078de55c7a124eb5286ceb3ac5d1a7a314e8e"} Jul 09 16:23:24.483664 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:24.483629 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-vdkcb" Jul 09 16:23:24.535055 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:24.535006 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ht6r7" event={"ID":"d4dcd1d8-c53e-4296-9e8c-a0538895f0b1","Type":"ContainerStarted","Data":"959d0be5f5a3761de2d40266aac82d564dd0bf70da45690cb6d9058b3aa3cca2"} Jul 09 16:23:24.535236 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:24.535128 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ht6r7" Jul 09 16:23:28.505117 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:28.505080 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-grv6b" Jul 09 16:23:28.522764 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:28.522701 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ht6r7" podStartSLOduration=5.727980627 podStartE2EDuration="11.522684545s" totalImagesPullingTime="5.794703918s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:23:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:23:24.568445625 +0000 UTC m=+489.761552006" watchObservedRunningTime="2026-07-09 16:23:28.522684545 +0000 UTC m=+493.715790925" Jul 09 16:23:35.542183 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:35.542155 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ht6r7" Jul 09 16:23:37.472884 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:37.472851 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ht6r7"] Jul 09 16:23:37.473264 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:37.473059 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ht6r7" podUID="d4dcd1d8-c53e-4296-9e8c-a0538895f0b1" containerName="manager" containerID="cri-o://959d0be5f5a3761de2d40266aac82d564dd0bf70da45690cb6d9058b3aa3cca2" gracePeriod=2 Jul 09 16:23:37.490085 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:37.490056 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ht6r7"] Jul 09 16:23:37.502181 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:23:37.502143 2569 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"d4dcd1d8-c53e-4296-9e8c-a0538895f0b1\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-ht6r7\": pods \"kuadrant-operator-controller-manager-8684c9ff49-ht6r7\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ht6r7" Jul 09 16:23:37.505270 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:37.505239 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-grv6b"] Jul 09 16:23:37.505507 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:37.505483 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-grv6b" podUID="ffc04543-a138-455c-be42-0497ca3b93e2" containerName="manager" containerID="cri-o://374d77bc6c1ae2bc7fb1392c42b549671861f3406f90d2bc8bfbc0b6415f146b" gracePeriod=2 Jul 09 16:23:37.506740 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:37.506715 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hfp2r"] Jul 09 16:23:37.507090 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:37.507076 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4dcd1d8-c53e-4296-9e8c-a0538895f0b1" containerName="manager" Jul 09 16:23:37.507143 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:37.507093 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4dcd1d8-c53e-4296-9e8c-a0538895f0b1" containerName="manager" Jul 09 16:23:37.507182 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:37.507158 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4dcd1d8-c53e-4296-9e8c-a0538895f0b1" containerName="manager" Jul 09 16:23:37.510784 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:37.510759 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hfp2r" Jul 09 16:23:37.521527 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:37.521499 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-grv6b"] Jul 09 16:23:37.528733 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:37.528705 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hfp2r"] Jul 09 16:23:37.528999 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:23:37.528959 2569 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"ffc04543-a138-455c-be42-0497ca3b93e2\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-859d7b78bd-grv6b\": pods \"limitador-operator-controller-manager-859d7b78bd-grv6b\" not found" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-grv6b" Jul 09 16:23:37.531530 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:23:37.531493 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-ht6r7\" is forbidden: User \"system:node:ip-10-0-135-114.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-114.ec2.internal' and this object" podUID="d4dcd1d8-c53e-4296-9e8c-a0538895f0b1" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ht6r7" Jul 09 16:23:37.534001 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:23:37.533970 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-grv6b\" is forbidden: User \"system:node:ip-10-0-135-114.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-114.ec2.internal' and this object" podUID="ffc04543-a138-455c-be42-0497ca3b93e2" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-grv6b" Jul 09 16:23:37.543944 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:37.543917 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q2k74"] Jul 09 16:23:37.544292 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:37.544279 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffc04543-a138-455c-be42-0497ca3b93e2" containerName="manager" Jul 09 16:23:37.544352 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:37.544293 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffc04543-a138-455c-be42-0497ca3b93e2" containerName="manager" Jul 09 16:23:37.544352 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:37.544349 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffc04543-a138-455c-be42-0497ca3b93e2" containerName="manager" Jul 09 16:23:37.547929 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:37.547911 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q2k74" Jul 09 16:23:37.562622 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:23:37.562586 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-ht6r7\" is forbidden: User \"system:node:ip-10-0-135-114.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-114.ec2.internal' and this object" podUID="d4dcd1d8-c53e-4296-9e8c-a0538895f0b1" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ht6r7" Jul 09 16:23:37.566130 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:37.566099 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q2k74"] Jul 09 16:23:37.582457 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:23:37.582423 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-ht6r7\" is forbidden: User \"system:node:ip-10-0-135-114.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-114.ec2.internal' and this object" podUID="d4dcd1d8-c53e-4296-9e8c-a0538895f0b1" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ht6r7" Jul 09 16:23:37.584606 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:23:37.584584 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-grv6b\" is forbidden: User \"system:node:ip-10-0-135-114.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-114.ec2.internal' and this object" podUID="ffc04543-a138-455c-be42-0497ca3b93e2" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-grv6b" Jul 09 16:23:37.596662 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:37.596638 2569 generic.go:356] "Generic (PLEG): container finished" podID="d4dcd1d8-c53e-4296-9e8c-a0538895f0b1" containerID="959d0be5f5a3761de2d40266aac82d564dd0bf70da45690cb6d9058b3aa3cca2" exitCode=0 Jul 09 16:23:37.648684 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:37.648644 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw469\" (UniqueName: \"kubernetes.io/projected/9b9c3dc8-c38d-4289-a8bb-fe5059fdb845-kube-api-access-nw469\") pod \"kuadrant-operator-controller-manager-8684c9ff49-hfp2r\" (UID: \"9b9c3dc8-c38d-4289-a8bb-fe5059fdb845\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hfp2r" Jul 09 16:23:37.648868 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:37.648701 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9b9c3dc8-c38d-4289-a8bb-fe5059fdb845-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-hfp2r\" (UID: \"9b9c3dc8-c38d-4289-a8bb-fe5059fdb845\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hfp2r" Jul 09 16:23:37.648868 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:37.648796 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlmgx\" (UniqueName: \"kubernetes.io/projected/97412e3e-1a86-48b2-b77e-c8620809630d-kube-api-access-jlmgx\") pod \"limitador-operator-controller-manager-859d7b78bd-q2k74\" (UID: \"97412e3e-1a86-48b2-b77e-c8620809630d\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q2k74" Jul 09 16:23:37.743167 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:37.743144 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ht6r7" Jul 09 16:23:37.745613 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:23:37.745584 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-grv6b\" is forbidden: User \"system:node:ip-10-0-135-114.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-114.ec2.internal' and this object" podUID="ffc04543-a138-455c-be42-0497ca3b93e2" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-grv6b" Jul 09 16:23:37.747820 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:23:37.747794 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-ht6r7\" is forbidden: User \"system:node:ip-10-0-135-114.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-114.ec2.internal' and this object" podUID="d4dcd1d8-c53e-4296-9e8c-a0538895f0b1" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ht6r7" Jul 09 16:23:37.749784 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:37.749760 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlmgx\" (UniqueName: \"kubernetes.io/projected/97412e3e-1a86-48b2-b77e-c8620809630d-kube-api-access-jlmgx\") pod \"limitador-operator-controller-manager-859d7b78bd-q2k74\" (UID: \"97412e3e-1a86-48b2-b77e-c8620809630d\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q2k74" Jul 09 16:23:37.749934 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:37.749915 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw469\" (UniqueName: \"kubernetes.io/projected/9b9c3dc8-c38d-4289-a8bb-fe5059fdb845-kube-api-access-nw469\") pod \"kuadrant-operator-controller-manager-8684c9ff49-hfp2r\" (UID: \"9b9c3dc8-c38d-4289-a8bb-fe5059fdb845\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hfp2r" Jul 09 16:23:37.749989 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:37.749946 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9b9c3dc8-c38d-4289-a8bb-fe5059fdb845-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-hfp2r\" (UID: \"9b9c3dc8-c38d-4289-a8bb-fe5059fdb845\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hfp2r" Jul 09 16:23:37.750309 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:37.750291 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9b9c3dc8-c38d-4289-a8bb-fe5059fdb845-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-hfp2r\" (UID: \"9b9c3dc8-c38d-4289-a8bb-fe5059fdb845\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hfp2r" Jul 09 16:23:37.755183 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:37.755165 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-grv6b" Jul 09 16:23:37.758537 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:23:37.758512 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-grv6b\" is forbidden: User \"system:node:ip-10-0-135-114.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-114.ec2.internal' and this object" podUID="ffc04543-a138-455c-be42-0497ca3b93e2" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-grv6b" Jul 09 16:23:37.760192 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:37.760172 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw469\" (UniqueName: \"kubernetes.io/projected/9b9c3dc8-c38d-4289-a8bb-fe5059fdb845-kube-api-access-nw469\") pod \"kuadrant-operator-controller-manager-8684c9ff49-hfp2r\" (UID: \"9b9c3dc8-c38d-4289-a8bb-fe5059fdb845\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hfp2r" Jul 09 16:23:37.760557 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:23:37.760532 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-ht6r7\" is forbidden: User \"system:node:ip-10-0-135-114.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-114.ec2.internal' and this object" podUID="d4dcd1d8-c53e-4296-9e8c-a0538895f0b1" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ht6r7" Jul 09 16:23:37.761711 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:37.761691 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlmgx\" (UniqueName: \"kubernetes.io/projected/97412e3e-1a86-48b2-b77e-c8620809630d-kube-api-access-jlmgx\") pod \"limitador-operator-controller-manager-859d7b78bd-q2k74\" (UID: \"97412e3e-1a86-48b2-b77e-c8620809630d\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q2k74" Jul 09 16:23:37.850645 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:37.850612 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-28gcz\" (UniqueName: \"kubernetes.io/projected/d4dcd1d8-c53e-4296-9e8c-a0538895f0b1-kube-api-access-28gcz\") pod \"d4dcd1d8-c53e-4296-9e8c-a0538895f0b1\" (UID: \"d4dcd1d8-c53e-4296-9e8c-a0538895f0b1\") " Jul 09 16:23:37.850837 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:37.850678 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qhgxf\" (UniqueName: \"kubernetes.io/projected/ffc04543-a138-455c-be42-0497ca3b93e2-kube-api-access-qhgxf\") pod \"ffc04543-a138-455c-be42-0497ca3b93e2\" (UID: \"ffc04543-a138-455c-be42-0497ca3b93e2\") " Jul 09 16:23:37.850837 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:37.850727 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d4dcd1d8-c53e-4296-9e8c-a0538895f0b1-extensions-socket-volume\") pod \"d4dcd1d8-c53e-4296-9e8c-a0538895f0b1\" (UID: \"d4dcd1d8-c53e-4296-9e8c-a0538895f0b1\") " Jul 09 16:23:37.851207 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:37.851179 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4dcd1d8-c53e-4296-9e8c-a0538895f0b1-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "d4dcd1d8-c53e-4296-9e8c-a0538895f0b1" (UID: "d4dcd1d8-c53e-4296-9e8c-a0538895f0b1"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:23:37.852778 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:37.852754 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4dcd1d8-c53e-4296-9e8c-a0538895f0b1-kube-api-access-28gcz" (OuterVolumeSpecName: "kube-api-access-28gcz") pod "d4dcd1d8-c53e-4296-9e8c-a0538895f0b1" (UID: "d4dcd1d8-c53e-4296-9e8c-a0538895f0b1"). InnerVolumeSpecName "kube-api-access-28gcz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 16:23:37.852778 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:37.852762 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffc04543-a138-455c-be42-0497ca3b93e2-kube-api-access-qhgxf" (OuterVolumeSpecName: "kube-api-access-qhgxf") pod "ffc04543-a138-455c-be42-0497ca3b93e2" (UID: "ffc04543-a138-455c-be42-0497ca3b93e2"). InnerVolumeSpecName "kube-api-access-qhgxf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 16:23:37.929306 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:37.929274 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hfp2r" Jul 09 16:23:37.935543 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:37.935520 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q2k74" Jul 09 16:23:37.951868 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:37.951837 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28gcz\" (UniqueName: \"kubernetes.io/projected/d4dcd1d8-c53e-4296-9e8c-a0538895f0b1-kube-api-access-28gcz\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:23:37.951868 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:37.951864 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhgxf\" (UniqueName: \"kubernetes.io/projected/ffc04543-a138-455c-be42-0497ca3b93e2-kube-api-access-qhgxf\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:23:37.952077 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:37.951891 2569 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d4dcd1d8-c53e-4296-9e8c-a0538895f0b1-extensions-socket-volume\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:23:38.087572 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:38.087549 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hfp2r"] Jul 09 16:23:38.089978 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:23:38.089948 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b9c3dc8_c38d_4289_a8bb_fe5059fdb845.slice/crio-b277c8dede9d6aa27e8d2941b8926de309bfce4c64393467cfe3c85ff1378578 WatchSource:0}: Error finding container b277c8dede9d6aa27e8d2941b8926de309bfce4c64393467cfe3c85ff1378578: Status 404 returned error can't find the container with id b277c8dede9d6aa27e8d2941b8926de309bfce4c64393467cfe3c85ff1378578 Jul 09 16:23:38.110059 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:38.110040 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q2k74"] Jul 09 16:23:38.111097 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:23:38.111074 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97412e3e_1a86_48b2_b77e_c8620809630d.slice/crio-f9952ca8d78de7d36077a2d742ef481fcf954eadb56f545448f09e6270bd4833 WatchSource:0}: Error finding container f9952ca8d78de7d36077a2d742ef481fcf954eadb56f545448f09e6270bd4833: Status 404 returned error can't find the container with id f9952ca8d78de7d36077a2d742ef481fcf954eadb56f545448f09e6270bd4833 Jul 09 16:23:38.605672 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:38.605637 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q2k74" event={"ID":"97412e3e-1a86-48b2-b77e-c8620809630d","Type":"ContainerStarted","Data":"e770491b3a9c36ea25ac1c1bf03efa39375d834d5b6377ba51f66f7b91cee9fb"} Jul 09 16:23:38.606253 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:38.605702 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q2k74" event={"ID":"97412e3e-1a86-48b2-b77e-c8620809630d","Type":"ContainerStarted","Data":"f9952ca8d78de7d36077a2d742ef481fcf954eadb56f545448f09e6270bd4833"} Jul 09 16:23:38.606253 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:38.605727 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q2k74" Jul 09 16:23:38.607011 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:38.606992 2569 generic.go:356] "Generic (PLEG): container finished" podID="ffc04543-a138-455c-be42-0497ca3b93e2" containerID="374d77bc6c1ae2bc7fb1392c42b549671861f3406f90d2bc8bfbc0b6415f146b" exitCode=0 Jul 09 16:23:38.607134 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:38.607029 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-grv6b" Jul 09 16:23:38.607134 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:38.607068 2569 scope.go:117] "RemoveContainer" containerID="374d77bc6c1ae2bc7fb1392c42b549671861f3406f90d2bc8bfbc0b6415f146b" Jul 09 16:23:38.615794 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:38.615717 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ht6r7" Jul 09 16:23:38.617932 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:38.617829 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hfp2r" event={"ID":"9b9c3dc8-c38d-4289-a8bb-fe5059fdb845","Type":"ContainerStarted","Data":"c15833a7d3a61c3969f794b1f1c0c9be59ff809e61adeac96035343df3607f8b"} Jul 09 16:23:38.618058 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:38.617937 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hfp2r" event={"ID":"9b9c3dc8-c38d-4289-a8bb-fe5059fdb845","Type":"ContainerStarted","Data":"b277c8dede9d6aa27e8d2941b8926de309bfce4c64393467cfe3c85ff1378578"} Jul 09 16:23:38.618108 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:38.618078 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hfp2r" Jul 09 16:23:38.625105 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:38.624318 2569 scope.go:117] "RemoveContainer" containerID="374d77bc6c1ae2bc7fb1392c42b549671861f3406f90d2bc8bfbc0b6415f146b" Jul 09 16:23:38.625105 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:23:38.624657 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"374d77bc6c1ae2bc7fb1392c42b549671861f3406f90d2bc8bfbc0b6415f146b\": container with ID starting with 374d77bc6c1ae2bc7fb1392c42b549671861f3406f90d2bc8bfbc0b6415f146b not found: ID does not exist" containerID="374d77bc6c1ae2bc7fb1392c42b549671861f3406f90d2bc8bfbc0b6415f146b" Jul 09 16:23:38.625105 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:38.624696 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"374d77bc6c1ae2bc7fb1392c42b549671861f3406f90d2bc8bfbc0b6415f146b"} err="failed to get container status \"374d77bc6c1ae2bc7fb1392c42b549671861f3406f90d2bc8bfbc0b6415f146b\": rpc error: code = NotFound desc = could not find container \"374d77bc6c1ae2bc7fb1392c42b549671861f3406f90d2bc8bfbc0b6415f146b\": container with ID starting with 374d77bc6c1ae2bc7fb1392c42b549671861f3406f90d2bc8bfbc0b6415f146b not found: ID does not exist" Jul 09 16:23:38.625105 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:38.624722 2569 scope.go:117] "RemoveContainer" containerID="959d0be5f5a3761de2d40266aac82d564dd0bf70da45690cb6d9058b3aa3cca2" Jul 09 16:23:38.634488 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:38.634446 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q2k74" podStartSLOduration=1.634430975 podStartE2EDuration="1.634430975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:23:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:23:38.631214404 +0000 UTC m=+503.824320784" watchObservedRunningTime="2026-07-09 16:23:38.634430975 +0000 UTC m=+503.827537356" Jul 09 16:23:38.637149 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:23:38.637124 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-ht6r7\" is forbidden: User \"system:node:ip-10-0-135-114.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-114.ec2.internal' and this object" podUID="d4dcd1d8-c53e-4296-9e8c-a0538895f0b1" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ht6r7" Jul 09 16:23:38.639512 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:23:38.639491 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-grv6b\" is forbidden: User \"system:node:ip-10-0-135-114.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-114.ec2.internal' and this object" podUID="ffc04543-a138-455c-be42-0497ca3b93e2" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-grv6b" Jul 09 16:23:38.641772 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:23:38.641751 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-ht6r7\" is forbidden: User \"system:node:ip-10-0-135-114.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-114.ec2.internal' and this object" podUID="d4dcd1d8-c53e-4296-9e8c-a0538895f0b1" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ht6r7" Jul 09 16:23:38.649150 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:23:38.649122 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-grv6b\" is forbidden: User \"system:node:ip-10-0-135-114.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-114.ec2.internal' and this object" podUID="ffc04543-a138-455c-be42-0497ca3b93e2" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-grv6b" Jul 09 16:23:38.699935 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:38.699865 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hfp2r" podStartSLOduration=1.6998472420000001 podStartE2EDuration="1.699847242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:23:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:23:38.698073784 +0000 UTC m=+503.891180179" watchObservedRunningTime="2026-07-09 16:23:38.699847242 +0000 UTC m=+503.892953666" Jul 09 16:23:39.390602 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:39.390565 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4dcd1d8-c53e-4296-9e8c-a0538895f0b1" path="/var/lib/kubelet/pods/d4dcd1d8-c53e-4296-9e8c-a0538895f0b1/volumes" Jul 09 16:23:39.390921 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:39.390908 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ffc04543-a138-455c-be42-0497ca3b93e2" path="/var/lib/kubelet/pods/ffc04543-a138-455c-be42-0497ca3b93e2/volumes" Jul 09 16:23:49.627474 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:49.627444 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hfp2r" Jul 09 16:23:49.627982 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:49.627595 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q2k74" Jul 09 16:23:53.937939 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:53.937899 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hfp2r"] Jul 09 16:23:53.938402 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:53.938131 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hfp2r" podUID="9b9c3dc8-c38d-4289-a8bb-fe5059fdb845" containerName="manager" containerID="cri-o://c15833a7d3a61c3969f794b1f1c0c9be59ff809e61adeac96035343df3607f8b" gracePeriod=10 Jul 09 16:23:55.381087 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:55.381065 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hfp2r" Jul 09 16:23:55.500368 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:55.500284 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9b9c3dc8-c38d-4289-a8bb-fe5059fdb845-extensions-socket-volume\") pod \"9b9c3dc8-c38d-4289-a8bb-fe5059fdb845\" (UID: \"9b9c3dc8-c38d-4289-a8bb-fe5059fdb845\") " Jul 09 16:23:55.500368 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:55.500340 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nw469\" (UniqueName: \"kubernetes.io/projected/9b9c3dc8-c38d-4289-a8bb-fe5059fdb845-kube-api-access-nw469\") pod \"9b9c3dc8-c38d-4289-a8bb-fe5059fdb845\" (UID: \"9b9c3dc8-c38d-4289-a8bb-fe5059fdb845\") " Jul 09 16:23:55.500685 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:55.500662 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b9c3dc8-c38d-4289-a8bb-fe5059fdb845-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "9b9c3dc8-c38d-4289-a8bb-fe5059fdb845" (UID: "9b9c3dc8-c38d-4289-a8bb-fe5059fdb845"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:23:55.502405 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:55.502378 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b9c3dc8-c38d-4289-a8bb-fe5059fdb845-kube-api-access-nw469" (OuterVolumeSpecName: "kube-api-access-nw469") pod "9b9c3dc8-c38d-4289-a8bb-fe5059fdb845" (UID: "9b9c3dc8-c38d-4289-a8bb-fe5059fdb845"). InnerVolumeSpecName "kube-api-access-nw469". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 16:23:55.601718 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:55.601683 2569 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9b9c3dc8-c38d-4289-a8bb-fe5059fdb845-extensions-socket-volume\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:23:55.601718 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:55.601711 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nw469\" (UniqueName: \"kubernetes.io/projected/9b9c3dc8-c38d-4289-a8bb-fe5059fdb845-kube-api-access-nw469\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:23:55.702151 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:55.702113 2569 generic.go:356] "Generic (PLEG): container finished" podID="9b9c3dc8-c38d-4289-a8bb-fe5059fdb845" containerID="c15833a7d3a61c3969f794b1f1c0c9be59ff809e61adeac96035343df3607f8b" exitCode=0 Jul 09 16:23:55.702323 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:55.702166 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hfp2r" Jul 09 16:23:55.702323 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:55.702186 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hfp2r" event={"ID":"9b9c3dc8-c38d-4289-a8bb-fe5059fdb845","Type":"ContainerDied","Data":"c15833a7d3a61c3969f794b1f1c0c9be59ff809e61adeac96035343df3607f8b"} Jul 09 16:23:55.702323 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:55.702268 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hfp2r" event={"ID":"9b9c3dc8-c38d-4289-a8bb-fe5059fdb845","Type":"ContainerDied","Data":"b277c8dede9d6aa27e8d2941b8926de309bfce4c64393467cfe3c85ff1378578"} Jul 09 16:23:55.702323 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:55.702291 2569 scope.go:117] "RemoveContainer" containerID="c15833a7d3a61c3969f794b1f1c0c9be59ff809e61adeac96035343df3607f8b" Jul 09 16:23:55.713030 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:55.713005 2569 scope.go:117] "RemoveContainer" containerID="c15833a7d3a61c3969f794b1f1c0c9be59ff809e61adeac96035343df3607f8b" Jul 09 16:23:55.713272 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:23:55.713253 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c15833a7d3a61c3969f794b1f1c0c9be59ff809e61adeac96035343df3607f8b\": container with ID starting with c15833a7d3a61c3969f794b1f1c0c9be59ff809e61adeac96035343df3607f8b not found: ID does not exist" containerID="c15833a7d3a61c3969f794b1f1c0c9be59ff809e61adeac96035343df3607f8b" Jul 09 16:23:55.713313 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:55.713281 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c15833a7d3a61c3969f794b1f1c0c9be59ff809e61adeac96035343df3607f8b"} err="failed to get container status \"c15833a7d3a61c3969f794b1f1c0c9be59ff809e61adeac96035343df3607f8b\": rpc error: code = NotFound desc = could not find container \"c15833a7d3a61c3969f794b1f1c0c9be59ff809e61adeac96035343df3607f8b\": container with ID starting with c15833a7d3a61c3969f794b1f1c0c9be59ff809e61adeac96035343df3607f8b not found: ID does not exist" Jul 09 16:23:55.754676 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:55.754599 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hfp2r"] Jul 09 16:23:55.763614 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:55.763586 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hfp2r"] Jul 09 16:23:57.389342 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:23:57.389298 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b9c3dc8-c38d-4289-a8bb-fe5059fdb845" path="/var/lib/kubelet/pods/9b9c3dc8-c38d-4289-a8bb-fe5059fdb845/volumes" Jul 09 16:24:12.135584 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:12.135496 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-nv6j7"] Jul 09 16:24:12.135955 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:12.135809 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b9c3dc8-c38d-4289-a8bb-fe5059fdb845" containerName="manager" Jul 09 16:24:12.135955 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:12.135820 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b9c3dc8-c38d-4289-a8bb-fe5059fdb845" containerName="manager" Jul 09 16:24:12.135955 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:12.135913 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b9c3dc8-c38d-4289-a8bb-fe5059fdb845" containerName="manager" Jul 09 16:24:12.138895 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:12.138859 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-nv6j7" Jul 09 16:24:12.141673 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:12.141650 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 09 16:24:12.141809 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:12.141650 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-97vnj\"" Jul 09 16:24:12.149040 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:12.149013 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-nv6j7"] Jul 09 16:24:12.233448 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:12.233407 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-nv6j7"] Jul 09 16:24:12.235503 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:12.235475 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f455fb69-f8be-4cca-ab76-2bb6bd551c49-config-file\") pod \"limitador-limitador-64c94fd64f-nv6j7\" (UID: \"f455fb69-f8be-4cca-ab76-2bb6bd551c49\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-nv6j7" Jul 09 16:24:12.235656 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:12.235531 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmxcb\" (UniqueName: \"kubernetes.io/projected/f455fb69-f8be-4cca-ab76-2bb6bd551c49-kube-api-access-mmxcb\") pod \"limitador-limitador-64c94fd64f-nv6j7\" (UID: \"f455fb69-f8be-4cca-ab76-2bb6bd551c49\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-nv6j7" Jul 09 16:24:12.336771 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:12.336731 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f455fb69-f8be-4cca-ab76-2bb6bd551c49-config-file\") pod \"limitador-limitador-64c94fd64f-nv6j7\" (UID: \"f455fb69-f8be-4cca-ab76-2bb6bd551c49\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-nv6j7" Jul 09 16:24:12.336771 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:12.336776 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmxcb\" (UniqueName: \"kubernetes.io/projected/f455fb69-f8be-4cca-ab76-2bb6bd551c49-kube-api-access-mmxcb\") pod \"limitador-limitador-64c94fd64f-nv6j7\" (UID: \"f455fb69-f8be-4cca-ab76-2bb6bd551c49\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-nv6j7" Jul 09 16:24:12.337417 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:12.337396 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f455fb69-f8be-4cca-ab76-2bb6bd551c49-config-file\") pod \"limitador-limitador-64c94fd64f-nv6j7\" (UID: \"f455fb69-f8be-4cca-ab76-2bb6bd551c49\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-nv6j7" Jul 09 16:24:12.345430 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:12.345402 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmxcb\" (UniqueName: \"kubernetes.io/projected/f455fb69-f8be-4cca-ab76-2bb6bd551c49-kube-api-access-mmxcb\") pod \"limitador-limitador-64c94fd64f-nv6j7\" (UID: \"f455fb69-f8be-4cca-ab76-2bb6bd551c49\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-nv6j7" Jul 09 16:24:12.450221 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:12.450136 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-nv6j7" Jul 09 16:24:12.598417 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:12.598380 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-nv6j7"] Jul 09 16:24:12.601730 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:24:12.601703 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf455fb69_f8be_4cca_ab76_2bb6bd551c49.slice/crio-abeefac74b9c00bf1f40a64ff05ce622fac5352e9893e90607bb2f4440d36b5a WatchSource:0}: Error finding container abeefac74b9c00bf1f40a64ff05ce622fac5352e9893e90607bb2f4440d36b5a: Status 404 returned error can't find the container with id abeefac74b9c00bf1f40a64ff05ce622fac5352e9893e90607bb2f4440d36b5a Jul 09 16:24:12.780584 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:12.780488 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-nv6j7" event={"ID":"f455fb69-f8be-4cca-ab76-2bb6bd551c49","Type":"ContainerStarted","Data":"abeefac74b9c00bf1f40a64ff05ce622fac5352e9893e90607bb2f4440d36b5a"} Jul 09 16:24:13.187042 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:13.187009 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-n94dc"] Jul 09 16:24:13.191257 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:13.191239 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-n94dc" Jul 09 16:24:13.193956 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:13.193924 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-2jwtt\"" Jul 09 16:24:13.204073 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:13.204045 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-n94dc"] Jul 09 16:24:13.244274 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:13.244231 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6sfg\" (UniqueName: \"kubernetes.io/projected/c3e01d96-31f6-4659-8533-37b194105b99-kube-api-access-r6sfg\") pod \"authorino-75d556bd77-n94dc\" (UID: \"c3e01d96-31f6-4659-8533-37b194105b99\") " pod="kuadrant-system/authorino-75d556bd77-n94dc" Jul 09 16:24:13.345707 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:13.345664 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6sfg\" (UniqueName: \"kubernetes.io/projected/c3e01d96-31f6-4659-8533-37b194105b99-kube-api-access-r6sfg\") pod \"authorino-75d556bd77-n94dc\" (UID: \"c3e01d96-31f6-4659-8533-37b194105b99\") " pod="kuadrant-system/authorino-75d556bd77-n94dc" Jul 09 16:24:13.354894 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:13.354845 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6sfg\" (UniqueName: \"kubernetes.io/projected/c3e01d96-31f6-4659-8533-37b194105b99-kube-api-access-r6sfg\") pod \"authorino-75d556bd77-n94dc\" (UID: \"c3e01d96-31f6-4659-8533-37b194105b99\") " pod="kuadrant-system/authorino-75d556bd77-n94dc" Jul 09 16:24:13.501322 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:13.501232 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-n94dc" Jul 09 16:24:13.711897 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:13.711833 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-n94dc"] Jul 09 16:24:13.716866 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:24:13.716769 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc3e01d96_31f6_4659_8533_37b194105b99.slice/crio-4c409bfdfabaf17f5bffeee55b485520977b6a1599f4daaa96c9e1dd38ae3c30 WatchSource:0}: Error finding container 4c409bfdfabaf17f5bffeee55b485520977b6a1599f4daaa96c9e1dd38ae3c30: Status 404 returned error can't find the container with id 4c409bfdfabaf17f5bffeee55b485520977b6a1599f4daaa96c9e1dd38ae3c30 Jul 09 16:24:13.788292 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:13.788201 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-n94dc" event={"ID":"c3e01d96-31f6-4659-8533-37b194105b99","Type":"ContainerStarted","Data":"4c409bfdfabaf17f5bffeee55b485520977b6a1599f4daaa96c9e1dd38ae3c30"} Jul 09 16:24:15.802222 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:15.802134 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-nv6j7" event={"ID":"f455fb69-f8be-4cca-ab76-2bb6bd551c49","Type":"ContainerStarted","Data":"3f2a6d412d391bc1ab8471aa2cc98bc09e3df49bacc95060955f59259006bbbd"} Jul 09 16:24:15.803022 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:15.802954 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-nv6j7" Jul 09 16:24:18.818317 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:18.818278 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-n94dc" event={"ID":"c3e01d96-31f6-4659-8533-37b194105b99","Type":"ContainerStarted","Data":"faa558d3c808adbd624bdff65c62cf0710803f9ccc1d895fd07940bdb7833452"} Jul 09 16:24:18.834568 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:18.834521 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-nv6j7" podStartSLOduration=3.869716348 podStartE2EDuration="6.834504089s" totalImagesPullingTime="2.964787741s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:24:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:24:15.851134084 +0000 UTC m=+541.044240469" watchObservedRunningTime="2026-07-09 16:24:18.834504089 +0000 UTC m=+544.027610469" Jul 09 16:24:18.835010 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:18.834986 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-n94dc" podStartSLOduration=1.560540372 podStartE2EDuration="5.834970182s" totalImagesPullingTime="4.27442981s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:24:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:24:18.833549537 +0000 UTC m=+544.026655919" watchObservedRunningTime="2026-07-09 16:24:18.834970182 +0000 UTC m=+544.028076563" Jul 09 16:24:27.813813 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:27.813781 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-nv6j7" Jul 09 16:24:29.648757 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:29.648719 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-nv6j7"] Jul 09 16:24:29.649269 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:29.648938 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-nv6j7" podUID="f455fb69-f8be-4cca-ab76-2bb6bd551c49" containerName="limitador" containerID="cri-o://3f2a6d412d391bc1ab8471aa2cc98bc09e3df49bacc95060955f59259006bbbd" gracePeriod=30 Jul 09 16:24:30.205248 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:30.205222 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-nv6j7" Jul 09 16:24:30.302337 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:30.302248 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mmxcb\" (UniqueName: \"kubernetes.io/projected/f455fb69-f8be-4cca-ab76-2bb6bd551c49-kube-api-access-mmxcb\") pod \"f455fb69-f8be-4cca-ab76-2bb6bd551c49\" (UID: \"f455fb69-f8be-4cca-ab76-2bb6bd551c49\") " Jul 09 16:24:30.302337 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:30.302309 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f455fb69-f8be-4cca-ab76-2bb6bd551c49-config-file\") pod \"f455fb69-f8be-4cca-ab76-2bb6bd551c49\" (UID: \"f455fb69-f8be-4cca-ab76-2bb6bd551c49\") " Jul 09 16:24:30.302682 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:30.302658 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f455fb69-f8be-4cca-ab76-2bb6bd551c49-config-file" (OuterVolumeSpecName: "config-file") pod "f455fb69-f8be-4cca-ab76-2bb6bd551c49" (UID: "f455fb69-f8be-4cca-ab76-2bb6bd551c49"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 16:24:30.304396 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:30.304361 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f455fb69-f8be-4cca-ab76-2bb6bd551c49-kube-api-access-mmxcb" (OuterVolumeSpecName: "kube-api-access-mmxcb") pod "f455fb69-f8be-4cca-ab76-2bb6bd551c49" (UID: "f455fb69-f8be-4cca-ab76-2bb6bd551c49"). InnerVolumeSpecName "kube-api-access-mmxcb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 16:24:30.403971 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:30.403928 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmxcb\" (UniqueName: \"kubernetes.io/projected/f455fb69-f8be-4cca-ab76-2bb6bd551c49-kube-api-access-mmxcb\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:24:30.403971 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:30.403962 2569 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f455fb69-f8be-4cca-ab76-2bb6bd551c49-config-file\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:24:30.882297 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:30.882264 2569 generic.go:356] "Generic (PLEG): container finished" podID="f455fb69-f8be-4cca-ab76-2bb6bd551c49" containerID="3f2a6d412d391bc1ab8471aa2cc98bc09e3df49bacc95060955f59259006bbbd" exitCode=0 Jul 09 16:24:30.882696 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:30.882305 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-nv6j7" Jul 09 16:24:30.882696 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:30.882346 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-nv6j7" event={"ID":"f455fb69-f8be-4cca-ab76-2bb6bd551c49","Type":"ContainerDied","Data":"3f2a6d412d391bc1ab8471aa2cc98bc09e3df49bacc95060955f59259006bbbd"} Jul 09 16:24:30.882696 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:30.882406 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-nv6j7" event={"ID":"f455fb69-f8be-4cca-ab76-2bb6bd551c49","Type":"ContainerDied","Data":"abeefac74b9c00bf1f40a64ff05ce622fac5352e9893e90607bb2f4440d36b5a"} Jul 09 16:24:30.882696 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:30.882434 2569 scope.go:117] "RemoveContainer" containerID="3f2a6d412d391bc1ab8471aa2cc98bc09e3df49bacc95060955f59259006bbbd" Jul 09 16:24:30.891674 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:30.891658 2569 scope.go:117] "RemoveContainer" containerID="3f2a6d412d391bc1ab8471aa2cc98bc09e3df49bacc95060955f59259006bbbd" Jul 09 16:24:30.892004 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:24:30.891985 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f2a6d412d391bc1ab8471aa2cc98bc09e3df49bacc95060955f59259006bbbd\": container with ID starting with 3f2a6d412d391bc1ab8471aa2cc98bc09e3df49bacc95060955f59259006bbbd not found: ID does not exist" containerID="3f2a6d412d391bc1ab8471aa2cc98bc09e3df49bacc95060955f59259006bbbd" Jul 09 16:24:30.892079 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:30.892017 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f2a6d412d391bc1ab8471aa2cc98bc09e3df49bacc95060955f59259006bbbd"} err="failed to get container status \"3f2a6d412d391bc1ab8471aa2cc98bc09e3df49bacc95060955f59259006bbbd\": rpc error: code = NotFound desc = could not find container \"3f2a6d412d391bc1ab8471aa2cc98bc09e3df49bacc95060955f59259006bbbd\": container with ID starting with 3f2a6d412d391bc1ab8471aa2cc98bc09e3df49bacc95060955f59259006bbbd not found: ID does not exist" Jul 09 16:24:30.911657 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:30.911623 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-nv6j7"] Jul 09 16:24:30.918189 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:30.918162 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-nv6j7"] Jul 09 16:24:31.389470 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:31.389431 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f455fb69-f8be-4cca-ab76-2bb6bd551c49" path="/var/lib/kubelet/pods/f455fb69-f8be-4cca-ab76-2bb6bd551c49/volumes" Jul 09 16:24:34.074625 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:34.074591 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-7jfm6"] Jul 09 16:24:34.075042 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:34.074955 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f455fb69-f8be-4cca-ab76-2bb6bd551c49" containerName="limitador" Jul 09 16:24:34.075042 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:34.074967 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f455fb69-f8be-4cca-ab76-2bb6bd551c49" containerName="limitador" Jul 09 16:24:34.075042 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:34.075021 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="f455fb69-f8be-4cca-ab76-2bb6bd551c49" containerName="limitador" Jul 09 16:24:34.079308 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:34.079289 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-7jfm6" Jul 09 16:24:34.082732 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:34.082710 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 09 16:24:34.082865 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:34.082710 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 09 16:24:34.082865 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:34.082747 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 09 16:24:34.084033 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:34.084013 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-q6npn\"" Jul 09 16:24:34.087776 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:34.087756 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-7jfm6"] Jul 09 16:24:34.242659 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:34.242619 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtwvd\" (UniqueName: \"kubernetes.io/projected/aea78521-1cd6-45a6-b7e4-df4d1d5cdd44-kube-api-access-xtwvd\") pod \"postgres-69fb4594cf-7jfm6\" (UID: \"aea78521-1cd6-45a6-b7e4-df4d1d5cdd44\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-7jfm6" Jul 09 16:24:34.242839 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:34.242673 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/aea78521-1cd6-45a6-b7e4-df4d1d5cdd44-data\") pod \"postgres-69fb4594cf-7jfm6\" (UID: \"aea78521-1cd6-45a6-b7e4-df4d1d5cdd44\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-7jfm6" Jul 09 16:24:34.343929 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:34.343898 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtwvd\" (UniqueName: \"kubernetes.io/projected/aea78521-1cd6-45a6-b7e4-df4d1d5cdd44-kube-api-access-xtwvd\") pod \"postgres-69fb4594cf-7jfm6\" (UID: \"aea78521-1cd6-45a6-b7e4-df4d1d5cdd44\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-7jfm6" Jul 09 16:24:34.344127 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:34.343945 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/aea78521-1cd6-45a6-b7e4-df4d1d5cdd44-data\") pod \"postgres-69fb4594cf-7jfm6\" (UID: \"aea78521-1cd6-45a6-b7e4-df4d1d5cdd44\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-7jfm6" Jul 09 16:24:34.344334 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:34.344312 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/aea78521-1cd6-45a6-b7e4-df4d1d5cdd44-data\") pod \"postgres-69fb4594cf-7jfm6\" (UID: \"aea78521-1cd6-45a6-b7e4-df4d1d5cdd44\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-7jfm6" Jul 09 16:24:34.354712 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:34.354689 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtwvd\" (UniqueName: \"kubernetes.io/projected/aea78521-1cd6-45a6-b7e4-df4d1d5cdd44-kube-api-access-xtwvd\") pod \"postgres-69fb4594cf-7jfm6\" (UID: \"aea78521-1cd6-45a6-b7e4-df4d1d5cdd44\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-7jfm6" Jul 09 16:24:34.390558 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:34.390521 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-7jfm6" Jul 09 16:24:34.527481 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:34.527452 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-7jfm6"] Jul 09 16:24:34.529809 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:24:34.529777 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaea78521_1cd6_45a6_b7e4_df4d1d5cdd44.slice/crio-8cfb7da7c4aa5812f708b94e497956b58c13429547839c3473379d86ee0e93df WatchSource:0}: Error finding container 8cfb7da7c4aa5812f708b94e497956b58c13429547839c3473379d86ee0e93df: Status 404 returned error can't find the container with id 8cfb7da7c4aa5812f708b94e497956b58c13429547839c3473379d86ee0e93df Jul 09 16:24:34.905413 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:34.905376 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-7jfm6" event={"ID":"aea78521-1cd6-45a6-b7e4-df4d1d5cdd44","Type":"ContainerStarted","Data":"8cfb7da7c4aa5812f708b94e497956b58c13429547839c3473379d86ee0e93df"} Jul 09 16:24:41.941711 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:41.941676 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-7jfm6" event={"ID":"aea78521-1cd6-45a6-b7e4-df4d1d5cdd44","Type":"ContainerStarted","Data":"33ae04ac23f166b000e77bd31a6dda0317df63bcc4737e12324131d602d19308"} Jul 09 16:24:41.942158 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:41.941827 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-7jfm6" Jul 09 16:24:41.959970 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:41.959921 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-7jfm6" podStartSLOduration=1.510276086 podStartE2EDuration="7.959906808s" totalImagesPullingTime="6.449630722s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:24:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:24:41.957991275 +0000 UTC m=+567.151097656" watchObservedRunningTime="2026-07-09 16:24:41.959906808 +0000 UTC m=+567.153013188" Jul 09 16:24:47.983104 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:47.983074 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-7jfm6" Jul 09 16:24:48.556686 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:48.556646 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-fbhdg"] Jul 09 16:24:48.560232 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:48.560214 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-fbhdg" Jul 09 16:24:48.569802 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:48.569778 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-fbhdg"] Jul 09 16:24:48.666041 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:48.666001 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqxrt\" (UniqueName: \"kubernetes.io/projected/e9cc7b4d-51af-460b-867f-ba3008eed2ba-kube-api-access-fqxrt\") pod \"authorino-5d4b67c446-fbhdg\" (UID: \"e9cc7b4d-51af-460b-867f-ba3008eed2ba\") " pod="kuadrant-system/authorino-5d4b67c446-fbhdg" Jul 09 16:24:48.766534 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:48.766492 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqxrt\" (UniqueName: \"kubernetes.io/projected/e9cc7b4d-51af-460b-867f-ba3008eed2ba-kube-api-access-fqxrt\") pod \"authorino-5d4b67c446-fbhdg\" (UID: \"e9cc7b4d-51af-460b-867f-ba3008eed2ba\") " pod="kuadrant-system/authorino-5d4b67c446-fbhdg" Jul 09 16:24:48.778126 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:48.778088 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqxrt\" (UniqueName: \"kubernetes.io/projected/e9cc7b4d-51af-460b-867f-ba3008eed2ba-kube-api-access-fqxrt\") pod \"authorino-5d4b67c446-fbhdg\" (UID: \"e9cc7b4d-51af-460b-867f-ba3008eed2ba\") " pod="kuadrant-system/authorino-5d4b67c446-fbhdg" Jul 09 16:24:48.857212 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:48.857176 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-fbhdg"] Jul 09 16:24:48.857448 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:48.857435 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-fbhdg" Jul 09 16:24:48.884262 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:48.884235 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5cd8b6f45b-fpb44"] Jul 09 16:24:48.888546 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:48.888525 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5cd8b6f45b-fpb44" Jul 09 16:24:48.891686 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:48.891661 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 09 16:24:48.898538 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:48.898515 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5cd8b6f45b-fpb44"] Jul 09 16:24:48.999557 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:48.999533 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-fbhdg"] Jul 09 16:24:49.002160 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:24:49.002130 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode9cc7b4d_51af_460b_867f_ba3008eed2ba.slice/crio-1c623c009104fc116392b14ee1624b90c52202fcdce618e425da4bde36120f50 WatchSource:0}: Error finding container 1c623c009104fc116392b14ee1624b90c52202fcdce618e425da4bde36120f50: Status 404 returned error can't find the container with id 1c623c009104fc116392b14ee1624b90c52202fcdce618e425da4bde36120f50 Jul 09 16:24:49.070451 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:49.070416 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjfmb\" (UniqueName: \"kubernetes.io/projected/ab6b37a9-4810-4c69-9b98-e134ab2ae1f4-kube-api-access-sjfmb\") pod \"authorino-5cd8b6f45b-fpb44\" (UID: \"ab6b37a9-4810-4c69-9b98-e134ab2ae1f4\") " pod="kuadrant-system/authorino-5cd8b6f45b-fpb44" Jul 09 16:24:49.070617 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:49.070474 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ab6b37a9-4810-4c69-9b98-e134ab2ae1f4-tls-cert\") pod \"authorino-5cd8b6f45b-fpb44\" (UID: \"ab6b37a9-4810-4c69-9b98-e134ab2ae1f4\") " pod="kuadrant-system/authorino-5cd8b6f45b-fpb44" Jul 09 16:24:49.171963 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:49.171854 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjfmb\" (UniqueName: \"kubernetes.io/projected/ab6b37a9-4810-4c69-9b98-e134ab2ae1f4-kube-api-access-sjfmb\") pod \"authorino-5cd8b6f45b-fpb44\" (UID: \"ab6b37a9-4810-4c69-9b98-e134ab2ae1f4\") " pod="kuadrant-system/authorino-5cd8b6f45b-fpb44" Jul 09 16:24:49.172110 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:49.171969 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ab6b37a9-4810-4c69-9b98-e134ab2ae1f4-tls-cert\") pod \"authorino-5cd8b6f45b-fpb44\" (UID: \"ab6b37a9-4810-4c69-9b98-e134ab2ae1f4\") " pod="kuadrant-system/authorino-5cd8b6f45b-fpb44" Jul 09 16:24:49.174548 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:49.174513 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ab6b37a9-4810-4c69-9b98-e134ab2ae1f4-tls-cert\") pod \"authorino-5cd8b6f45b-fpb44\" (UID: \"ab6b37a9-4810-4c69-9b98-e134ab2ae1f4\") " pod="kuadrant-system/authorino-5cd8b6f45b-fpb44" Jul 09 16:24:49.182869 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:49.182846 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjfmb\" (UniqueName: \"kubernetes.io/projected/ab6b37a9-4810-4c69-9b98-e134ab2ae1f4-kube-api-access-sjfmb\") pod \"authorino-5cd8b6f45b-fpb44\" (UID: \"ab6b37a9-4810-4c69-9b98-e134ab2ae1f4\") " pod="kuadrant-system/authorino-5cd8b6f45b-fpb44" Jul 09 16:24:49.198700 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:49.198666 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5cd8b6f45b-fpb44" Jul 09 16:24:49.328828 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:49.328802 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5cd8b6f45b-fpb44"] Jul 09 16:24:49.331027 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:24:49.330999 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podab6b37a9_4810_4c69_9b98_e134ab2ae1f4.slice/crio-453cf903e22fe6ee9e6168df586f552c3ec745db54dfbb97df9db13aeb5c9ad4 WatchSource:0}: Error finding container 453cf903e22fe6ee9e6168df586f552c3ec745db54dfbb97df9db13aeb5c9ad4: Status 404 returned error can't find the container with id 453cf903e22fe6ee9e6168df586f552c3ec745db54dfbb97df9db13aeb5c9ad4 Jul 09 16:24:49.985021 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:49.984914 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-fbhdg" event={"ID":"e9cc7b4d-51af-460b-867f-ba3008eed2ba","Type":"ContainerStarted","Data":"72d36d93a2f9079fa181f89b5a6688ce70c5d3397ec18b52a7b447f39bfc1377"} Jul 09 16:24:49.985021 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:49.984936 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-fbhdg" podUID="e9cc7b4d-51af-460b-867f-ba3008eed2ba" containerName="authorino" containerID="cri-o://72d36d93a2f9079fa181f89b5a6688ce70c5d3397ec18b52a7b447f39bfc1377" gracePeriod=30 Jul 09 16:24:49.985021 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:49.984969 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-fbhdg" event={"ID":"e9cc7b4d-51af-460b-867f-ba3008eed2ba","Type":"ContainerStarted","Data":"1c623c009104fc116392b14ee1624b90c52202fcdce618e425da4bde36120f50"} Jul 09 16:24:49.986616 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:49.986589 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5cd8b6f45b-fpb44" event={"ID":"ab6b37a9-4810-4c69-9b98-e134ab2ae1f4","Type":"ContainerStarted","Data":"5b83453181e84c4185d01efba0d8062845863a2746882ed3f33cabeb7a09d3a0"} Jul 09 16:24:49.986731 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:49.986631 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5cd8b6f45b-fpb44" event={"ID":"ab6b37a9-4810-4c69-9b98-e134ab2ae1f4","Type":"ContainerStarted","Data":"453cf903e22fe6ee9e6168df586f552c3ec745db54dfbb97df9db13aeb5c9ad4"} Jul 09 16:24:50.002162 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:50.002116 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-fbhdg" podStartSLOduration=1.540714716 podStartE2EDuration="2.002097221s" totalImagesPullingTime="461.382505ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:24:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:24:50.001381766 +0000 UTC m=+575.194488149" watchObservedRunningTime="2026-07-09 16:24:50.002097221 +0000 UTC m=+575.195203610" Jul 09 16:24:50.022576 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:50.022523 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5cd8b6f45b-fpb44" podStartSLOduration=1.654882487 podStartE2EDuration="2.022505495s" totalImagesPullingTime="367.623008ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:24:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:24:50.020992292 +0000 UTC m=+575.214098673" watchObservedRunningTime="2026-07-09 16:24:50.022505495 +0000 UTC m=+575.215611875" Jul 09 16:24:50.053715 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:50.053677 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-n94dc"] Jul 09 16:24:50.054062 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:50.053922 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-n94dc" podUID="c3e01d96-31f6-4659-8533-37b194105b99" containerName="authorino" containerID="cri-o://faa558d3c808adbd624bdff65c62cf0710803f9ccc1d895fd07940bdb7833452" gracePeriod=30 Jul 09 16:24:50.319344 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:50.319316 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-fbhdg" Jul 09 16:24:50.323439 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:50.323418 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-n94dc" Jul 09 16:24:50.382627 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:50.382595 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r6sfg\" (UniqueName: \"kubernetes.io/projected/c3e01d96-31f6-4659-8533-37b194105b99-kube-api-access-r6sfg\") pod \"c3e01d96-31f6-4659-8533-37b194105b99\" (UID: \"c3e01d96-31f6-4659-8533-37b194105b99\") " Jul 09 16:24:50.382814 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:50.382699 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fqxrt\" (UniqueName: \"kubernetes.io/projected/e9cc7b4d-51af-460b-867f-ba3008eed2ba-kube-api-access-fqxrt\") pod \"e9cc7b4d-51af-460b-867f-ba3008eed2ba\" (UID: \"e9cc7b4d-51af-460b-867f-ba3008eed2ba\") " Jul 09 16:24:50.384759 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:50.384728 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3e01d96-31f6-4659-8533-37b194105b99-kube-api-access-r6sfg" (OuterVolumeSpecName: "kube-api-access-r6sfg") pod "c3e01d96-31f6-4659-8533-37b194105b99" (UID: "c3e01d96-31f6-4659-8533-37b194105b99"). InnerVolumeSpecName "kube-api-access-r6sfg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 16:24:50.384888 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:50.384734 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9cc7b4d-51af-460b-867f-ba3008eed2ba-kube-api-access-fqxrt" (OuterVolumeSpecName: "kube-api-access-fqxrt") pod "e9cc7b4d-51af-460b-867f-ba3008eed2ba" (UID: "e9cc7b4d-51af-460b-867f-ba3008eed2ba"). InnerVolumeSpecName "kube-api-access-fqxrt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 16:24:50.483936 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:50.483870 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqxrt\" (UniqueName: \"kubernetes.io/projected/e9cc7b4d-51af-460b-867f-ba3008eed2ba-kube-api-access-fqxrt\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:24:50.483936 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:50.483937 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6sfg\" (UniqueName: \"kubernetes.io/projected/c3e01d96-31f6-4659-8533-37b194105b99-kube-api-access-r6sfg\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:24:50.992562 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:50.992530 2569 generic.go:356] "Generic (PLEG): container finished" podID="e9cc7b4d-51af-460b-867f-ba3008eed2ba" containerID="72d36d93a2f9079fa181f89b5a6688ce70c5d3397ec18b52a7b447f39bfc1377" exitCode=0 Jul 09 16:24:50.992744 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:50.992574 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-fbhdg" Jul 09 16:24:50.992744 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:50.992621 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-fbhdg" event={"ID":"e9cc7b4d-51af-460b-867f-ba3008eed2ba","Type":"ContainerDied","Data":"72d36d93a2f9079fa181f89b5a6688ce70c5d3397ec18b52a7b447f39bfc1377"} Jul 09 16:24:50.992744 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:50.992663 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-fbhdg" event={"ID":"e9cc7b4d-51af-460b-867f-ba3008eed2ba","Type":"ContainerDied","Data":"1c623c009104fc116392b14ee1624b90c52202fcdce618e425da4bde36120f50"} Jul 09 16:24:50.992744 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:50.992682 2569 scope.go:117] "RemoveContainer" containerID="72d36d93a2f9079fa181f89b5a6688ce70c5d3397ec18b52a7b447f39bfc1377" Jul 09 16:24:50.994008 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:50.993940 2569 generic.go:356] "Generic (PLEG): container finished" podID="c3e01d96-31f6-4659-8533-37b194105b99" containerID="faa558d3c808adbd624bdff65c62cf0710803f9ccc1d895fd07940bdb7833452" exitCode=0 Jul 09 16:24:50.994008 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:50.993989 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-n94dc" Jul 09 16:24:50.994127 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:50.994023 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-n94dc" event={"ID":"c3e01d96-31f6-4659-8533-37b194105b99","Type":"ContainerDied","Data":"faa558d3c808adbd624bdff65c62cf0710803f9ccc1d895fd07940bdb7833452"} Jul 09 16:24:50.994127 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:50.994061 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-n94dc" event={"ID":"c3e01d96-31f6-4659-8533-37b194105b99","Type":"ContainerDied","Data":"4c409bfdfabaf17f5bffeee55b485520977b6a1599f4daaa96c9e1dd38ae3c30"} Jul 09 16:24:51.001735 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:51.001619 2569 scope.go:117] "RemoveContainer" containerID="72d36d93a2f9079fa181f89b5a6688ce70c5d3397ec18b52a7b447f39bfc1377" Jul 09 16:24:51.002048 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:24:51.002026 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72d36d93a2f9079fa181f89b5a6688ce70c5d3397ec18b52a7b447f39bfc1377\": container with ID starting with 72d36d93a2f9079fa181f89b5a6688ce70c5d3397ec18b52a7b447f39bfc1377 not found: ID does not exist" containerID="72d36d93a2f9079fa181f89b5a6688ce70c5d3397ec18b52a7b447f39bfc1377" Jul 09 16:24:51.002148 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:51.002057 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72d36d93a2f9079fa181f89b5a6688ce70c5d3397ec18b52a7b447f39bfc1377"} err="failed to get container status \"72d36d93a2f9079fa181f89b5a6688ce70c5d3397ec18b52a7b447f39bfc1377\": rpc error: code = NotFound desc = could not find container \"72d36d93a2f9079fa181f89b5a6688ce70c5d3397ec18b52a7b447f39bfc1377\": container with ID starting with 72d36d93a2f9079fa181f89b5a6688ce70c5d3397ec18b52a7b447f39bfc1377 not found: ID does not exist" Jul 09 16:24:51.002148 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:51.002078 2569 scope.go:117] "RemoveContainer" containerID="faa558d3c808adbd624bdff65c62cf0710803f9ccc1d895fd07940bdb7833452" Jul 09 16:24:51.009994 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:51.009972 2569 scope.go:117] "RemoveContainer" containerID="faa558d3c808adbd624bdff65c62cf0710803f9ccc1d895fd07940bdb7833452" Jul 09 16:24:51.010252 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:24:51.010232 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"faa558d3c808adbd624bdff65c62cf0710803f9ccc1d895fd07940bdb7833452\": container with ID starting with faa558d3c808adbd624bdff65c62cf0710803f9ccc1d895fd07940bdb7833452 not found: ID does not exist" containerID="faa558d3c808adbd624bdff65c62cf0710803f9ccc1d895fd07940bdb7833452" Jul 09 16:24:51.010344 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:51.010259 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"faa558d3c808adbd624bdff65c62cf0710803f9ccc1d895fd07940bdb7833452"} err="failed to get container status \"faa558d3c808adbd624bdff65c62cf0710803f9ccc1d895fd07940bdb7833452\": rpc error: code = NotFound desc = could not find container \"faa558d3c808adbd624bdff65c62cf0710803f9ccc1d895fd07940bdb7833452\": container with ID starting with faa558d3c808adbd624bdff65c62cf0710803f9ccc1d895fd07940bdb7833452 not found: ID does not exist" Jul 09 16:24:51.020456 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:51.020432 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-fbhdg"] Jul 09 16:24:51.028458 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:51.028435 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-fbhdg"] Jul 09 16:24:51.042372 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:51.042295 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-n94dc"] Jul 09 16:24:51.049359 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:51.049331 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-n94dc"] Jul 09 16:24:51.389291 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:51.389248 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3e01d96-31f6-4659-8533-37b194105b99" path="/var/lib/kubelet/pods/c3e01d96-31f6-4659-8533-37b194105b99/volumes" Jul 09 16:24:51.389692 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:51.389671 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9cc7b4d-51af-460b-867f-ba3008eed2ba" path="/var/lib/kubelet/pods/e9cc7b4d-51af-460b-867f-ba3008eed2ba/volumes" Jul 09 16:24:54.670346 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:54.670302 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-dc4ddbd74-kh7zj"] Jul 09 16:24:54.670847 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:54.670824 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9cc7b4d-51af-460b-867f-ba3008eed2ba" containerName="authorino" Jul 09 16:24:54.670948 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:54.670850 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9cc7b4d-51af-460b-867f-ba3008eed2ba" containerName="authorino" Jul 09 16:24:54.670948 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:54.670910 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3e01d96-31f6-4659-8533-37b194105b99" containerName="authorino" Jul 09 16:24:54.670948 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:54.670919 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3e01d96-31f6-4659-8533-37b194105b99" containerName="authorino" Jul 09 16:24:54.671096 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:54.671019 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9cc7b4d-51af-460b-867f-ba3008eed2ba" containerName="authorino" Jul 09 16:24:54.671096 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:54.671035 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3e01d96-31f6-4659-8533-37b194105b99" containerName="authorino" Jul 09 16:24:54.675286 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:54.675261 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-dc4ddbd74-kh7zj" Jul 09 16:24:54.680265 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:54.680237 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 09 16:24:54.680265 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:54.680254 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 09 16:24:54.680462 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:54.680283 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-l94hm\"" Jul 09 16:24:54.690007 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:54.689972 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-dc4ddbd74-kh7zj"] Jul 09 16:24:54.713900 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:54.713848 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jjmt\" (UniqueName: \"kubernetes.io/projected/953a1f08-d3f9-4814-bfb5-2d9ccd10d2de-kube-api-access-2jjmt\") pod \"maas-controller-dc4ddbd74-kh7zj\" (UID: \"953a1f08-d3f9-4814-bfb5-2d9ccd10d2de\") " pod="opendatahub/maas-controller-dc4ddbd74-kh7zj" Jul 09 16:24:54.714074 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:54.713992 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/953a1f08-d3f9-4814-bfb5-2d9ccd10d2de-webhook-cert\") pod \"maas-controller-dc4ddbd74-kh7zj\" (UID: \"953a1f08-d3f9-4814-bfb5-2d9ccd10d2de\") " pod="opendatahub/maas-controller-dc4ddbd74-kh7zj" Jul 09 16:24:54.814836 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:54.814789 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/953a1f08-d3f9-4814-bfb5-2d9ccd10d2de-webhook-cert\") pod \"maas-controller-dc4ddbd74-kh7zj\" (UID: \"953a1f08-d3f9-4814-bfb5-2d9ccd10d2de\") " pod="opendatahub/maas-controller-dc4ddbd74-kh7zj" Jul 09 16:24:54.815050 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:54.814905 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jjmt\" (UniqueName: \"kubernetes.io/projected/953a1f08-d3f9-4814-bfb5-2d9ccd10d2de-kube-api-access-2jjmt\") pod \"maas-controller-dc4ddbd74-kh7zj\" (UID: \"953a1f08-d3f9-4814-bfb5-2d9ccd10d2de\") " pod="opendatahub/maas-controller-dc4ddbd74-kh7zj" Jul 09 16:24:54.815050 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:24:54.814993 2569 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 09 16:24:54.815137 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:24:54.815079 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/953a1f08-d3f9-4814-bfb5-2d9ccd10d2de-webhook-cert podName:953a1f08-d3f9-4814-bfb5-2d9ccd10d2de nodeName:}" failed. No retries permitted until 2026-07-09 16:24:55.315057192 +0000 UTC m=+580.508163557 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/953a1f08-d3f9-4814-bfb5-2d9ccd10d2de-webhook-cert") pod "maas-controller-dc4ddbd74-kh7zj" (UID: "953a1f08-d3f9-4814-bfb5-2d9ccd10d2de") : secret "maas-controller-webhook-cert" not found Jul 09 16:24:54.824563 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:54.824524 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jjmt\" (UniqueName: \"kubernetes.io/projected/953a1f08-d3f9-4814-bfb5-2d9ccd10d2de-kube-api-access-2jjmt\") pod \"maas-controller-dc4ddbd74-kh7zj\" (UID: \"953a1f08-d3f9-4814-bfb5-2d9ccd10d2de\") " pod="opendatahub/maas-controller-dc4ddbd74-kh7zj" Jul 09 16:24:54.883703 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:54.883666 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-64fcf5b9b9-g6jmn"] Jul 09 16:24:54.887153 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:54.887130 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-64fcf5b9b9-g6jmn" Jul 09 16:24:54.900408 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:54.900382 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-64fcf5b9b9-g6jmn"] Jul 09 16:24:54.915436 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:54.915398 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ec0bea33-b563-4057-a750-8dc451111bd5-webhook-cert\") pod \"maas-controller-64fcf5b9b9-g6jmn\" (UID: \"ec0bea33-b563-4057-a750-8dc451111bd5\") " pod="opendatahub/maas-controller-64fcf5b9b9-g6jmn" Jul 09 16:24:54.916109 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:54.915468 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq9j2\" (UniqueName: \"kubernetes.io/projected/ec0bea33-b563-4057-a750-8dc451111bd5-kube-api-access-jq9j2\") pod \"maas-controller-64fcf5b9b9-g6jmn\" (UID: \"ec0bea33-b563-4057-a750-8dc451111bd5\") " pod="opendatahub/maas-controller-64fcf5b9b9-g6jmn" Jul 09 16:24:55.016030 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:55.015936 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ec0bea33-b563-4057-a750-8dc451111bd5-webhook-cert\") pod \"maas-controller-64fcf5b9b9-g6jmn\" (UID: \"ec0bea33-b563-4057-a750-8dc451111bd5\") " pod="opendatahub/maas-controller-64fcf5b9b9-g6jmn" Jul 09 16:24:55.016030 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:55.015997 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq9j2\" (UniqueName: \"kubernetes.io/projected/ec0bea33-b563-4057-a750-8dc451111bd5-kube-api-access-jq9j2\") pod \"maas-controller-64fcf5b9b9-g6jmn\" (UID: \"ec0bea33-b563-4057-a750-8dc451111bd5\") " pod="opendatahub/maas-controller-64fcf5b9b9-g6jmn" Jul 09 16:24:55.018546 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:55.018513 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ec0bea33-b563-4057-a750-8dc451111bd5-webhook-cert\") pod \"maas-controller-64fcf5b9b9-g6jmn\" (UID: \"ec0bea33-b563-4057-a750-8dc451111bd5\") " pod="opendatahub/maas-controller-64fcf5b9b9-g6jmn" Jul 09 16:24:55.025968 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:55.025941 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq9j2\" (UniqueName: \"kubernetes.io/projected/ec0bea33-b563-4057-a750-8dc451111bd5-kube-api-access-jq9j2\") pod \"maas-controller-64fcf5b9b9-g6jmn\" (UID: \"ec0bea33-b563-4057-a750-8dc451111bd5\") " pod="opendatahub/maas-controller-64fcf5b9b9-g6jmn" Jul 09 16:24:55.198038 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:55.197997 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-64fcf5b9b9-g6jmn" Jul 09 16:24:55.318227 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:55.318195 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/953a1f08-d3f9-4814-bfb5-2d9ccd10d2de-webhook-cert\") pod \"maas-controller-dc4ddbd74-kh7zj\" (UID: \"953a1f08-d3f9-4814-bfb5-2d9ccd10d2de\") " pod="opendatahub/maas-controller-dc4ddbd74-kh7zj" Jul 09 16:24:55.320613 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:55.320590 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/953a1f08-d3f9-4814-bfb5-2d9ccd10d2de-webhook-cert\") pod \"maas-controller-dc4ddbd74-kh7zj\" (UID: \"953a1f08-d3f9-4814-bfb5-2d9ccd10d2de\") " pod="opendatahub/maas-controller-dc4ddbd74-kh7zj" Jul 09 16:24:55.335236 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:55.335208 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-64fcf5b9b9-g6jmn"] Jul 09 16:24:55.338289 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:24:55.338244 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec0bea33_b563_4057_a750_8dc451111bd5.slice/crio-043aaa341dd8567e0fff76a0605b00a14e43518252a6ef8efeba7326858708e4 WatchSource:0}: Error finding container 043aaa341dd8567e0fff76a0605b00a14e43518252a6ef8efeba7326858708e4: Status 404 returned error can't find the container with id 043aaa341dd8567e0fff76a0605b00a14e43518252a6ef8efeba7326858708e4 Jul 09 16:24:55.586299 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:55.586202 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-dc4ddbd74-kh7zj" Jul 09 16:24:55.722351 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:55.722321 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-dc4ddbd74-kh7zj"] Jul 09 16:24:55.725125 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:24:55.725087 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod953a1f08_d3f9_4814_bfb5_2d9ccd10d2de.slice/crio-e07c4de1242019b9bb58f104e20b240302f155468a0ff7cd6520e8d0370ab899 WatchSource:0}: Error finding container e07c4de1242019b9bb58f104e20b240302f155468a0ff7cd6520e8d0370ab899: Status 404 returned error can't find the container with id e07c4de1242019b9bb58f104e20b240302f155468a0ff7cd6520e8d0370ab899 Jul 09 16:24:56.018824 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:56.018790 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-64fcf5b9b9-g6jmn" event={"ID":"ec0bea33-b563-4057-a750-8dc451111bd5","Type":"ContainerStarted","Data":"043aaa341dd8567e0fff76a0605b00a14e43518252a6ef8efeba7326858708e4"} Jul 09 16:24:56.020003 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:56.019973 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-dc4ddbd74-kh7zj" event={"ID":"953a1f08-d3f9-4814-bfb5-2d9ccd10d2de","Type":"ContainerStarted","Data":"e07c4de1242019b9bb58f104e20b240302f155468a0ff7cd6520e8d0370ab899"} Jul 09 16:24:59.041405 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:59.041363 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-dc4ddbd74-kh7zj" event={"ID":"953a1f08-d3f9-4814-bfb5-2d9ccd10d2de","Type":"ContainerStarted","Data":"080bc41d6546d843dd9459762552653723910f732efb4a1140db678fb43761fd"} Jul 09 16:24:59.041896 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:59.041553 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-dc4ddbd74-kh7zj" Jul 09 16:24:59.043047 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:59.043023 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-64fcf5b9b9-g6jmn" event={"ID":"ec0bea33-b563-4057-a750-8dc451111bd5","Type":"ContainerStarted","Data":"88dde7b4fd6c87312fd7135367925279bf88ddfdf4f8d5060aeaecb63a112a04"} Jul 09 16:24:59.043154 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:59.043080 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-64fcf5b9b9-g6jmn" Jul 09 16:24:59.065232 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:59.065182 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-dc4ddbd74-kh7zj" podStartSLOduration=2.386633324 podStartE2EDuration="5.065164389s" totalImagesPullingTime="2.678531065s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:24:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:24:59.064636465 +0000 UTC m=+584.257742840" watchObservedRunningTime="2026-07-09 16:24:59.065164389 +0000 UTC m=+584.258270770" Jul 09 16:24:59.084456 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:24:59.084402 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-64fcf5b9b9-g6jmn" podStartSLOduration=2.026020419 podStartE2EDuration="5.084385802s" totalImagesPullingTime="3.058365383s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:24:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:24:59.083242495 +0000 UTC m=+584.276348877" watchObservedRunningTime="2026-07-09 16:24:59.084385802 +0000 UTC m=+584.277492183" Jul 09 16:25:10.055761 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:25:10.055728 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-64fcf5b9b9-g6jmn" Jul 09 16:25:10.056210 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:25:10.055920 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-dc4ddbd74-kh7zj" Jul 09 16:25:10.118074 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:25:10.118029 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-dc4ddbd74-kh7zj"] Jul 09 16:25:10.118296 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:25:10.118265 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-dc4ddbd74-kh7zj" podUID="953a1f08-d3f9-4814-bfb5-2d9ccd10d2de" containerName="manager" containerID="cri-o://080bc41d6546d843dd9459762552653723910f732efb4a1140db678fb43761fd" gracePeriod=30 Jul 09 16:25:10.361600 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:25:10.361568 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-dc4ddbd74-kh7zj" Jul 09 16:25:10.447846 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:25:10.447801 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2jjmt\" (UniqueName: \"kubernetes.io/projected/953a1f08-d3f9-4814-bfb5-2d9ccd10d2de-kube-api-access-2jjmt\") pod \"953a1f08-d3f9-4814-bfb5-2d9ccd10d2de\" (UID: \"953a1f08-d3f9-4814-bfb5-2d9ccd10d2de\") " Jul 09 16:25:10.447846 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:25:10.447853 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/953a1f08-d3f9-4814-bfb5-2d9ccd10d2de-webhook-cert\") pod \"953a1f08-d3f9-4814-bfb5-2d9ccd10d2de\" (UID: \"953a1f08-d3f9-4814-bfb5-2d9ccd10d2de\") " Jul 09 16:25:10.450153 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:25:10.450112 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/953a1f08-d3f9-4814-bfb5-2d9ccd10d2de-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "953a1f08-d3f9-4814-bfb5-2d9ccd10d2de" (UID: "953a1f08-d3f9-4814-bfb5-2d9ccd10d2de"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 16:25:10.450153 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:25:10.450124 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/953a1f08-d3f9-4814-bfb5-2d9ccd10d2de-kube-api-access-2jjmt" (OuterVolumeSpecName: "kube-api-access-2jjmt") pod "953a1f08-d3f9-4814-bfb5-2d9ccd10d2de" (UID: "953a1f08-d3f9-4814-bfb5-2d9ccd10d2de"). InnerVolumeSpecName "kube-api-access-2jjmt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 16:25:10.549106 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:25:10.549070 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jjmt\" (UniqueName: \"kubernetes.io/projected/953a1f08-d3f9-4814-bfb5-2d9ccd10d2de-kube-api-access-2jjmt\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:25:10.549106 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:25:10.549099 2569 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/953a1f08-d3f9-4814-bfb5-2d9ccd10d2de-webhook-cert\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:25:11.104488 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:25:11.104444 2569 generic.go:356] "Generic (PLEG): container finished" podID="953a1f08-d3f9-4814-bfb5-2d9ccd10d2de" containerID="080bc41d6546d843dd9459762552653723910f732efb4a1140db678fb43761fd" exitCode=0 Jul 09 16:25:11.104924 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:25:11.104500 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-dc4ddbd74-kh7zj" event={"ID":"953a1f08-d3f9-4814-bfb5-2d9ccd10d2de","Type":"ContainerDied","Data":"080bc41d6546d843dd9459762552653723910f732efb4a1140db678fb43761fd"} Jul 09 16:25:11.104924 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:25:11.104515 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-dc4ddbd74-kh7zj" Jul 09 16:25:11.104924 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:25:11.104582 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-dc4ddbd74-kh7zj" event={"ID":"953a1f08-d3f9-4814-bfb5-2d9ccd10d2de","Type":"ContainerDied","Data":"e07c4de1242019b9bb58f104e20b240302f155468a0ff7cd6520e8d0370ab899"} Jul 09 16:25:11.104924 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:25:11.104614 2569 scope.go:117] "RemoveContainer" containerID="080bc41d6546d843dd9459762552653723910f732efb4a1140db678fb43761fd" Jul 09 16:25:11.114029 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:25:11.114011 2569 scope.go:117] "RemoveContainer" containerID="080bc41d6546d843dd9459762552653723910f732efb4a1140db678fb43761fd" Jul 09 16:25:11.114607 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:25:11.114492 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"080bc41d6546d843dd9459762552653723910f732efb4a1140db678fb43761fd\": container with ID starting with 080bc41d6546d843dd9459762552653723910f732efb4a1140db678fb43761fd not found: ID does not exist" containerID="080bc41d6546d843dd9459762552653723910f732efb4a1140db678fb43761fd" Jul 09 16:25:11.114607 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:25:11.114549 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"080bc41d6546d843dd9459762552653723910f732efb4a1140db678fb43761fd"} err="failed to get container status \"080bc41d6546d843dd9459762552653723910f732efb4a1140db678fb43761fd\": rpc error: code = NotFound desc = could not find container \"080bc41d6546d843dd9459762552653723910f732efb4a1140db678fb43761fd\": container with ID starting with 080bc41d6546d843dd9459762552653723910f732efb4a1140db678fb43761fd not found: ID does not exist" Jul 09 16:25:11.139726 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:25:11.139691 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-dc4ddbd74-kh7zj"] Jul 09 16:25:11.145447 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:25:11.145420 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-dc4ddbd74-kh7zj"] Jul 09 16:25:11.388840 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:25:11.388758 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="953a1f08-d3f9-4814-bfb5-2d9ccd10d2de" path="/var/lib/kubelet/pods/953a1f08-d3f9-4814-bfb5-2d9ccd10d2de/volumes" Jul 09 16:25:15.266708 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:25:15.266681 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-nzrk9_a284ea64-610c-40f7-837b-fa0975c0a290/console-operator/2.log" Jul 09 16:25:15.267132 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:25:15.266836 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-nzrk9_a284ea64-610c-40f7-837b-fa0975c0a290/console-operator/2.log" Jul 09 16:25:15.274055 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:25:15.274030 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-49sqd_17313664-7a0f-472b-a597-39f8c5412bc3/ovn-acl-logging/0.log" Jul 09 16:25:15.274211 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:25:15.274030 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-49sqd_17313664-7a0f-472b-a597-39f8c5412bc3/ovn-acl-logging/0.log" Jul 09 16:25:29.285736 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:25:29.285696 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-54556c5fb4-6kntc"] Jul 09 16:25:29.286275 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:25:29.286262 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="953a1f08-d3f9-4814-bfb5-2d9ccd10d2de" containerName="manager" Jul 09 16:25:29.286330 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:25:29.286279 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="953a1f08-d3f9-4814-bfb5-2d9ccd10d2de" containerName="manager" Jul 09 16:25:29.286397 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:25:29.286385 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="953a1f08-d3f9-4814-bfb5-2d9ccd10d2de" containerName="manager" Jul 09 16:25:29.307083 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:25:29.307026 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-54556c5fb4-6kntc"] Jul 09 16:25:29.307271 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:25:29.307174 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-54556c5fb4-6kntc" Jul 09 16:25:29.310553 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:25:29.310531 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-f92zq\"" Jul 09 16:25:29.310685 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:25:29.310531 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 09 16:25:29.410839 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:25:29.410796 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68ff35f9-7681-411d-ad27-8f49d787c5a0-openshift-service-ca\") pod \"maas-api-54556c5fb4-6kntc\" (UID: \"68ff35f9-7681-411d-ad27-8f49d787c5a0\") " pod="odh-ai-gateway-infra/maas-api-54556c5fb4-6kntc" Jul 09 16:25:29.411036 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:25:29.410907 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68ff35f9-7681-411d-ad27-8f49d787c5a0-maas-api-tls\") pod \"maas-api-54556c5fb4-6kntc\" (UID: \"68ff35f9-7681-411d-ad27-8f49d787c5a0\") " pod="odh-ai-gateway-infra/maas-api-54556c5fb4-6kntc" Jul 09 16:25:29.411036 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:25:29.410949 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6msq\" (UniqueName: \"kubernetes.io/projected/68ff35f9-7681-411d-ad27-8f49d787c5a0-kube-api-access-f6msq\") pod \"maas-api-54556c5fb4-6kntc\" (UID: \"68ff35f9-7681-411d-ad27-8f49d787c5a0\") " pod="odh-ai-gateway-infra/maas-api-54556c5fb4-6kntc" Jul 09 16:25:29.512084 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:25:29.512049 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68ff35f9-7681-411d-ad27-8f49d787c5a0-openshift-service-ca\") pod \"maas-api-54556c5fb4-6kntc\" (UID: \"68ff35f9-7681-411d-ad27-8f49d787c5a0\") " pod="odh-ai-gateway-infra/maas-api-54556c5fb4-6kntc" Jul 09 16:25:29.512292 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:25:29.512144 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68ff35f9-7681-411d-ad27-8f49d787c5a0-maas-api-tls\") pod \"maas-api-54556c5fb4-6kntc\" (UID: \"68ff35f9-7681-411d-ad27-8f49d787c5a0\") " pod="odh-ai-gateway-infra/maas-api-54556c5fb4-6kntc" Jul 09 16:25:29.512292 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:25:29.512177 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6msq\" (UniqueName: \"kubernetes.io/projected/68ff35f9-7681-411d-ad27-8f49d787c5a0-kube-api-access-f6msq\") pod \"maas-api-54556c5fb4-6kntc\" (UID: \"68ff35f9-7681-411d-ad27-8f49d787c5a0\") " pod="odh-ai-gateway-infra/maas-api-54556c5fb4-6kntc" Jul 09 16:25:29.512809 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:25:29.512785 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68ff35f9-7681-411d-ad27-8f49d787c5a0-openshift-service-ca\") pod \"maas-api-54556c5fb4-6kntc\" (UID: \"68ff35f9-7681-411d-ad27-8f49d787c5a0\") " pod="odh-ai-gateway-infra/maas-api-54556c5fb4-6kntc" Jul 09 16:25:29.514706 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:25:29.514681 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68ff35f9-7681-411d-ad27-8f49d787c5a0-maas-api-tls\") pod \"maas-api-54556c5fb4-6kntc\" (UID: \"68ff35f9-7681-411d-ad27-8f49d787c5a0\") " pod="odh-ai-gateway-infra/maas-api-54556c5fb4-6kntc" Jul 09 16:25:29.521339 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:25:29.521312 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6msq\" (UniqueName: \"kubernetes.io/projected/68ff35f9-7681-411d-ad27-8f49d787c5a0-kube-api-access-f6msq\") pod \"maas-api-54556c5fb4-6kntc\" (UID: \"68ff35f9-7681-411d-ad27-8f49d787c5a0\") " pod="odh-ai-gateway-infra/maas-api-54556c5fb4-6kntc" Jul 09 16:25:29.618286 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:25:29.618252 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-54556c5fb4-6kntc" Jul 09 16:25:29.790431 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:25:29.790399 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-54556c5fb4-6kntc"] Jul 09 16:25:29.793569 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:25:29.793535 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68ff35f9_7681_411d_ad27_8f49d787c5a0.slice/crio-8e551434c8768a503eac26f48d9be3bc600fa37f2011833c42340dd7f0108f6d WatchSource:0}: Error finding container 8e551434c8768a503eac26f48d9be3bc600fa37f2011833c42340dd7f0108f6d: Status 404 returned error can't find the container with id 8e551434c8768a503eac26f48d9be3bc600fa37f2011833c42340dd7f0108f6d Jul 09 16:25:29.951501 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:25:29.951413 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-9tvlf"] Jul 09 16:25:29.956477 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:25:29.956449 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-9tvlf" Jul 09 16:25:29.959300 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:25:29.959277 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 09 16:25:29.959501 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:25:29.959482 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-9gtgz\"" Jul 09 16:25:29.965431 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:25:29.965411 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-9tvlf"] Jul 09 16:25:29.985555 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:25:29.985523 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-t2tml"] Jul 09 16:25:29.989092 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:25:29.989075 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-t2tml" Jul 09 16:25:30.000582 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:25:30.000553 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-t2tml"] Jul 09 16:25:30.016792 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:25:30.016758 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jp6w6\" (UniqueName: \"kubernetes.io/projected/58f7fc44-58a9-45a4-95d3-12d4983bb487-kube-api-access-jp6w6\") pod \"payload-processing-58f86c7798-9tvlf\" (UID: \"58f7fc44-58a9-45a4-95d3-12d4983bb487\") " pod="openshift-ingress/payload-processing-58f86c7798-9tvlf" Jul 09 16:25:30.016792 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:25:30.016790 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58f7fc44-58a9-45a4-95d3-12d4983bb487-plugins-config-volume\") pod \"payload-processing-58f86c7798-9tvlf\" (UID: \"58f7fc44-58a9-45a4-95d3-12d4983bb487\") " pod="openshift-ingress/payload-processing-58f86c7798-9tvlf" Jul 09 16:25:30.117985 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:25:30.117946 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jp6w6\" (UniqueName: \"kubernetes.io/projected/58f7fc44-58a9-45a4-95d3-12d4983bb487-kube-api-access-jp6w6\") pod \"payload-processing-58f86c7798-9tvlf\" (UID: \"58f7fc44-58a9-45a4-95d3-12d4983bb487\") " pod="openshift-ingress/payload-processing-58f86c7798-9tvlf" Jul 09 16:25:30.118176 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:25:30.117997 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9972f323-cbe6-4d1b-bf97-f7f1261f3d74-plugins-config-volume\") pod \"payload-pre-processing-7964457696-t2tml\" (UID: \"9972f323-cbe6-4d1b-bf97-f7f1261f3d74\") " pod="openshift-ingress/payload-pre-processing-7964457696-t2tml" Jul 09 16:25:30.118176 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:25:30.118028 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cszz9\" (UniqueName: \"kubernetes.io/projected/9972f323-cbe6-4d1b-bf97-f7f1261f3d74-kube-api-access-cszz9\") pod \"payload-pre-processing-7964457696-t2tml\" (UID: \"9972f323-cbe6-4d1b-bf97-f7f1261f3d74\") " pod="openshift-ingress/payload-pre-processing-7964457696-t2tml" Jul 09 16:25:30.118176 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:25:30.118091 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58f7fc44-58a9-45a4-95d3-12d4983bb487-plugins-config-volume\") pod \"payload-processing-58f86c7798-9tvlf\" (UID: \"58f7fc44-58a9-45a4-95d3-12d4983bb487\") " pod="openshift-ingress/payload-processing-58f86c7798-9tvlf" Jul 09 16:25:30.118725 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:25:30.118706 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58f7fc44-58a9-45a4-95d3-12d4983bb487-plugins-config-volume\") pod \"payload-processing-58f86c7798-9tvlf\" (UID: \"58f7fc44-58a9-45a4-95d3-12d4983bb487\") " pod="openshift-ingress/payload-processing-58f86c7798-9tvlf" Jul 09 16:25:30.131389 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:25:30.131362 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp6w6\" (UniqueName: \"kubernetes.io/projected/58f7fc44-58a9-45a4-95d3-12d4983bb487-kube-api-access-jp6w6\") pod \"payload-processing-58f86c7798-9tvlf\" (UID: \"58f7fc44-58a9-45a4-95d3-12d4983bb487\") " pod="openshift-ingress/payload-processing-58f86c7798-9tvlf" Jul 09 16:25:30.195158 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:25:30.195116 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-54556c5fb4-6kntc" event={"ID":"68ff35f9-7681-411d-ad27-8f49d787c5a0","Type":"ContainerStarted","Data":"8e551434c8768a503eac26f48d9be3bc600fa37f2011833c42340dd7f0108f6d"} Jul 09 16:25:30.219513 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:25:30.219428 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9972f323-cbe6-4d1b-bf97-f7f1261f3d74-plugins-config-volume\") pod \"payload-pre-processing-7964457696-t2tml\" (UID: \"9972f323-cbe6-4d1b-bf97-f7f1261f3d74\") " pod="openshift-ingress/payload-pre-processing-7964457696-t2tml" Jul 09 16:25:30.219513 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:25:30.219464 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cszz9\" (UniqueName: \"kubernetes.io/projected/9972f323-cbe6-4d1b-bf97-f7f1261f3d74-kube-api-access-cszz9\") pod \"payload-pre-processing-7964457696-t2tml\" (UID: \"9972f323-cbe6-4d1b-bf97-f7f1261f3d74\") " pod="openshift-ingress/payload-pre-processing-7964457696-t2tml" Jul 09 16:25:30.220158 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:25:30.220129 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9972f323-cbe6-4d1b-bf97-f7f1261f3d74-plugins-config-volume\") pod \"payload-pre-processing-7964457696-t2tml\" (UID: \"9972f323-cbe6-4d1b-bf97-f7f1261f3d74\") " pod="openshift-ingress/payload-pre-processing-7964457696-t2tml" Jul 09 16:25:30.228901 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:25:30.228862 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cszz9\" (UniqueName: \"kubernetes.io/projected/9972f323-cbe6-4d1b-bf97-f7f1261f3d74-kube-api-access-cszz9\") pod \"payload-pre-processing-7964457696-t2tml\" (UID: \"9972f323-cbe6-4d1b-bf97-f7f1261f3d74\") " pod="openshift-ingress/payload-pre-processing-7964457696-t2tml" Jul 09 16:25:30.269397 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:25:30.269364 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-9tvlf" Jul 09 16:25:30.299789 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:25:30.299755 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-t2tml" Jul 09 16:25:30.432121 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:25:30.432092 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-9tvlf"] Jul 09 16:25:30.433744 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:25:30.433719 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58f7fc44_58a9_45a4_95d3_12d4983bb487.slice/crio-2c3534eeb4cf7719671410d0689233e28d992ca993a3b035421a8e38f8595ca9 WatchSource:0}: Error finding container 2c3534eeb4cf7719671410d0689233e28d992ca993a3b035421a8e38f8595ca9: Status 404 returned error can't find the container with id 2c3534eeb4cf7719671410d0689233e28d992ca993a3b035421a8e38f8595ca9 Jul 09 16:25:30.462445 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:25:30.462421 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-t2tml"] Jul 09 16:25:30.464815 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:25:30.464784 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9972f323_cbe6_4d1b_bf97_f7f1261f3d74.slice/crio-6eaf3a699070b277765f4d097f6d3ea7f85e30f74103e481811c7df0450515e0 WatchSource:0}: Error finding container 6eaf3a699070b277765f4d097f6d3ea7f85e30f74103e481811c7df0450515e0: Status 404 returned error can't find the container with id 6eaf3a699070b277765f4d097f6d3ea7f85e30f74103e481811c7df0450515e0 Jul 09 16:25:31.200933 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:25:31.200895 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-t2tml" event={"ID":"9972f323-cbe6-4d1b-bf97-f7f1261f3d74","Type":"ContainerStarted","Data":"6eaf3a699070b277765f4d097f6d3ea7f85e30f74103e481811c7df0450515e0"} Jul 09 16:25:31.202160 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:25:31.202133 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-9tvlf" event={"ID":"58f7fc44-58a9-45a4-95d3-12d4983bb487","Type":"ContainerStarted","Data":"2c3534eeb4cf7719671410d0689233e28d992ca993a3b035421a8e38f8595ca9"} Jul 09 16:25:33.214288 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:25:33.214247 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-54556c5fb4-6kntc" event={"ID":"68ff35f9-7681-411d-ad27-8f49d787c5a0","Type":"ContainerStarted","Data":"00dd9166780ff9b9bf8e17a419b3dab257e7ce3377538220d86670a8b69bc078"} Jul 09 16:25:33.214737 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:25:33.214326 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-54556c5fb4-6kntc" Jul 09 16:25:33.215762 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:25:33.215727 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-t2tml" event={"ID":"9972f323-cbe6-4d1b-bf97-f7f1261f3d74","Type":"ContainerStarted","Data":"77e5151ccbcc4e7a32300e72b2411aac80eed051a2040e17c70197bdf9dda76e"} Jul 09 16:25:33.215913 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:25:33.215823 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7964457696-t2tml" Jul 09 16:25:33.217227 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:25:33.217196 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-9tvlf" event={"ID":"58f7fc44-58a9-45a4-95d3-12d4983bb487","Type":"ContainerStarted","Data":"ac3e596868d3f0a381c9c38ee6a759ba28c74459c861f9a733f144bea146509e"} Jul 09 16:25:33.217336 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:25:33.217251 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-58f86c7798-9tvlf" Jul 09 16:25:33.238302 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:25:33.238251 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-54556c5fb4-6kntc" podStartSLOduration=1.586225089 podStartE2EDuration="4.238229727s" totalImagesPullingTime="2.652004638s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:25:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:25:33.23644657 +0000 UTC m=+618.429552953" watchObservedRunningTime="2026-07-09 16:25:33.238229727 +0000 UTC m=+618.431336108" Jul 09 16:25:33.253988 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:25:33.253939 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7964457696-t2tml" podStartSLOduration=2.318662264 podStartE2EDuration="4.2539233s" totalImagesPullingTime="1.935261036s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:25:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:25:33.252693687 +0000 UTC m=+618.445800071" watchObservedRunningTime="2026-07-09 16:25:33.2539233 +0000 UTC m=+618.447029681" Jul 09 16:25:33.269420 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:25:33.269361 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-58f86c7798-9tvlf" podStartSLOduration=2.300123582 podStartE2EDuration="4.269341722s" totalImagesPullingTime="1.96921814s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:25:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:25:33.267993011 +0000 UTC m=+618.461099393" watchObservedRunningTime="2026-07-09 16:25:33.269341722 +0000 UTC m=+618.462448102" Jul 09 16:25:39.228539 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:25:39.228457 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-54556c5fb4-6kntc" Jul 09 16:25:44.224862 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:25:44.224831 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-58f86c7798-9tvlf" Jul 09 16:25:44.225311 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:25:44.225153 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7964457696-t2tml" Jul 09 16:26:02.482379 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:02.482339 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-t8wx2"] Jul 09 16:26:02.487516 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:02.487493 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t8wx2" Jul 09 16:26:02.491934 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:02.491743 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 09 16:26:02.491934 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:02.491747 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-8g5kt\"" Jul 09 16:26:02.491934 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:02.491777 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 09 16:26:02.492182 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:02.492017 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 09 16:26:02.497374 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:02.497350 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-t8wx2"] Jul 09 16:26:02.500602 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:02.500569 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43458597-f606-4740-b30d-c3b6a3f82245-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-t8wx2\" (UID: \"43458597-f606-4740-b30d-c3b6a3f82245\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t8wx2" Jul 09 16:26:02.500778 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:02.500761 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43458597-f606-4740-b30d-c3b6a3f82245-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-t8wx2\" (UID: \"43458597-f606-4740-b30d-c3b6a3f82245\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t8wx2" Jul 09 16:26:02.500938 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:02.500920 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43458597-f606-4740-b30d-c3b6a3f82245-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-t8wx2\" (UID: \"43458597-f606-4740-b30d-c3b6a3f82245\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t8wx2" Jul 09 16:26:02.501091 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:02.501075 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43458597-f606-4740-b30d-c3b6a3f82245-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-t8wx2\" (UID: \"43458597-f606-4740-b30d-c3b6a3f82245\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t8wx2" Jul 09 16:26:02.501245 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:02.501225 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql6hq\" (UniqueName: \"kubernetes.io/projected/43458597-f606-4740-b30d-c3b6a3f82245-kube-api-access-ql6hq\") pod \"e2e-distinct-simulated-kserve-59975d69f9-t8wx2\" (UID: \"43458597-f606-4740-b30d-c3b6a3f82245\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t8wx2" Jul 09 16:26:02.501319 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:02.501293 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43458597-f606-4740-b30d-c3b6a3f82245-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-t8wx2\" (UID: \"43458597-f606-4740-b30d-c3b6a3f82245\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t8wx2" Jul 09 16:26:02.602119 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:02.602088 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43458597-f606-4740-b30d-c3b6a3f82245-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-t8wx2\" (UID: \"43458597-f606-4740-b30d-c3b6a3f82245\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t8wx2" Jul 09 16:26:02.602119 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:02.602121 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43458597-f606-4740-b30d-c3b6a3f82245-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-t8wx2\" (UID: \"43458597-f606-4740-b30d-c3b6a3f82245\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t8wx2" Jul 09 16:26:02.602394 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:02.602147 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43458597-f606-4740-b30d-c3b6a3f82245-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-t8wx2\" (UID: \"43458597-f606-4740-b30d-c3b6a3f82245\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t8wx2" Jul 09 16:26:02.602394 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:02.602181 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43458597-f606-4740-b30d-c3b6a3f82245-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-t8wx2\" (UID: \"43458597-f606-4740-b30d-c3b6a3f82245\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t8wx2" Jul 09 16:26:02.602394 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:02.602233 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql6hq\" (UniqueName: \"kubernetes.io/projected/43458597-f606-4740-b30d-c3b6a3f82245-kube-api-access-ql6hq\") pod \"e2e-distinct-simulated-kserve-59975d69f9-t8wx2\" (UID: \"43458597-f606-4740-b30d-c3b6a3f82245\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t8wx2" Jul 09 16:26:02.602394 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:02.602279 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43458597-f606-4740-b30d-c3b6a3f82245-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-t8wx2\" (UID: \"43458597-f606-4740-b30d-c3b6a3f82245\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t8wx2" Jul 09 16:26:02.602606 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:02.602585 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43458597-f606-4740-b30d-c3b6a3f82245-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-t8wx2\" (UID: \"43458597-f606-4740-b30d-c3b6a3f82245\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t8wx2" Jul 09 16:26:02.602692 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:02.602670 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43458597-f606-4740-b30d-c3b6a3f82245-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-t8wx2\" (UID: \"43458597-f606-4740-b30d-c3b6a3f82245\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t8wx2" Jul 09 16:26:02.602741 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:02.602680 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43458597-f606-4740-b30d-c3b6a3f82245-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-t8wx2\" (UID: \"43458597-f606-4740-b30d-c3b6a3f82245\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t8wx2" Jul 09 16:26:02.604575 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:02.604553 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43458597-f606-4740-b30d-c3b6a3f82245-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-t8wx2\" (UID: \"43458597-f606-4740-b30d-c3b6a3f82245\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t8wx2" Jul 09 16:26:02.604745 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:02.604729 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43458597-f606-4740-b30d-c3b6a3f82245-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-t8wx2\" (UID: \"43458597-f606-4740-b30d-c3b6a3f82245\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t8wx2" Jul 09 16:26:02.611368 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:02.611340 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql6hq\" (UniqueName: \"kubernetes.io/projected/43458597-f606-4740-b30d-c3b6a3f82245-kube-api-access-ql6hq\") pod \"e2e-distinct-simulated-kserve-59975d69f9-t8wx2\" (UID: \"43458597-f606-4740-b30d-c3b6a3f82245\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t8wx2" Jul 09 16:26:02.802110 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:02.802022 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t8wx2" Jul 09 16:26:02.943794 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:02.943761 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-t8wx2"] Jul 09 16:26:02.945283 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:26:02.945254 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43458597_f606_4740_b30d_c3b6a3f82245.slice/crio-b8de5ccc9a84bc2b07859f65be0e08d36e66d3a20a3ca70746c2fc4907cb04b6 WatchSource:0}: Error finding container b8de5ccc9a84bc2b07859f65be0e08d36e66d3a20a3ca70746c2fc4907cb04b6: Status 404 returned error can't find the container with id b8de5ccc9a84bc2b07859f65be0e08d36e66d3a20a3ca70746c2fc4907cb04b6 Jul 09 16:26:03.367662 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:03.367627 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t8wx2" event={"ID":"43458597-f606-4740-b30d-c3b6a3f82245","Type":"ContainerStarted","Data":"b8de5ccc9a84bc2b07859f65be0e08d36e66d3a20a3ca70746c2fc4907cb04b6"} Jul 09 16:26:04.374458 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:04.374357 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t8wx2" event={"ID":"43458597-f606-4740-b30d-c3b6a3f82245","Type":"ContainerStarted","Data":"21204e9a7c3ae564c187badd93bba1b5680937124f13c4a8861507508aee0ad5"} Jul 09 16:26:04.393506 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:04.393447 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t8wx2" podStartSLOduration=1.248324429 podStartE2EDuration="2.393430254s" totalImagesPullingTime="1.145105825s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:26:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:26:04.392857082 +0000 UTC m=+649.585963500" watchObservedRunningTime="2026-07-09 16:26:04.393430254 +0000 UTC m=+649.586536636" Jul 09 16:26:12.802570 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:12.802522 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t8wx2" Jul 09 16:26:12.803109 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:12.802594 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t8wx2" Jul 09 16:26:12.815421 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:12.815391 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t8wx2" Jul 09 16:26:13.427672 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:13.427639 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t8wx2" Jul 09 16:26:28.793184 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:28.793141 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnmpk9"] Jul 09 16:26:28.797153 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:28.797126 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnmpk9" Jul 09 16:26:28.799805 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:28.799770 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 09 16:26:28.805299 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:28.805274 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnmpk9"] Jul 09 16:26:28.949761 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:28.949711 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b052225-8cfe-4adb-b6e2-e1e0195efa56-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnmpk9\" (UID: \"9b052225-8cfe-4adb-b6e2-e1e0195efa56\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnmpk9" Jul 09 16:26:28.950016 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:28.949837 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b052225-8cfe-4adb-b6e2-e1e0195efa56-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnmpk9\" (UID: \"9b052225-8cfe-4adb-b6e2-e1e0195efa56\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnmpk9" Jul 09 16:26:28.950016 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:28.949953 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzkk6\" (UniqueName: \"kubernetes.io/projected/9b052225-8cfe-4adb-b6e2-e1e0195efa56-kube-api-access-mzkk6\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnmpk9\" (UID: \"9b052225-8cfe-4adb-b6e2-e1e0195efa56\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnmpk9" Jul 09 16:26:28.950160 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:28.950029 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b052225-8cfe-4adb-b6e2-e1e0195efa56-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnmpk9\" (UID: \"9b052225-8cfe-4adb-b6e2-e1e0195efa56\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnmpk9" Jul 09 16:26:28.950160 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:28.950075 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b052225-8cfe-4adb-b6e2-e1e0195efa56-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnmpk9\" (UID: \"9b052225-8cfe-4adb-b6e2-e1e0195efa56\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnmpk9" Jul 09 16:26:28.950160 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:28.950101 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b052225-8cfe-4adb-b6e2-e1e0195efa56-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnmpk9\" (UID: \"9b052225-8cfe-4adb-b6e2-e1e0195efa56\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnmpk9" Jul 09 16:26:29.051065 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:29.050963 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b052225-8cfe-4adb-b6e2-e1e0195efa56-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnmpk9\" (UID: \"9b052225-8cfe-4adb-b6e2-e1e0195efa56\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnmpk9" Jul 09 16:26:29.051065 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:29.051019 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzkk6\" (UniqueName: \"kubernetes.io/projected/9b052225-8cfe-4adb-b6e2-e1e0195efa56-kube-api-access-mzkk6\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnmpk9\" (UID: \"9b052225-8cfe-4adb-b6e2-e1e0195efa56\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnmpk9" Jul 09 16:26:29.051065 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:29.051062 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b052225-8cfe-4adb-b6e2-e1e0195efa56-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnmpk9\" (UID: \"9b052225-8cfe-4adb-b6e2-e1e0195efa56\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnmpk9" Jul 09 16:26:29.051347 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:29.051093 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b052225-8cfe-4adb-b6e2-e1e0195efa56-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnmpk9\" (UID: \"9b052225-8cfe-4adb-b6e2-e1e0195efa56\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnmpk9" Jul 09 16:26:29.051347 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:29.051108 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b052225-8cfe-4adb-b6e2-e1e0195efa56-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnmpk9\" (UID: \"9b052225-8cfe-4adb-b6e2-e1e0195efa56\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnmpk9" Jul 09 16:26:29.051347 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:29.051249 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b052225-8cfe-4adb-b6e2-e1e0195efa56-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnmpk9\" (UID: \"9b052225-8cfe-4adb-b6e2-e1e0195efa56\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnmpk9" Jul 09 16:26:29.051543 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:29.051517 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b052225-8cfe-4adb-b6e2-e1e0195efa56-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnmpk9\" (UID: \"9b052225-8cfe-4adb-b6e2-e1e0195efa56\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnmpk9" Jul 09 16:26:29.051543 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:29.051532 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b052225-8cfe-4adb-b6e2-e1e0195efa56-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnmpk9\" (UID: \"9b052225-8cfe-4adb-b6e2-e1e0195efa56\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnmpk9" Jul 09 16:26:29.051624 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:29.051588 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b052225-8cfe-4adb-b6e2-e1e0195efa56-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnmpk9\" (UID: \"9b052225-8cfe-4adb-b6e2-e1e0195efa56\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnmpk9" Jul 09 16:26:29.053343 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:29.053315 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b052225-8cfe-4adb-b6e2-e1e0195efa56-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnmpk9\" (UID: \"9b052225-8cfe-4adb-b6e2-e1e0195efa56\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnmpk9" Jul 09 16:26:29.053691 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:29.053671 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b052225-8cfe-4adb-b6e2-e1e0195efa56-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnmpk9\" (UID: \"9b052225-8cfe-4adb-b6e2-e1e0195efa56\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnmpk9" Jul 09 16:26:29.059316 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:29.059292 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzkk6\" (UniqueName: \"kubernetes.io/projected/9b052225-8cfe-4adb-b6e2-e1e0195efa56-kube-api-access-mzkk6\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnmpk9\" (UID: \"9b052225-8cfe-4adb-b6e2-e1e0195efa56\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnmpk9" Jul 09 16:26:29.109506 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:29.109469 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnmpk9" Jul 09 16:26:29.255409 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:29.255382 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnmpk9"] Jul 09 16:26:29.259953 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:26:29.259916 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b052225_8cfe_4adb_b6e2_e1e0195efa56.slice/crio-9bf529a9c63622a43a9b0207697c74b115584e9f50fbffee44feae018363f6a2 WatchSource:0}: Error finding container 9bf529a9c63622a43a9b0207697c74b115584e9f50fbffee44feae018363f6a2: Status 404 returned error can't find the container with id 9bf529a9c63622a43a9b0207697c74b115584e9f50fbffee44feae018363f6a2 Jul 09 16:26:29.261660 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:29.261629 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 16:26:29.492831 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:29.492793 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnmpk9" event={"ID":"9b052225-8cfe-4adb-b6e2-e1e0195efa56","Type":"ContainerStarted","Data":"9bf529a9c63622a43a9b0207697c74b115584e9f50fbffee44feae018363f6a2"} Jul 09 16:26:30.499472 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:30.499436 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnmpk9" event={"ID":"9b052225-8cfe-4adb-b6e2-e1e0195efa56","Type":"ContainerStarted","Data":"56f427a1b22b8c3c8c2b7fb2a44a2e6e0a72951fede5f0d56211dc515148156d"} Jul 09 16:26:30.521973 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:30.521919 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnmpk9" podStartSLOduration=2.328804227 podStartE2EDuration="2.521902041s" totalImagesPullingTime="193.097814ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:26:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:26:30.521161225 +0000 UTC m=+675.714267608" watchObservedRunningTime="2026-07-09 16:26:30.521902041 +0000 UTC m=+675.715008440" Jul 09 16:26:39.110058 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:39.110021 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnmpk9" Jul 09 16:26:39.110635 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:39.110112 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnmpk9" Jul 09 16:26:39.122860 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:39.122827 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnmpk9" Jul 09 16:26:39.555623 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:39.555538 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnmpk9" Jul 09 16:26:49.083760 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:49.083726 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cp5nl"] Jul 09 16:26:49.087435 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:49.087406 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cp5nl" Jul 09 16:26:49.090701 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:49.090675 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 09 16:26:49.099738 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:49.099708 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cp5nl"] Jul 09 16:26:49.123920 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:49.123889 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fa5d4e6-ec46-432f-a0f8-e37e910a2725-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cp5nl\" (UID: \"3fa5d4e6-ec46-432f-a0f8-e37e910a2725\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cp5nl" Jul 09 16:26:49.124087 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:49.123961 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fa5d4e6-ec46-432f-a0f8-e37e910a2725-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cp5nl\" (UID: \"3fa5d4e6-ec46-432f-a0f8-e37e910a2725\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cp5nl" Jul 09 16:26:49.124087 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:49.124008 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fa5d4e6-ec46-432f-a0f8-e37e910a2725-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cp5nl\" (UID: \"3fa5d4e6-ec46-432f-a0f8-e37e910a2725\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cp5nl" Jul 09 16:26:49.124087 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:49.124024 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fa5d4e6-ec46-432f-a0f8-e37e910a2725-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cp5nl\" (UID: \"3fa5d4e6-ec46-432f-a0f8-e37e910a2725\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cp5nl" Jul 09 16:26:49.124087 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:49.124060 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2srf\" (UniqueName: \"kubernetes.io/projected/3fa5d4e6-ec46-432f-a0f8-e37e910a2725-kube-api-access-w2srf\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cp5nl\" (UID: \"3fa5d4e6-ec46-432f-a0f8-e37e910a2725\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cp5nl" Jul 09 16:26:49.124256 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:49.124133 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fa5d4e6-ec46-432f-a0f8-e37e910a2725-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cp5nl\" (UID: \"3fa5d4e6-ec46-432f-a0f8-e37e910a2725\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cp5nl" Jul 09 16:26:49.225468 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:49.225427 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fa5d4e6-ec46-432f-a0f8-e37e910a2725-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cp5nl\" (UID: \"3fa5d4e6-ec46-432f-a0f8-e37e910a2725\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cp5nl" Jul 09 16:26:49.225664 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:49.225476 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fa5d4e6-ec46-432f-a0f8-e37e910a2725-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cp5nl\" (UID: \"3fa5d4e6-ec46-432f-a0f8-e37e910a2725\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cp5nl" Jul 09 16:26:49.225664 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:49.225536 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fa5d4e6-ec46-432f-a0f8-e37e910a2725-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cp5nl\" (UID: \"3fa5d4e6-ec46-432f-a0f8-e37e910a2725\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cp5nl" Jul 09 16:26:49.225782 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:49.225666 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fa5d4e6-ec46-432f-a0f8-e37e910a2725-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cp5nl\" (UID: \"3fa5d4e6-ec46-432f-a0f8-e37e910a2725\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cp5nl" Jul 09 16:26:49.225782 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:49.225701 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fa5d4e6-ec46-432f-a0f8-e37e910a2725-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cp5nl\" (UID: \"3fa5d4e6-ec46-432f-a0f8-e37e910a2725\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cp5nl" Jul 09 16:26:49.225782 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:49.225721 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2srf\" (UniqueName: \"kubernetes.io/projected/3fa5d4e6-ec46-432f-a0f8-e37e910a2725-kube-api-access-w2srf\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cp5nl\" (UID: \"3fa5d4e6-ec46-432f-a0f8-e37e910a2725\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cp5nl" Jul 09 16:26:49.226026 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:49.226006 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fa5d4e6-ec46-432f-a0f8-e37e910a2725-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cp5nl\" (UID: \"3fa5d4e6-ec46-432f-a0f8-e37e910a2725\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cp5nl" Jul 09 16:26:49.226117 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:49.226094 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fa5d4e6-ec46-432f-a0f8-e37e910a2725-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cp5nl\" (UID: \"3fa5d4e6-ec46-432f-a0f8-e37e910a2725\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cp5nl" Jul 09 16:26:49.226276 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:49.226205 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fa5d4e6-ec46-432f-a0f8-e37e910a2725-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cp5nl\" (UID: \"3fa5d4e6-ec46-432f-a0f8-e37e910a2725\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cp5nl" Jul 09 16:26:49.227967 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:49.227944 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fa5d4e6-ec46-432f-a0f8-e37e910a2725-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cp5nl\" (UID: \"3fa5d4e6-ec46-432f-a0f8-e37e910a2725\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cp5nl" Jul 09 16:26:49.228166 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:49.228149 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fa5d4e6-ec46-432f-a0f8-e37e910a2725-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cp5nl\" (UID: \"3fa5d4e6-ec46-432f-a0f8-e37e910a2725\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cp5nl" Jul 09 16:26:49.237541 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:49.237521 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2srf\" (UniqueName: \"kubernetes.io/projected/3fa5d4e6-ec46-432f-a0f8-e37e910a2725-kube-api-access-w2srf\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-cp5nl\" (UID: \"3fa5d4e6-ec46-432f-a0f8-e37e910a2725\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cp5nl" Jul 09 16:26:49.398331 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:49.398295 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cp5nl" Jul 09 16:26:49.544508 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:49.544481 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cp5nl"] Jul 09 16:26:49.546599 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:26:49.546565 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3fa5d4e6_ec46_432f_a0f8_e37e910a2725.slice/crio-db90ea5a1009e6f4039e12bbc43c98f99af2885dce0a2f5c09fd6b5ea5444455 WatchSource:0}: Error finding container db90ea5a1009e6f4039e12bbc43c98f99af2885dce0a2f5c09fd6b5ea5444455: Status 404 returned error can't find the container with id db90ea5a1009e6f4039e12bbc43c98f99af2885dce0a2f5c09fd6b5ea5444455 Jul 09 16:26:49.597739 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:49.597702 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cp5nl" event={"ID":"3fa5d4e6-ec46-432f-a0f8-e37e910a2725","Type":"ContainerStarted","Data":"db90ea5a1009e6f4039e12bbc43c98f99af2885dce0a2f5c09fd6b5ea5444455"} Jul 09 16:26:50.604163 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:50.604119 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cp5nl" event={"ID":"3fa5d4e6-ec46-432f-a0f8-e37e910a2725","Type":"ContainerStarted","Data":"0c286ac70d2ad03184e93dc96fc02558e6533719fe6716af34613cb3b298c680"} Jul 09 16:26:50.623224 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:50.623163 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cp5nl" podStartSLOduration=1.390689332 podStartE2EDuration="1.62314625s" totalImagesPullingTime="232.456918ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:26:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:26:50.621226274 +0000 UTC m=+695.814332655" watchObservedRunningTime="2026-07-09 16:26:50.62314625 +0000 UTC m=+695.816252631" Jul 09 16:26:59.398722 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:59.398672 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cp5nl" Jul 09 16:26:59.399158 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:59.398736 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cp5nl" Jul 09 16:26:59.411564 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:59.411540 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cp5nl" Jul 09 16:26:59.660206 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:26:59.660122 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-cp5nl" Jul 09 16:27:31.627586 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:27:31.627536 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5cd8b6f45b-fpb44"] Jul 09 16:27:31.628224 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:27:31.627738 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5cd8b6f45b-fpb44" podUID="ab6b37a9-4810-4c69-9b98-e134ab2ae1f4" containerName="authorino" containerID="cri-o://5b83453181e84c4185d01efba0d8062845863a2746882ed3f33cabeb7a09d3a0" gracePeriod=30 Jul 09 16:27:31.813927 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:27:31.813895 2569 generic.go:356] "Generic (PLEG): container finished" podID="ab6b37a9-4810-4c69-9b98-e134ab2ae1f4" containerID="5b83453181e84c4185d01efba0d8062845863a2746882ed3f33cabeb7a09d3a0" exitCode=0 Jul 09 16:27:31.813927 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:27:31.813909 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5cd8b6f45b-fpb44" event={"ID":"ab6b37a9-4810-4c69-9b98-e134ab2ae1f4","Type":"ContainerDied","Data":"5b83453181e84c4185d01efba0d8062845863a2746882ed3f33cabeb7a09d3a0"} Jul 09 16:27:31.880639 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:27:31.880573 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5cd8b6f45b-fpb44" Jul 09 16:27:31.910101 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:27:31.910071 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sjfmb\" (UniqueName: \"kubernetes.io/projected/ab6b37a9-4810-4c69-9b98-e134ab2ae1f4-kube-api-access-sjfmb\") pod \"ab6b37a9-4810-4c69-9b98-e134ab2ae1f4\" (UID: \"ab6b37a9-4810-4c69-9b98-e134ab2ae1f4\") " Jul 09 16:27:31.910306 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:27:31.910149 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ab6b37a9-4810-4c69-9b98-e134ab2ae1f4-tls-cert\") pod \"ab6b37a9-4810-4c69-9b98-e134ab2ae1f4\" (UID: \"ab6b37a9-4810-4c69-9b98-e134ab2ae1f4\") " Jul 09 16:27:31.912196 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:27:31.912167 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab6b37a9-4810-4c69-9b98-e134ab2ae1f4-kube-api-access-sjfmb" (OuterVolumeSpecName: "kube-api-access-sjfmb") pod "ab6b37a9-4810-4c69-9b98-e134ab2ae1f4" (UID: "ab6b37a9-4810-4c69-9b98-e134ab2ae1f4"). InnerVolumeSpecName "kube-api-access-sjfmb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 16:27:31.923146 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:27:31.923113 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab6b37a9-4810-4c69-9b98-e134ab2ae1f4-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "ab6b37a9-4810-4c69-9b98-e134ab2ae1f4" (UID: "ab6b37a9-4810-4c69-9b98-e134ab2ae1f4"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 16:27:32.011696 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:27:32.011657 2569 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ab6b37a9-4810-4c69-9b98-e134ab2ae1f4-tls-cert\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:27:32.011696 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:27:32.011691 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjfmb\" (UniqueName: \"kubernetes.io/projected/ab6b37a9-4810-4c69-9b98-e134ab2ae1f4-kube-api-access-sjfmb\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:27:32.820493 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:27:32.820465 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5cd8b6f45b-fpb44" Jul 09 16:27:32.820933 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:27:32.820488 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5cd8b6f45b-fpb44" event={"ID":"ab6b37a9-4810-4c69-9b98-e134ab2ae1f4","Type":"ContainerDied","Data":"453cf903e22fe6ee9e6168df586f552c3ec745db54dfbb97df9db13aeb5c9ad4"} Jul 09 16:27:32.820933 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:27:32.820539 2569 scope.go:117] "RemoveContainer" containerID="5b83453181e84c4185d01efba0d8062845863a2746882ed3f33cabeb7a09d3a0" Jul 09 16:27:32.845928 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:27:32.845899 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5cd8b6f45b-fpb44"] Jul 09 16:27:32.850021 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:27:32.849993 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5cd8b6f45b-fpb44"] Jul 09 16:27:33.390179 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:27:33.390146 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab6b37a9-4810-4c69-9b98-e134ab2ae1f4" path="/var/lib/kubelet/pods/ab6b37a9-4810-4c69-9b98-e134ab2ae1f4/volumes" Jul 09 16:28:43.978175 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:28:43.978090 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-64fcf5b9b9-g6jmn"] Jul 09 16:28:43.978634 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:28:43.978319 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-64fcf5b9b9-g6jmn" podUID="ec0bea33-b563-4057-a750-8dc451111bd5" containerName="manager" containerID="cri-o://88dde7b4fd6c87312fd7135367925279bf88ddfdf4f8d5060aeaecb63a112a04" gracePeriod=30 Jul 09 16:28:44.176225 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:28:44.176148 2569 generic.go:356] "Generic (PLEG): container finished" podID="ec0bea33-b563-4057-a750-8dc451111bd5" containerID="88dde7b4fd6c87312fd7135367925279bf88ddfdf4f8d5060aeaecb63a112a04" exitCode=0 Jul 09 16:28:44.176383 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:28:44.176223 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-64fcf5b9b9-g6jmn" event={"ID":"ec0bea33-b563-4057-a750-8dc451111bd5","Type":"ContainerDied","Data":"88dde7b4fd6c87312fd7135367925279bf88ddfdf4f8d5060aeaecb63a112a04"} Jul 09 16:28:44.234041 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:28:44.233976 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-64fcf5b9b9-g6jmn" Jul 09 16:28:44.368340 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:28:44.368305 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jq9j2\" (UniqueName: \"kubernetes.io/projected/ec0bea33-b563-4057-a750-8dc451111bd5-kube-api-access-jq9j2\") pod \"ec0bea33-b563-4057-a750-8dc451111bd5\" (UID: \"ec0bea33-b563-4057-a750-8dc451111bd5\") " Jul 09 16:28:44.368537 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:28:44.368378 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ec0bea33-b563-4057-a750-8dc451111bd5-webhook-cert\") pod \"ec0bea33-b563-4057-a750-8dc451111bd5\" (UID: \"ec0bea33-b563-4057-a750-8dc451111bd5\") " Jul 09 16:28:44.370397 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:28:44.370366 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec0bea33-b563-4057-a750-8dc451111bd5-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "ec0bea33-b563-4057-a750-8dc451111bd5" (UID: "ec0bea33-b563-4057-a750-8dc451111bd5"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 16:28:44.370492 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:28:44.370471 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec0bea33-b563-4057-a750-8dc451111bd5-kube-api-access-jq9j2" (OuterVolumeSpecName: "kube-api-access-jq9j2") pod "ec0bea33-b563-4057-a750-8dc451111bd5" (UID: "ec0bea33-b563-4057-a750-8dc451111bd5"). InnerVolumeSpecName "kube-api-access-jq9j2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 16:28:44.469305 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:28:44.469271 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jq9j2\" (UniqueName: \"kubernetes.io/projected/ec0bea33-b563-4057-a750-8dc451111bd5-kube-api-access-jq9j2\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:28:44.469305 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:28:44.469304 2569 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ec0bea33-b563-4057-a750-8dc451111bd5-webhook-cert\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:28:45.182425 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:28:45.182396 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-64fcf5b9b9-g6jmn" Jul 09 16:28:45.182899 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:28:45.182395 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-64fcf5b9b9-g6jmn" event={"ID":"ec0bea33-b563-4057-a750-8dc451111bd5","Type":"ContainerDied","Data":"043aaa341dd8567e0fff76a0605b00a14e43518252a6ef8efeba7326858708e4"} Jul 09 16:28:45.182899 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:28:45.182548 2569 scope.go:117] "RemoveContainer" containerID="88dde7b4fd6c87312fd7135367925279bf88ddfdf4f8d5060aeaecb63a112a04" Jul 09 16:28:45.205821 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:28:45.205786 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-64fcf5b9b9-g6jmn"] Jul 09 16:28:45.213371 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:28:45.213343 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-64fcf5b9b9-g6jmn"] Jul 09 16:28:45.390255 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:28:45.390225 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec0bea33-b563-4057-a750-8dc451111bd5" path="/var/lib/kubelet/pods/ec0bea33-b563-4057-a750-8dc451111bd5/volumes" Jul 09 16:28:45.981796 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:28:45.981763 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-64fcf5b9b9-79wkk"] Jul 09 16:28:45.982176 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:28:45.982164 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec0bea33-b563-4057-a750-8dc451111bd5" containerName="manager" Jul 09 16:28:45.982231 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:28:45.982178 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec0bea33-b563-4057-a750-8dc451111bd5" containerName="manager" Jul 09 16:28:45.982231 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:28:45.982190 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab6b37a9-4810-4c69-9b98-e134ab2ae1f4" containerName="authorino" Jul 09 16:28:45.982231 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:28:45.982195 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab6b37a9-4810-4c69-9b98-e134ab2ae1f4" containerName="authorino" Jul 09 16:28:45.982328 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:28:45.982260 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab6b37a9-4810-4c69-9b98-e134ab2ae1f4" containerName="authorino" Jul 09 16:28:45.982328 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:28:45.982268 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec0bea33-b563-4057-a750-8dc451111bd5" containerName="manager" Jul 09 16:28:45.986836 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:28:45.986815 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-64fcf5b9b9-79wkk" Jul 09 16:28:45.990794 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:28:45.990770 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 09 16:28:45.990954 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:28:45.990771 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-l94hm\"" Jul 09 16:28:45.990954 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:28:45.990832 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 09 16:28:45.995840 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:28:45.995812 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-64fcf5b9b9-79wkk"] Jul 09 16:28:46.084147 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:28:46.084109 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn88h\" (UniqueName: \"kubernetes.io/projected/08a340b4-8031-4dd5-b4e9-dd683adeaa99-kube-api-access-xn88h\") pod \"maas-controller-64fcf5b9b9-79wkk\" (UID: \"08a340b4-8031-4dd5-b4e9-dd683adeaa99\") " pod="opendatahub/maas-controller-64fcf5b9b9-79wkk" Jul 09 16:28:46.084325 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:28:46.084215 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/08a340b4-8031-4dd5-b4e9-dd683adeaa99-webhook-cert\") pod \"maas-controller-64fcf5b9b9-79wkk\" (UID: \"08a340b4-8031-4dd5-b4e9-dd683adeaa99\") " pod="opendatahub/maas-controller-64fcf5b9b9-79wkk" Jul 09 16:28:46.185227 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:28:46.185193 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/08a340b4-8031-4dd5-b4e9-dd683adeaa99-webhook-cert\") pod \"maas-controller-64fcf5b9b9-79wkk\" (UID: \"08a340b4-8031-4dd5-b4e9-dd683adeaa99\") " pod="opendatahub/maas-controller-64fcf5b9b9-79wkk" Jul 09 16:28:46.185769 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:28:46.185259 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn88h\" (UniqueName: \"kubernetes.io/projected/08a340b4-8031-4dd5-b4e9-dd683adeaa99-kube-api-access-xn88h\") pod \"maas-controller-64fcf5b9b9-79wkk\" (UID: \"08a340b4-8031-4dd5-b4e9-dd683adeaa99\") " pod="opendatahub/maas-controller-64fcf5b9b9-79wkk" Jul 09 16:28:46.187885 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:28:46.187849 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/08a340b4-8031-4dd5-b4e9-dd683adeaa99-webhook-cert\") pod \"maas-controller-64fcf5b9b9-79wkk\" (UID: \"08a340b4-8031-4dd5-b4e9-dd683adeaa99\") " pod="opendatahub/maas-controller-64fcf5b9b9-79wkk" Jul 09 16:28:46.197943 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:28:46.197915 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn88h\" (UniqueName: \"kubernetes.io/projected/08a340b4-8031-4dd5-b4e9-dd683adeaa99-kube-api-access-xn88h\") pod \"maas-controller-64fcf5b9b9-79wkk\" (UID: \"08a340b4-8031-4dd5-b4e9-dd683adeaa99\") " pod="opendatahub/maas-controller-64fcf5b9b9-79wkk" Jul 09 16:28:46.298757 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:28:46.298656 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-64fcf5b9b9-79wkk" Jul 09 16:28:46.454187 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:28:46.454001 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-64fcf5b9b9-79wkk"] Jul 09 16:28:46.457308 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:28:46.457272 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08a340b4_8031_4dd5_b4e9_dd683adeaa99.slice/crio-1db04765e65559222fe2a7a4f6bdbaeae77ebb2ab8f4ad9649dea4804d070adb WatchSource:0}: Error finding container 1db04765e65559222fe2a7a4f6bdbaeae77ebb2ab8f4ad9649dea4804d070adb: Status 404 returned error can't find the container with id 1db04765e65559222fe2a7a4f6bdbaeae77ebb2ab8f4ad9649dea4804d070adb Jul 09 16:28:47.196105 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:28:47.196070 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-64fcf5b9b9-79wkk" event={"ID":"08a340b4-8031-4dd5-b4e9-dd683adeaa99","Type":"ContainerStarted","Data":"1db04765e65559222fe2a7a4f6bdbaeae77ebb2ab8f4ad9649dea4804d070adb"} Jul 09 16:28:48.202454 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:28:48.202418 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-64fcf5b9b9-79wkk" event={"ID":"08a340b4-8031-4dd5-b4e9-dd683adeaa99","Type":"ContainerStarted","Data":"5adc179cf09c40c3a34b96396a34cb4a9e25496b3d90070f6f1018dc310c6b93"} Jul 09 16:28:48.202844 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:28:48.202535 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-64fcf5b9b9-79wkk" Jul 09 16:28:48.221031 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:28:48.220979 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-64fcf5b9b9-79wkk" podStartSLOduration=2.46478443 podStartE2EDuration="3.220961949s" totalImagesPullingTime="756.177519ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:28:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:28:48.220321599 +0000 UTC m=+813.413427979" watchObservedRunningTime="2026-07-09 16:28:48.220961949 +0000 UTC m=+813.414068330" Jul 09 16:28:59.213444 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:28:59.213404 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-64fcf5b9b9-79wkk" Jul 09 16:30:00.144065 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:30:00.144027 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726910-cspxl"] Jul 09 16:30:00.147769 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:30:00.147747 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726910-cspxl" Jul 09 16:30:00.155766 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:30:00.155732 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrp2v\" (UniqueName: \"kubernetes.io/projected/821c0c28-cf66-4cea-af43-976f48085c77-kube-api-access-zrp2v\") pod \"maas-api-key-cleanup-29726910-cspxl\" (UID: \"821c0c28-cf66-4cea-af43-976f48085c77\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726910-cspxl" Jul 09 16:30:00.156136 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:30:00.156111 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726910-cspxl"] Jul 09 16:30:00.256285 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:30:00.256248 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrp2v\" (UniqueName: \"kubernetes.io/projected/821c0c28-cf66-4cea-af43-976f48085c77-kube-api-access-zrp2v\") pod \"maas-api-key-cleanup-29726910-cspxl\" (UID: \"821c0c28-cf66-4cea-af43-976f48085c77\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726910-cspxl" Jul 09 16:30:00.264952 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:30:00.264922 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrp2v\" (UniqueName: \"kubernetes.io/projected/821c0c28-cf66-4cea-af43-976f48085c77-kube-api-access-zrp2v\") pod \"maas-api-key-cleanup-29726910-cspxl\" (UID: \"821c0c28-cf66-4cea-af43-976f48085c77\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726910-cspxl" Jul 09 16:30:00.459170 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:30:00.459097 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726910-cspxl" Jul 09 16:30:00.601216 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:30:00.601189 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726910-cspxl"] Jul 09 16:30:00.603069 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:30:00.603027 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod821c0c28_cf66_4cea_af43_976f48085c77.slice/crio-fdd8fa9b6a967fb2a5a1dbe0bf3e62b8ac91778436fb8b9b6044736d0194fbb5 WatchSource:0}: Error finding container fdd8fa9b6a967fb2a5a1dbe0bf3e62b8ac91778436fb8b9b6044736d0194fbb5: Status 404 returned error can't find the container with id fdd8fa9b6a967fb2a5a1dbe0bf3e62b8ac91778436fb8b9b6044736d0194fbb5 Jul 09 16:30:01.570356 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:30:01.570316 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726910-cspxl" event={"ID":"821c0c28-cf66-4cea-af43-976f48085c77","Type":"ContainerStarted","Data":"fdd8fa9b6a967fb2a5a1dbe0bf3e62b8ac91778436fb8b9b6044736d0194fbb5"} Jul 09 16:30:03.583181 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:30:03.583088 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726910-cspxl" event={"ID":"821c0c28-cf66-4cea-af43-976f48085c77","Type":"ContainerStarted","Data":"a8c08af1e956d30f95f692eb2468746cebbd04c74f4af80c548d92d1800acc19"} Jul 09 16:30:03.600936 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:30:03.600869 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726910-cspxl" podStartSLOduration=1.331418165 podStartE2EDuration="3.600852504s" totalImagesPullingTime="2.269434339s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:30:03.599859364 +0000 UTC m=+888.792965745" watchObservedRunningTime="2026-07-09 16:30:03.600852504 +0000 UTC m=+888.793958892" Jul 09 16:30:15.295745 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:30:15.295671 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-nzrk9_a284ea64-610c-40f7-837b-fa0975c0a290/console-operator/2.log" Jul 09 16:30:15.297685 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:30:15.297661 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-nzrk9_a284ea64-610c-40f7-837b-fa0975c0a290/console-operator/2.log" Jul 09 16:30:15.303602 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:30:15.303577 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-49sqd_17313664-7a0f-472b-a597-39f8c5412bc3/ovn-acl-logging/0.log" Jul 09 16:30:15.305504 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:30:15.305480 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-49sqd_17313664-7a0f-472b-a597-39f8c5412bc3/ovn-acl-logging/0.log" Jul 09 16:31:00.011905 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:31:00.011840 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726910-cspxl"] Jul 09 16:31:00.014588 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:31:00.012060 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726910-cspxl" podUID="821c0c28-cf66-4cea-af43-976f48085c77" containerName="cleanup" containerID="cri-o://a8c08af1e956d30f95f692eb2468746cebbd04c74f4af80c548d92d1800acc19" gracePeriod=30 Jul 09 16:31:30.199712 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:31:30.199687 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726910-cspxl" Jul 09 16:31:30.239622 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:31:30.239595 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zrp2v\" (UniqueName: \"kubernetes.io/projected/821c0c28-cf66-4cea-af43-976f48085c77-kube-api-access-zrp2v\") pod \"821c0c28-cf66-4cea-af43-976f48085c77\" (UID: \"821c0c28-cf66-4cea-af43-976f48085c77\") " Jul 09 16:31:30.241781 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:31:30.241745 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/821c0c28-cf66-4cea-af43-976f48085c77-kube-api-access-zrp2v" (OuterVolumeSpecName: "kube-api-access-zrp2v") pod "821c0c28-cf66-4cea-af43-976f48085c77" (UID: "821c0c28-cf66-4cea-af43-976f48085c77"). InnerVolumeSpecName "kube-api-access-zrp2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 16:31:30.340925 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:31:30.340866 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrp2v\" (UniqueName: \"kubernetes.io/projected/821c0c28-cf66-4cea-af43-976f48085c77-kube-api-access-zrp2v\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:31:31.018835 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:31:31.018796 2569 generic.go:356] "Generic (PLEG): container finished" podID="821c0c28-cf66-4cea-af43-976f48085c77" containerID="a8c08af1e956d30f95f692eb2468746cebbd04c74f4af80c548d92d1800acc19" exitCode=137 Jul 09 16:31:31.019055 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:31:31.018853 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726910-cspxl" Jul 09 16:31:31.019055 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:31:31.018894 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726910-cspxl" event={"ID":"821c0c28-cf66-4cea-af43-976f48085c77","Type":"ContainerDied","Data":"a8c08af1e956d30f95f692eb2468746cebbd04c74f4af80c548d92d1800acc19"} Jul 09 16:31:31.019055 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:31:31.018944 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726910-cspxl" event={"ID":"821c0c28-cf66-4cea-af43-976f48085c77","Type":"ContainerDied","Data":"fdd8fa9b6a967fb2a5a1dbe0bf3e62b8ac91778436fb8b9b6044736d0194fbb5"} Jul 09 16:31:31.019055 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:31:31.018973 2569 scope.go:117] "RemoveContainer" containerID="a8c08af1e956d30f95f692eb2468746cebbd04c74f4af80c548d92d1800acc19" Jul 09 16:31:31.030346 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:31:31.030321 2569 scope.go:117] "RemoveContainer" containerID="a8c08af1e956d30f95f692eb2468746cebbd04c74f4af80c548d92d1800acc19" Jul 09 16:31:31.030939 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:31:31.030910 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8c08af1e956d30f95f692eb2468746cebbd04c74f4af80c548d92d1800acc19\": container with ID starting with a8c08af1e956d30f95f692eb2468746cebbd04c74f4af80c548d92d1800acc19 not found: ID does not exist" containerID="a8c08af1e956d30f95f692eb2468746cebbd04c74f4af80c548d92d1800acc19" Jul 09 16:31:31.031028 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:31:31.030950 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8c08af1e956d30f95f692eb2468746cebbd04c74f4af80c548d92d1800acc19"} err="failed to get container status \"a8c08af1e956d30f95f692eb2468746cebbd04c74f4af80c548d92d1800acc19\": rpc error: code = NotFound desc = could not find container \"a8c08af1e956d30f95f692eb2468746cebbd04c74f4af80c548d92d1800acc19\": container with ID starting with a8c08af1e956d30f95f692eb2468746cebbd04c74f4af80c548d92d1800acc19 not found: ID does not exist" Jul 09 16:31:31.040751 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:31:31.040727 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726910-cspxl"] Jul 09 16:31:31.045483 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:31:31.045458 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726910-cspxl"] Jul 09 16:31:31.390122 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:31:31.390091 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="821c0c28-cf66-4cea-af43-976f48085c77" path="/var/lib/kubelet/pods/821c0c28-cf66-4cea-af43-976f48085c77/volumes" Jul 09 16:35:15.326185 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:35:15.326152 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-nzrk9_a284ea64-610c-40f7-837b-fa0975c0a290/console-operator/2.log" Jul 09 16:35:15.329902 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:35:15.329851 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-nzrk9_a284ea64-610c-40f7-837b-fa0975c0a290/console-operator/2.log" Jul 09 16:35:15.333952 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:35:15.333929 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-49sqd_17313664-7a0f-472b-a597-39f8c5412bc3/ovn-acl-logging/0.log" Jul 09 16:35:15.337660 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:35:15.337637 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-49sqd_17313664-7a0f-472b-a597-39f8c5412bc3/ovn-acl-logging/0.log" Jul 09 16:40:15.356339 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:40:15.356308 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-nzrk9_a284ea64-610c-40f7-837b-fa0975c0a290/console-operator/2.log" Jul 09 16:40:15.360624 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:40:15.360598 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-nzrk9_a284ea64-610c-40f7-837b-fa0975c0a290/console-operator/2.log" Jul 09 16:40:15.363910 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:40:15.363865 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-49sqd_17313664-7a0f-472b-a597-39f8c5412bc3/ovn-acl-logging/0.log" Jul 09 16:40:15.368026 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:40:15.368007 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-49sqd_17313664-7a0f-472b-a597-39f8c5412bc3/ovn-acl-logging/0.log" Jul 09 16:41:08.423319 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:08.423278 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-36b77c4f-76864bbff5-bsglh"] Jul 09 16:41:08.423944 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:08.423845 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="821c0c28-cf66-4cea-af43-976f48085c77" containerName="cleanup" Jul 09 16:41:08.423944 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:08.423867 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="821c0c28-cf66-4cea-af43-976f48085c77" containerName="cleanup" Jul 09 16:41:08.424082 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:08.424003 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="821c0c28-cf66-4cea-af43-976f48085c77" containerName="cleanup" Jul 09 16:41:08.427143 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:08.427122 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-36b77c4f-76864bbff5-bsglh" Jul 09 16:41:08.431390 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:08.431370 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-36b77c4f-serving-cert\"" Jul 09 16:41:08.448700 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:08.448672 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-36b77c4f-76864bbff5-bsglh"] Jul 09 16:41:08.589995 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:08.589962 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c853d58c-7eba-4066-b153-cdda0bbd3d56-maas-api-tls\") pod \"maas-api-e2e-ait-36b77c4f-76864bbff5-bsglh\" (UID: \"c853d58c-7eba-4066-b153-cdda0bbd3d56\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-36b77c4f-76864bbff5-bsglh" Jul 09 16:41:08.589995 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:08.590000 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-788ww\" (UniqueName: \"kubernetes.io/projected/c853d58c-7eba-4066-b153-cdda0bbd3d56-kube-api-access-788ww\") pod \"maas-api-e2e-ait-36b77c4f-76864bbff5-bsglh\" (UID: \"c853d58c-7eba-4066-b153-cdda0bbd3d56\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-36b77c4f-76864bbff5-bsglh" Jul 09 16:41:08.590202 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:08.590166 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c853d58c-7eba-4066-b153-cdda0bbd3d56-openshift-service-ca\") pod \"maas-api-e2e-ait-36b77c4f-76864bbff5-bsglh\" (UID: \"c853d58c-7eba-4066-b153-cdda0bbd3d56\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-36b77c4f-76864bbff5-bsglh" Jul 09 16:41:08.691017 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:08.690925 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c853d58c-7eba-4066-b153-cdda0bbd3d56-openshift-service-ca\") pod \"maas-api-e2e-ait-36b77c4f-76864bbff5-bsglh\" (UID: \"c853d58c-7eba-4066-b153-cdda0bbd3d56\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-36b77c4f-76864bbff5-bsglh" Jul 09 16:41:08.691017 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:08.690979 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c853d58c-7eba-4066-b153-cdda0bbd3d56-maas-api-tls\") pod \"maas-api-e2e-ait-36b77c4f-76864bbff5-bsglh\" (UID: \"c853d58c-7eba-4066-b153-cdda0bbd3d56\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-36b77c4f-76864bbff5-bsglh" Jul 09 16:41:08.691017 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:08.691004 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-788ww\" (UniqueName: \"kubernetes.io/projected/c853d58c-7eba-4066-b153-cdda0bbd3d56-kube-api-access-788ww\") pod \"maas-api-e2e-ait-36b77c4f-76864bbff5-bsglh\" (UID: \"c853d58c-7eba-4066-b153-cdda0bbd3d56\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-36b77c4f-76864bbff5-bsglh" Jul 09 16:41:08.691586 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:08.691560 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c853d58c-7eba-4066-b153-cdda0bbd3d56-openshift-service-ca\") pod \"maas-api-e2e-ait-36b77c4f-76864bbff5-bsglh\" (UID: \"c853d58c-7eba-4066-b153-cdda0bbd3d56\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-36b77c4f-76864bbff5-bsglh" Jul 09 16:41:08.693852 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:08.693824 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c853d58c-7eba-4066-b153-cdda0bbd3d56-maas-api-tls\") pod \"maas-api-e2e-ait-36b77c4f-76864bbff5-bsglh\" (UID: \"c853d58c-7eba-4066-b153-cdda0bbd3d56\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-36b77c4f-76864bbff5-bsglh" Jul 09 16:41:08.704425 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:08.704396 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-788ww\" (UniqueName: \"kubernetes.io/projected/c853d58c-7eba-4066-b153-cdda0bbd3d56-kube-api-access-788ww\") pod \"maas-api-e2e-ait-36b77c4f-76864bbff5-bsglh\" (UID: \"c853d58c-7eba-4066-b153-cdda0bbd3d56\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-36b77c4f-76864bbff5-bsglh" Jul 09 16:41:08.737376 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:08.737339 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-36b77c4f-76864bbff5-bsglh" Jul 09 16:41:08.886014 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:08.885986 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-36b77c4f-76864bbff5-bsglh"] Jul 09 16:41:08.889189 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:41:08.889161 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc853d58c_7eba_4066_b153_cdda0bbd3d56.slice/crio-edd386d5f4866fb08f58baa64ec32963e29ae7aa30e55b03e43d3dfd598f232e WatchSource:0}: Error finding container edd386d5f4866fb08f58baa64ec32963e29ae7aa30e55b03e43d3dfd598f232e: Status 404 returned error can't find the container with id edd386d5f4866fb08f58baa64ec32963e29ae7aa30e55b03e43d3dfd598f232e Jul 09 16:41:08.890997 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:08.890975 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 16:41:09.863726 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:09.863689 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-36b77c4f-76864bbff5-bsglh" event={"ID":"c853d58c-7eba-4066-b153-cdda0bbd3d56","Type":"ContainerStarted","Data":"f5f93c1d883d96f06fd31412337081b79144203413f58fcde9c74b13e1ea0211"} Jul 09 16:41:09.864125 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:09.863743 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-36b77c4f-76864bbff5-bsglh" event={"ID":"c853d58c-7eba-4066-b153-cdda0bbd3d56","Type":"ContainerStarted","Data":"edd386d5f4866fb08f58baa64ec32963e29ae7aa30e55b03e43d3dfd598f232e"} Jul 09 16:41:09.864125 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:09.863787 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-36b77c4f-76864bbff5-bsglh" Jul 09 16:41:09.882235 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:09.882187 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-36b77c4f-76864bbff5-bsglh" podStartSLOduration=1.389445608 podStartE2EDuration="1.882169856s" totalImagesPullingTime="492.724248ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:41:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:41:09.87996341 +0000 UTC m=+1555.073069793" watchObservedRunningTime="2026-07-09 16:41:09.882169856 +0000 UTC m=+1555.075276236" Jul 09 16:41:13.609570 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:13.609542 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-36b77c4f-76864bbff5-bsglh"] Jul 09 16:41:13.610023 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:13.609745 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-36b77c4f-76864bbff5-bsglh" podUID="c853d58c-7eba-4066-b153-cdda0bbd3d56" containerName="maas-api" containerID="cri-o://f5f93c1d883d96f06fd31412337081b79144203413f58fcde9c74b13e1ea0211" gracePeriod=30 Jul 09 16:41:13.860640 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:13.860565 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-36b77c4f-76864bbff5-bsglh" Jul 09 16:41:13.886727 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:13.886694 2569 generic.go:356] "Generic (PLEG): container finished" podID="c853d58c-7eba-4066-b153-cdda0bbd3d56" containerID="f5f93c1d883d96f06fd31412337081b79144203413f58fcde9c74b13e1ea0211" exitCode=0 Jul 09 16:41:13.886913 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:13.886740 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-36b77c4f-76864bbff5-bsglh" Jul 09 16:41:13.886913 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:13.886789 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-36b77c4f-76864bbff5-bsglh" event={"ID":"c853d58c-7eba-4066-b153-cdda0bbd3d56","Type":"ContainerDied","Data":"f5f93c1d883d96f06fd31412337081b79144203413f58fcde9c74b13e1ea0211"} Jul 09 16:41:13.886913 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:13.886860 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-36b77c4f-76864bbff5-bsglh" event={"ID":"c853d58c-7eba-4066-b153-cdda0bbd3d56","Type":"ContainerDied","Data":"edd386d5f4866fb08f58baa64ec32963e29ae7aa30e55b03e43d3dfd598f232e"} Jul 09 16:41:13.886913 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:13.886912 2569 scope.go:117] "RemoveContainer" containerID="f5f93c1d883d96f06fd31412337081b79144203413f58fcde9c74b13e1ea0211" Jul 09 16:41:13.897013 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:13.896987 2569 scope.go:117] "RemoveContainer" containerID="f5f93c1d883d96f06fd31412337081b79144203413f58fcde9c74b13e1ea0211" Jul 09 16:41:13.897408 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:41:13.897384 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5f93c1d883d96f06fd31412337081b79144203413f58fcde9c74b13e1ea0211\": container with ID starting with f5f93c1d883d96f06fd31412337081b79144203413f58fcde9c74b13e1ea0211 not found: ID does not exist" containerID="f5f93c1d883d96f06fd31412337081b79144203413f58fcde9c74b13e1ea0211" Jul 09 16:41:13.897494 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:13.897418 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5f93c1d883d96f06fd31412337081b79144203413f58fcde9c74b13e1ea0211"} err="failed to get container status \"f5f93c1d883d96f06fd31412337081b79144203413f58fcde9c74b13e1ea0211\": rpc error: code = NotFound desc = could not find container \"f5f93c1d883d96f06fd31412337081b79144203413f58fcde9c74b13e1ea0211\": container with ID starting with f5f93c1d883d96f06fd31412337081b79144203413f58fcde9c74b13e1ea0211 not found: ID does not exist" Jul 09 16:41:14.041716 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:14.041682 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-788ww\" (UniqueName: \"kubernetes.io/projected/c853d58c-7eba-4066-b153-cdda0bbd3d56-kube-api-access-788ww\") pod \"c853d58c-7eba-4066-b153-cdda0bbd3d56\" (UID: \"c853d58c-7eba-4066-b153-cdda0bbd3d56\") " Jul 09 16:41:14.041935 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:14.041753 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c853d58c-7eba-4066-b153-cdda0bbd3d56-openshift-service-ca\") pod \"c853d58c-7eba-4066-b153-cdda0bbd3d56\" (UID: \"c853d58c-7eba-4066-b153-cdda0bbd3d56\") " Jul 09 16:41:14.041935 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:14.041797 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c853d58c-7eba-4066-b153-cdda0bbd3d56-maas-api-tls\") pod \"c853d58c-7eba-4066-b153-cdda0bbd3d56\" (UID: \"c853d58c-7eba-4066-b153-cdda0bbd3d56\") " Jul 09 16:41:14.042186 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:14.042162 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c853d58c-7eba-4066-b153-cdda0bbd3d56-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c853d58c-7eba-4066-b153-cdda0bbd3d56" (UID: "c853d58c-7eba-4066-b153-cdda0bbd3d56"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 16:41:14.043809 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:14.043780 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c853d58c-7eba-4066-b153-cdda0bbd3d56-kube-api-access-788ww" (OuterVolumeSpecName: "kube-api-access-788ww") pod "c853d58c-7eba-4066-b153-cdda0bbd3d56" (UID: "c853d58c-7eba-4066-b153-cdda0bbd3d56"). InnerVolumeSpecName "kube-api-access-788ww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 16:41:14.043809 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:14.043801 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c853d58c-7eba-4066-b153-cdda0bbd3d56-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c853d58c-7eba-4066-b153-cdda0bbd3d56" (UID: "c853d58c-7eba-4066-b153-cdda0bbd3d56"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 16:41:14.142769 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:14.142674 2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c853d58c-7eba-4066-b153-cdda0bbd3d56-maas-api-tls\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:41:14.142769 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:14.142706 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-788ww\" (UniqueName: \"kubernetes.io/projected/c853d58c-7eba-4066-b153-cdda0bbd3d56-kube-api-access-788ww\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:41:14.142769 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:14.142717 2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c853d58c-7eba-4066-b153-cdda0bbd3d56-openshift-service-ca\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:41:14.209994 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:14.209962 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-36b77c4f-76864bbff5-bsglh"] Jul 09 16:41:14.211726 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:14.211699 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-36b77c4f-76864bbff5-bsglh"] Jul 09 16:41:15.389312 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:15.389280 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c853d58c-7eba-4066-b153-cdda0bbd3d56" path="/var/lib/kubelet/pods/c853d58c-7eba-4066-b153-cdda0bbd3d56/volumes" Jul 09 16:41:20.966644 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:20.966606 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e2c0cec8-69948c8c68-xvhk7"] Jul 09 16:41:20.967165 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:20.967090 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c853d58c-7eba-4066-b153-cdda0bbd3d56" containerName="maas-api" Jul 09 16:41:20.967165 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:20.967103 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c853d58c-7eba-4066-b153-cdda0bbd3d56" containerName="maas-api" Jul 09 16:41:20.967299 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:20.967195 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="c853d58c-7eba-4066-b153-cdda0bbd3d56" containerName="maas-api" Jul 09 16:41:20.970395 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:20.970368 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2c0cec8-69948c8c68-xvhk7" Jul 09 16:41:20.973173 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:20.973151 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-e2c0cec8-serving-cert\"" Jul 09 16:41:20.982398 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:20.982371 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e2c0cec8-69948c8c68-xvhk7"] Jul 09 16:41:21.104550 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:21.104505 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34bd6557-3478-4f6a-ac80-32ac12d8b462-maas-api-tls\") pod \"maas-api-e2e-ait-e2c0cec8-69948c8c68-xvhk7\" (UID: \"34bd6557-3478-4f6a-ac80-32ac12d8b462\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2c0cec8-69948c8c68-xvhk7" Jul 09 16:41:21.104729 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:21.104561 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34bd6557-3478-4f6a-ac80-32ac12d8b462-openshift-service-ca\") pod \"maas-api-e2e-ait-e2c0cec8-69948c8c68-xvhk7\" (UID: \"34bd6557-3478-4f6a-ac80-32ac12d8b462\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2c0cec8-69948c8c68-xvhk7" Jul 09 16:41:21.104729 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:21.104588 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgdvv\" (UniqueName: \"kubernetes.io/projected/34bd6557-3478-4f6a-ac80-32ac12d8b462-kube-api-access-bgdvv\") pod \"maas-api-e2e-ait-e2c0cec8-69948c8c68-xvhk7\" (UID: \"34bd6557-3478-4f6a-ac80-32ac12d8b462\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2c0cec8-69948c8c68-xvhk7" Jul 09 16:41:21.205997 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:21.205953 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34bd6557-3478-4f6a-ac80-32ac12d8b462-maas-api-tls\") pod \"maas-api-e2e-ait-e2c0cec8-69948c8c68-xvhk7\" (UID: \"34bd6557-3478-4f6a-ac80-32ac12d8b462\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2c0cec8-69948c8c68-xvhk7" Jul 09 16:41:21.206206 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:21.206004 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34bd6557-3478-4f6a-ac80-32ac12d8b462-openshift-service-ca\") pod \"maas-api-e2e-ait-e2c0cec8-69948c8c68-xvhk7\" (UID: \"34bd6557-3478-4f6a-ac80-32ac12d8b462\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2c0cec8-69948c8c68-xvhk7" Jul 09 16:41:21.206206 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:21.206031 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgdvv\" (UniqueName: \"kubernetes.io/projected/34bd6557-3478-4f6a-ac80-32ac12d8b462-kube-api-access-bgdvv\") pod \"maas-api-e2e-ait-e2c0cec8-69948c8c68-xvhk7\" (UID: \"34bd6557-3478-4f6a-ac80-32ac12d8b462\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2c0cec8-69948c8c68-xvhk7" Jul 09 16:41:21.206206 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:41:21.206114 2569 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-e2c0cec8-serving-cert: secret "maas-api-e2e-ait-e2c0cec8-serving-cert" not found Jul 09 16:41:21.206206 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:41:21.206195 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34bd6557-3478-4f6a-ac80-32ac12d8b462-maas-api-tls podName:34bd6557-3478-4f6a-ac80-32ac12d8b462 nodeName:}" failed. No retries permitted until 2026-07-09 16:41:21.706174362 +0000 UTC m=+1566.899280722 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/34bd6557-3478-4f6a-ac80-32ac12d8b462-maas-api-tls") pod "maas-api-e2e-ait-e2c0cec8-69948c8c68-xvhk7" (UID: "34bd6557-3478-4f6a-ac80-32ac12d8b462") : secret "maas-api-e2e-ait-e2c0cec8-serving-cert" not found Jul 09 16:41:21.206698 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:21.206674 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34bd6557-3478-4f6a-ac80-32ac12d8b462-openshift-service-ca\") pod \"maas-api-e2e-ait-e2c0cec8-69948c8c68-xvhk7\" (UID: \"34bd6557-3478-4f6a-ac80-32ac12d8b462\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2c0cec8-69948c8c68-xvhk7" Jul 09 16:41:21.214950 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:21.214920 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgdvv\" (UniqueName: \"kubernetes.io/projected/34bd6557-3478-4f6a-ac80-32ac12d8b462-kube-api-access-bgdvv\") pod \"maas-api-e2e-ait-e2c0cec8-69948c8c68-xvhk7\" (UID: \"34bd6557-3478-4f6a-ac80-32ac12d8b462\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2c0cec8-69948c8c68-xvhk7" Jul 09 16:41:21.711317 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:21.711274 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34bd6557-3478-4f6a-ac80-32ac12d8b462-maas-api-tls\") pod \"maas-api-e2e-ait-e2c0cec8-69948c8c68-xvhk7\" (UID: \"34bd6557-3478-4f6a-ac80-32ac12d8b462\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2c0cec8-69948c8c68-xvhk7" Jul 09 16:41:21.713751 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:21.713727 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34bd6557-3478-4f6a-ac80-32ac12d8b462-maas-api-tls\") pod \"maas-api-e2e-ait-e2c0cec8-69948c8c68-xvhk7\" (UID: \"34bd6557-3478-4f6a-ac80-32ac12d8b462\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2c0cec8-69948c8c68-xvhk7" Jul 09 16:41:21.883768 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:21.883727 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2c0cec8-69948c8c68-xvhk7" Jul 09 16:41:22.032096 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:22.032062 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e2c0cec8-69948c8c68-xvhk7"] Jul 09 16:41:22.034284 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:41:22.034261 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34bd6557_3478_4f6a_ac80_32ac12d8b462.slice/crio-a7bc95a9b1a5f4f1abad063986c9e81b1cefdd5f41ef4d5be60ad0641d24aa33 WatchSource:0}: Error finding container a7bc95a9b1a5f4f1abad063986c9e81b1cefdd5f41ef4d5be60ad0641d24aa33: Status 404 returned error can't find the container with id a7bc95a9b1a5f4f1abad063986c9e81b1cefdd5f41ef4d5be60ad0641d24aa33 Jul 09 16:41:22.936582 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:22.936544 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2c0cec8-69948c8c68-xvhk7" event={"ID":"34bd6557-3478-4f6a-ac80-32ac12d8b462","Type":"ContainerStarted","Data":"ba6effe168a503384cb972b26635fffa8873f643d573a0952a2c0a026f1d2c80"} Jul 09 16:41:22.936744 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:22.936606 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2c0cec8-69948c8c68-xvhk7" event={"ID":"34bd6557-3478-4f6a-ac80-32ac12d8b462","Type":"ContainerStarted","Data":"a7bc95a9b1a5f4f1abad063986c9e81b1cefdd5f41ef4d5be60ad0641d24aa33"} Jul 09 16:41:22.936744 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:22.936712 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2c0cec8-69948c8c68-xvhk7" Jul 09 16:41:22.955346 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:22.955297 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2c0cec8-69948c8c68-xvhk7" podStartSLOduration=2.460021709 podStartE2EDuration="2.955278429s" totalImagesPullingTime="495.25672ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:41:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:41:22.954158545 +0000 UTC m=+1568.147264927" watchObservedRunningTime="2026-07-09 16:41:22.955278429 +0000 UTC m=+1568.148384809" Jul 09 16:41:26.292431 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:26.292400 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e2c0cec8-69948c8c68-xvhk7"] Jul 09 16:41:26.292909 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:26.292686 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2c0cec8-69948c8c68-xvhk7" podUID="34bd6557-3478-4f6a-ac80-32ac12d8b462" containerName="maas-api" containerID="cri-o://ba6effe168a503384cb972b26635fffa8873f643d573a0952a2c0a026f1d2c80" gracePeriod=30 Jul 09 16:41:26.545096 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:26.545031 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2c0cec8-69948c8c68-xvhk7" Jul 09 16:41:26.550870 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:26.550841 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34bd6557-3478-4f6a-ac80-32ac12d8b462-openshift-service-ca\") pod \"34bd6557-3478-4f6a-ac80-32ac12d8b462\" (UID: \"34bd6557-3478-4f6a-ac80-32ac12d8b462\") " Jul 09 16:41:26.551039 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:26.550900 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bgdvv\" (UniqueName: \"kubernetes.io/projected/34bd6557-3478-4f6a-ac80-32ac12d8b462-kube-api-access-bgdvv\") pod \"34bd6557-3478-4f6a-ac80-32ac12d8b462\" (UID: \"34bd6557-3478-4f6a-ac80-32ac12d8b462\") " Jul 09 16:41:26.551039 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:26.550964 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34bd6557-3478-4f6a-ac80-32ac12d8b462-maas-api-tls\") pod \"34bd6557-3478-4f6a-ac80-32ac12d8b462\" (UID: \"34bd6557-3478-4f6a-ac80-32ac12d8b462\") " Jul 09 16:41:26.551214 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:26.551183 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34bd6557-3478-4f6a-ac80-32ac12d8b462-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "34bd6557-3478-4f6a-ac80-32ac12d8b462" (UID: "34bd6557-3478-4f6a-ac80-32ac12d8b462"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 16:41:26.551325 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:26.551258 2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34bd6557-3478-4f6a-ac80-32ac12d8b462-openshift-service-ca\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:41:26.553054 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:26.553018 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34bd6557-3478-4f6a-ac80-32ac12d8b462-kube-api-access-bgdvv" (OuterVolumeSpecName: "kube-api-access-bgdvv") pod "34bd6557-3478-4f6a-ac80-32ac12d8b462" (UID: "34bd6557-3478-4f6a-ac80-32ac12d8b462"). InnerVolumeSpecName "kube-api-access-bgdvv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 16:41:26.553174 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:26.553120 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34bd6557-3478-4f6a-ac80-32ac12d8b462-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "34bd6557-3478-4f6a-ac80-32ac12d8b462" (UID: "34bd6557-3478-4f6a-ac80-32ac12d8b462"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 16:41:26.652599 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:26.652560 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgdvv\" (UniqueName: \"kubernetes.io/projected/34bd6557-3478-4f6a-ac80-32ac12d8b462-kube-api-access-bgdvv\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:41:26.652599 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:26.652594 2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34bd6557-3478-4f6a-ac80-32ac12d8b462-maas-api-tls\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:41:26.956204 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:26.956168 2569 generic.go:356] "Generic (PLEG): container finished" podID="34bd6557-3478-4f6a-ac80-32ac12d8b462" containerID="ba6effe168a503384cb972b26635fffa8873f643d573a0952a2c0a026f1d2c80" exitCode=0 Jul 09 16:41:26.956379 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:26.956213 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2c0cec8-69948c8c68-xvhk7" Jul 09 16:41:26.956379 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:26.956247 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2c0cec8-69948c8c68-xvhk7" event={"ID":"34bd6557-3478-4f6a-ac80-32ac12d8b462","Type":"ContainerDied","Data":"ba6effe168a503384cb972b26635fffa8873f643d573a0952a2c0a026f1d2c80"} Jul 09 16:41:26.956379 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:26.956303 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e2c0cec8-69948c8c68-xvhk7" event={"ID":"34bd6557-3478-4f6a-ac80-32ac12d8b462","Type":"ContainerDied","Data":"a7bc95a9b1a5f4f1abad063986c9e81b1cefdd5f41ef4d5be60ad0641d24aa33"} Jul 09 16:41:26.956379 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:26.956327 2569 scope.go:117] "RemoveContainer" containerID="ba6effe168a503384cb972b26635fffa8873f643d573a0952a2c0a026f1d2c80" Jul 09 16:41:26.966055 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:26.965990 2569 scope.go:117] "RemoveContainer" containerID="ba6effe168a503384cb972b26635fffa8873f643d573a0952a2c0a026f1d2c80" Jul 09 16:41:26.966361 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:41:26.966341 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba6effe168a503384cb972b26635fffa8873f643d573a0952a2c0a026f1d2c80\": container with ID starting with ba6effe168a503384cb972b26635fffa8873f643d573a0952a2c0a026f1d2c80 not found: ID does not exist" containerID="ba6effe168a503384cb972b26635fffa8873f643d573a0952a2c0a026f1d2c80" Jul 09 16:41:26.966429 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:26.966374 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba6effe168a503384cb972b26635fffa8873f643d573a0952a2c0a026f1d2c80"} err="failed to get container status \"ba6effe168a503384cb972b26635fffa8873f643d573a0952a2c0a026f1d2c80\": rpc error: code = NotFound desc = could not find container \"ba6effe168a503384cb972b26635fffa8873f643d573a0952a2c0a026f1d2c80\": container with ID starting with ba6effe168a503384cb972b26635fffa8873f643d573a0952a2c0a026f1d2c80 not found: ID does not exist" Jul 09 16:41:26.980637 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:26.980609 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e2c0cec8-69948c8c68-xvhk7"] Jul 09 16:41:26.987327 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:26.987302 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e2c0cec8-69948c8c68-xvhk7"] Jul 09 16:41:27.389471 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:27.389438 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34bd6557-3478-4f6a-ac80-32ac12d8b462" path="/var/lib/kubelet/pods/34bd6557-3478-4f6a-ac80-32ac12d8b462/volumes" Jul 09 16:41:39.825936 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:39.825902 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-62c5cd5e-7fb9fd4cf9-snrjg"] Jul 09 16:41:39.826352 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:39.826318 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34bd6557-3478-4f6a-ac80-32ac12d8b462" containerName="maas-api" Jul 09 16:41:39.826352 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:39.826332 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="34bd6557-3478-4f6a-ac80-32ac12d8b462" containerName="maas-api" Jul 09 16:41:39.826422 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:39.826401 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="34bd6557-3478-4f6a-ac80-32ac12d8b462" containerName="maas-api" Jul 09 16:41:39.830534 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:39.830516 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-62c5cd5e-7fb9fd4cf9-snrjg" Jul 09 16:41:39.833785 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:39.833761 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-62c5cd5e-serving-cert\"" Jul 09 16:41:39.842239 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:39.842218 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-62c5cd5e-7fb9fd4cf9-snrjg"] Jul 09 16:41:39.872993 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:39.872962 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4540369d-562d-4fa5-b309-7bab95d6e3a5-openshift-service-ca\") pod \"maas-api-e2e-derive-62c5cd5e-7fb9fd4cf9-snrjg\" (UID: \"4540369d-562d-4fa5-b309-7bab95d6e3a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-62c5cd5e-7fb9fd4cf9-snrjg" Jul 09 16:41:39.873184 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:39.873021 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jfl4\" (UniqueName: \"kubernetes.io/projected/4540369d-562d-4fa5-b309-7bab95d6e3a5-kube-api-access-7jfl4\") pod \"maas-api-e2e-derive-62c5cd5e-7fb9fd4cf9-snrjg\" (UID: \"4540369d-562d-4fa5-b309-7bab95d6e3a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-62c5cd5e-7fb9fd4cf9-snrjg" Jul 09 16:41:39.873184 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:39.873090 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4540369d-562d-4fa5-b309-7bab95d6e3a5-maas-api-tls\") pod \"maas-api-e2e-derive-62c5cd5e-7fb9fd4cf9-snrjg\" (UID: \"4540369d-562d-4fa5-b309-7bab95d6e3a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-62c5cd5e-7fb9fd4cf9-snrjg" Jul 09 16:41:39.974440 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:39.974401 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4540369d-562d-4fa5-b309-7bab95d6e3a5-openshift-service-ca\") pod \"maas-api-e2e-derive-62c5cd5e-7fb9fd4cf9-snrjg\" (UID: \"4540369d-562d-4fa5-b309-7bab95d6e3a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-62c5cd5e-7fb9fd4cf9-snrjg" Jul 09 16:41:39.974613 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:39.974482 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jfl4\" (UniqueName: \"kubernetes.io/projected/4540369d-562d-4fa5-b309-7bab95d6e3a5-kube-api-access-7jfl4\") pod \"maas-api-e2e-derive-62c5cd5e-7fb9fd4cf9-snrjg\" (UID: \"4540369d-562d-4fa5-b309-7bab95d6e3a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-62c5cd5e-7fb9fd4cf9-snrjg" Jul 09 16:41:39.974613 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:39.974535 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4540369d-562d-4fa5-b309-7bab95d6e3a5-maas-api-tls\") pod \"maas-api-e2e-derive-62c5cd5e-7fb9fd4cf9-snrjg\" (UID: \"4540369d-562d-4fa5-b309-7bab95d6e3a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-62c5cd5e-7fb9fd4cf9-snrjg" Jul 09 16:41:39.975212 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:39.975189 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4540369d-562d-4fa5-b309-7bab95d6e3a5-openshift-service-ca\") pod \"maas-api-e2e-derive-62c5cd5e-7fb9fd4cf9-snrjg\" (UID: \"4540369d-562d-4fa5-b309-7bab95d6e3a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-62c5cd5e-7fb9fd4cf9-snrjg" Jul 09 16:41:39.976978 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:39.976956 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4540369d-562d-4fa5-b309-7bab95d6e3a5-maas-api-tls\") pod \"maas-api-e2e-derive-62c5cd5e-7fb9fd4cf9-snrjg\" (UID: \"4540369d-562d-4fa5-b309-7bab95d6e3a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-62c5cd5e-7fb9fd4cf9-snrjg" Jul 09 16:41:39.985861 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:39.985840 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jfl4\" (UniqueName: \"kubernetes.io/projected/4540369d-562d-4fa5-b309-7bab95d6e3a5-kube-api-access-7jfl4\") pod \"maas-api-e2e-derive-62c5cd5e-7fb9fd4cf9-snrjg\" (UID: \"4540369d-562d-4fa5-b309-7bab95d6e3a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-62c5cd5e-7fb9fd4cf9-snrjg" Jul 09 16:41:40.142065 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:40.142023 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-62c5cd5e-7fb9fd4cf9-snrjg" Jul 09 16:41:40.289796 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:40.289769 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-62c5cd5e-7fb9fd4cf9-snrjg"] Jul 09 16:41:40.292031 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:41:40.292005 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4540369d_562d_4fa5_b309_7bab95d6e3a5.slice/crio-8bf80ef4a4ada6dd92edff59ab695dfae928cb849eea1a04a4a3a254192d608c WatchSource:0}: Error finding container 8bf80ef4a4ada6dd92edff59ab695dfae928cb849eea1a04a4a3a254192d608c: Status 404 returned error can't find the container with id 8bf80ef4a4ada6dd92edff59ab695dfae928cb849eea1a04a4a3a254192d608c Jul 09 16:41:41.026777 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:41.026683 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-62c5cd5e-7fb9fd4cf9-snrjg" event={"ID":"4540369d-562d-4fa5-b309-7bab95d6e3a5","Type":"ContainerStarted","Data":"621d3a5ebc65aecf6d84ca22922cb9b22ec5df7e820eb67a66bf7f43b91a74b4"} Jul 09 16:41:41.026777 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:41.026744 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-62c5cd5e-7fb9fd4cf9-snrjg" event={"ID":"4540369d-562d-4fa5-b309-7bab95d6e3a5","Type":"ContainerStarted","Data":"8bf80ef4a4ada6dd92edff59ab695dfae928cb849eea1a04a4a3a254192d608c"} Jul 09 16:41:41.027420 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:41.026785 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-62c5cd5e-7fb9fd4cf9-snrjg" Jul 09 16:41:41.048904 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:41.048840 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-62c5cd5e-7fb9fd4cf9-snrjg" podStartSLOduration=1.6722198879999999 podStartE2EDuration="2.048820712s" totalImagesPullingTime="376.600824ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:41:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:41:41.047219208 +0000 UTC m=+1586.240325623" watchObservedRunningTime="2026-07-09 16:41:41.048820712 +0000 UTC m=+1586.241927093" Jul 09 16:41:44.539892 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:44.539847 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-62c5cd5e-7fb9fd4cf9-snrjg"] Jul 09 16:41:44.540331 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:44.540090 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-62c5cd5e-7fb9fd4cf9-snrjg" podUID="4540369d-562d-4fa5-b309-7bab95d6e3a5" containerName="maas-api" containerID="cri-o://621d3a5ebc65aecf6d84ca22922cb9b22ec5df7e820eb67a66bf7f43b91a74b4" gracePeriod=30 Jul 09 16:41:44.795714 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:44.795646 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-62c5cd5e-7fb9fd4cf9-snrjg" Jul 09 16:41:44.922207 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:44.922169 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7jfl4\" (UniqueName: \"kubernetes.io/projected/4540369d-562d-4fa5-b309-7bab95d6e3a5-kube-api-access-7jfl4\") pod \"4540369d-562d-4fa5-b309-7bab95d6e3a5\" (UID: \"4540369d-562d-4fa5-b309-7bab95d6e3a5\") " Jul 09 16:41:44.922384 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:44.922241 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4540369d-562d-4fa5-b309-7bab95d6e3a5-openshift-service-ca\") pod \"4540369d-562d-4fa5-b309-7bab95d6e3a5\" (UID: \"4540369d-562d-4fa5-b309-7bab95d6e3a5\") " Jul 09 16:41:44.922384 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:44.922262 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4540369d-562d-4fa5-b309-7bab95d6e3a5-maas-api-tls\") pod \"4540369d-562d-4fa5-b309-7bab95d6e3a5\" (UID: \"4540369d-562d-4fa5-b309-7bab95d6e3a5\") " Jul 09 16:41:44.922645 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:44.922613 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4540369d-562d-4fa5-b309-7bab95d6e3a5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4540369d-562d-4fa5-b309-7bab95d6e3a5" (UID: "4540369d-562d-4fa5-b309-7bab95d6e3a5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 16:41:44.924450 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:44.924414 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4540369d-562d-4fa5-b309-7bab95d6e3a5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4540369d-562d-4fa5-b309-7bab95d6e3a5" (UID: "4540369d-562d-4fa5-b309-7bab95d6e3a5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 16:41:44.924450 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:44.924429 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4540369d-562d-4fa5-b309-7bab95d6e3a5-kube-api-access-7jfl4" (OuterVolumeSpecName: "kube-api-access-7jfl4") pod "4540369d-562d-4fa5-b309-7bab95d6e3a5" (UID: "4540369d-562d-4fa5-b309-7bab95d6e3a5"). InnerVolumeSpecName "kube-api-access-7jfl4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 16:41:45.023846 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:45.023803 2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4540369d-562d-4fa5-b309-7bab95d6e3a5-openshift-service-ca\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:41:45.023846 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:45.023840 2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4540369d-562d-4fa5-b309-7bab95d6e3a5-maas-api-tls\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:41:45.023846 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:45.023849 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jfl4\" (UniqueName: \"kubernetes.io/projected/4540369d-562d-4fa5-b309-7bab95d6e3a5-kube-api-access-7jfl4\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:41:45.047400 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:45.047304 2569 generic.go:356] "Generic (PLEG): container finished" podID="4540369d-562d-4fa5-b309-7bab95d6e3a5" containerID="621d3a5ebc65aecf6d84ca22922cb9b22ec5df7e820eb67a66bf7f43b91a74b4" exitCode=0 Jul 09 16:41:45.047400 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:45.047354 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-62c5cd5e-7fb9fd4cf9-snrjg" Jul 09 16:41:45.047400 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:45.047368 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-62c5cd5e-7fb9fd4cf9-snrjg" event={"ID":"4540369d-562d-4fa5-b309-7bab95d6e3a5","Type":"ContainerDied","Data":"621d3a5ebc65aecf6d84ca22922cb9b22ec5df7e820eb67a66bf7f43b91a74b4"} Jul 09 16:41:45.047677 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:45.047448 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-62c5cd5e-7fb9fd4cf9-snrjg" event={"ID":"4540369d-562d-4fa5-b309-7bab95d6e3a5","Type":"ContainerDied","Data":"8bf80ef4a4ada6dd92edff59ab695dfae928cb849eea1a04a4a3a254192d608c"} Jul 09 16:41:45.047677 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:45.047472 2569 scope.go:117] "RemoveContainer" containerID="621d3a5ebc65aecf6d84ca22922cb9b22ec5df7e820eb67a66bf7f43b91a74b4" Jul 09 16:41:45.056909 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:45.056832 2569 scope.go:117] "RemoveContainer" containerID="621d3a5ebc65aecf6d84ca22922cb9b22ec5df7e820eb67a66bf7f43b91a74b4" Jul 09 16:41:45.057159 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:41:45.057141 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"621d3a5ebc65aecf6d84ca22922cb9b22ec5df7e820eb67a66bf7f43b91a74b4\": container with ID starting with 621d3a5ebc65aecf6d84ca22922cb9b22ec5df7e820eb67a66bf7f43b91a74b4 not found: ID does not exist" containerID="621d3a5ebc65aecf6d84ca22922cb9b22ec5df7e820eb67a66bf7f43b91a74b4" Jul 09 16:41:45.057226 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:45.057167 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"621d3a5ebc65aecf6d84ca22922cb9b22ec5df7e820eb67a66bf7f43b91a74b4"} err="failed to get container status \"621d3a5ebc65aecf6d84ca22922cb9b22ec5df7e820eb67a66bf7f43b91a74b4\": rpc error: code = NotFound desc = could not find container \"621d3a5ebc65aecf6d84ca22922cb9b22ec5df7e820eb67a66bf7f43b91a74b4\": container with ID starting with 621d3a5ebc65aecf6d84ca22922cb9b22ec5df7e820eb67a66bf7f43b91a74b4 not found: ID does not exist" Jul 09 16:41:45.069188 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:45.069156 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-62c5cd5e-7fb9fd4cf9-snrjg"] Jul 09 16:41:45.075928 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:45.075904 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-62c5cd5e-7fb9fd4cf9-snrjg"] Jul 09 16:41:45.389140 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:41:45.389108 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4540369d-562d-4fa5-b309-7bab95d6e3a5" path="/var/lib/kubelet/pods/4540369d-562d-4fa5-b309-7bab95d6e3a5/volumes" Jul 09 16:43:29.392919 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:43:29.392887 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-840481d3-776f8fdcb8-hz9sw"] Jul 09 16:43:29.393458 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:43:29.393443 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4540369d-562d-4fa5-b309-7bab95d6e3a5" containerName="maas-api" Jul 09 16:43:29.393510 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:43:29.393460 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="4540369d-562d-4fa5-b309-7bab95d6e3a5" containerName="maas-api" Jul 09 16:43:29.393594 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:43:29.393584 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="4540369d-562d-4fa5-b309-7bab95d6e3a5" containerName="maas-api" Jul 09 16:43:29.397049 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:43:29.397028 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-840481d3-776f8fdcb8-hz9sw" Jul 09 16:43:29.401062 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:43:29.401039 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-840481d3-776f8fdcb8-hz9sw"] Jul 09 16:43:29.401411 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:43:29.401390 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-840481d3-serving-cert\"" Jul 09 16:43:29.570064 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:43:29.570029 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be279ace-96d6-497e-97ab-ce4e123930a4-openshift-service-ca\") pod \"maas-api-e2e-mt-840481d3-776f8fdcb8-hz9sw\" (UID: \"be279ace-96d6-497e-97ab-ce4e123930a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-840481d3-776f8fdcb8-hz9sw" Jul 09 16:43:29.570064 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:43:29.570076 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgsww\" (UniqueName: \"kubernetes.io/projected/be279ace-96d6-497e-97ab-ce4e123930a4-kube-api-access-hgsww\") pod \"maas-api-e2e-mt-840481d3-776f8fdcb8-hz9sw\" (UID: \"be279ace-96d6-497e-97ab-ce4e123930a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-840481d3-776f8fdcb8-hz9sw" Jul 09 16:43:29.570319 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:43:29.570239 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be279ace-96d6-497e-97ab-ce4e123930a4-maas-api-tls\") pod \"maas-api-e2e-mt-840481d3-776f8fdcb8-hz9sw\" (UID: \"be279ace-96d6-497e-97ab-ce4e123930a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-840481d3-776f8fdcb8-hz9sw" Jul 09 16:43:29.671068 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:43:29.670965 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be279ace-96d6-497e-97ab-ce4e123930a4-maas-api-tls\") pod \"maas-api-e2e-mt-840481d3-776f8fdcb8-hz9sw\" (UID: \"be279ace-96d6-497e-97ab-ce4e123930a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-840481d3-776f8fdcb8-hz9sw" Jul 09 16:43:29.671068 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:43:29.671028 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be279ace-96d6-497e-97ab-ce4e123930a4-openshift-service-ca\") pod \"maas-api-e2e-mt-840481d3-776f8fdcb8-hz9sw\" (UID: \"be279ace-96d6-497e-97ab-ce4e123930a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-840481d3-776f8fdcb8-hz9sw" Jul 09 16:43:29.671068 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:43:29.671070 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgsww\" (UniqueName: \"kubernetes.io/projected/be279ace-96d6-497e-97ab-ce4e123930a4-kube-api-access-hgsww\") pod \"maas-api-e2e-mt-840481d3-776f8fdcb8-hz9sw\" (UID: \"be279ace-96d6-497e-97ab-ce4e123930a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-840481d3-776f8fdcb8-hz9sw" Jul 09 16:43:29.671771 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:43:29.671743 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be279ace-96d6-497e-97ab-ce4e123930a4-openshift-service-ca\") pod \"maas-api-e2e-mt-840481d3-776f8fdcb8-hz9sw\" (UID: \"be279ace-96d6-497e-97ab-ce4e123930a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-840481d3-776f8fdcb8-hz9sw" Jul 09 16:43:29.673505 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:43:29.673480 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be279ace-96d6-497e-97ab-ce4e123930a4-maas-api-tls\") pod \"maas-api-e2e-mt-840481d3-776f8fdcb8-hz9sw\" (UID: \"be279ace-96d6-497e-97ab-ce4e123930a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-840481d3-776f8fdcb8-hz9sw" Jul 09 16:43:29.679892 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:43:29.679854 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgsww\" (UniqueName: \"kubernetes.io/projected/be279ace-96d6-497e-97ab-ce4e123930a4-kube-api-access-hgsww\") pod \"maas-api-e2e-mt-840481d3-776f8fdcb8-hz9sw\" (UID: \"be279ace-96d6-497e-97ab-ce4e123930a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-840481d3-776f8fdcb8-hz9sw" Jul 09 16:43:29.710081 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:43:29.710046 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-840481d3-776f8fdcb8-hz9sw" Jul 09 16:43:29.857478 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:43:29.857441 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-840481d3-776f8fdcb8-hz9sw"] Jul 09 16:43:29.859973 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:43:29.859943 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe279ace_96d6_497e_97ab_ce4e123930a4.slice/crio-fbd1d5f53fd38e7b4d1e8e9c143b8b13d7ce584ec6ff127f37f5083606d80c25 WatchSource:0}: Error finding container fbd1d5f53fd38e7b4d1e8e9c143b8b13d7ce584ec6ff127f37f5083606d80c25: Status 404 returned error can't find the container with id fbd1d5f53fd38e7b4d1e8e9c143b8b13d7ce584ec6ff127f37f5083606d80c25 Jul 09 16:43:30.572429 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:43:30.572394 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-840481d3-776f8fdcb8-hz9sw" event={"ID":"be279ace-96d6-497e-97ab-ce4e123930a4","Type":"ContainerStarted","Data":"e976c9c42ee7d02373ab69f136fb50e4f13cf9bc22ee0926fc41937f18fe4fc4"} Jul 09 16:43:30.572813 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:43:30.572459 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-840481d3-776f8fdcb8-hz9sw" event={"ID":"be279ace-96d6-497e-97ab-ce4e123930a4","Type":"ContainerStarted","Data":"fbd1d5f53fd38e7b4d1e8e9c143b8b13d7ce584ec6ff127f37f5083606d80c25"} Jul 09 16:43:30.572813 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:43:30.572484 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-840481d3-776f8fdcb8-hz9sw" Jul 09 16:43:30.591802 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:43:30.591749 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-840481d3-776f8fdcb8-hz9sw" podStartSLOduration=1.080746292 podStartE2EDuration="1.591729819s" totalImagesPullingTime="510.983527ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:43:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:43:30.589738837 +0000 UTC m=+1695.782845221" watchObservedRunningTime="2026-07-09 16:43:30.591729819 +0000 UTC m=+1695.784836201" Jul 09 16:43:36.582938 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:43:36.582897 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-840481d3-776f8fdcb8-hz9sw" Jul 09 16:43:41.393219 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:43:41.393094 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-840481d3-776f8fdcb8-hz9sw"] Jul 09 16:43:41.393682 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:43:41.393313 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-840481d3-776f8fdcb8-hz9sw" podUID="be279ace-96d6-497e-97ab-ce4e123930a4" containerName="maas-api" containerID="cri-o://e976c9c42ee7d02373ab69f136fb50e4f13cf9bc22ee0926fc41937f18fe4fc4" gracePeriod=30 Jul 09 16:43:41.629724 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:43:41.629695 2569 generic.go:356] "Generic (PLEG): container finished" podID="be279ace-96d6-497e-97ab-ce4e123930a4" containerID="e976c9c42ee7d02373ab69f136fb50e4f13cf9bc22ee0926fc41937f18fe4fc4" exitCode=0 Jul 09 16:43:41.629898 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:43:41.629766 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-840481d3-776f8fdcb8-hz9sw" event={"ID":"be279ace-96d6-497e-97ab-ce4e123930a4","Type":"ContainerDied","Data":"e976c9c42ee7d02373ab69f136fb50e4f13cf9bc22ee0926fc41937f18fe4fc4"} Jul 09 16:43:41.629898 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:43:41.629813 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-840481d3-776f8fdcb8-hz9sw" event={"ID":"be279ace-96d6-497e-97ab-ce4e123930a4","Type":"ContainerDied","Data":"fbd1d5f53fd38e7b4d1e8e9c143b8b13d7ce584ec6ff127f37f5083606d80c25"} Jul 09 16:43:41.629898 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:43:41.629831 2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fbd1d5f53fd38e7b4d1e8e9c143b8b13d7ce584ec6ff127f37f5083606d80c25" Jul 09 16:43:41.642122 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:43:41.642095 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-840481d3-776f8fdcb8-hz9sw" Jul 09 16:43:41.680201 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:43:41.680118 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hgsww\" (UniqueName: \"kubernetes.io/projected/be279ace-96d6-497e-97ab-ce4e123930a4-kube-api-access-hgsww\") pod \"be279ace-96d6-497e-97ab-ce4e123930a4\" (UID: \"be279ace-96d6-497e-97ab-ce4e123930a4\") " Jul 09 16:43:41.680201 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:43:41.680155 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be279ace-96d6-497e-97ab-ce4e123930a4-openshift-service-ca\") pod \"be279ace-96d6-497e-97ab-ce4e123930a4\" (UID: \"be279ace-96d6-497e-97ab-ce4e123930a4\") " Jul 09 16:43:41.680201 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:43:41.680179 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be279ace-96d6-497e-97ab-ce4e123930a4-maas-api-tls\") pod \"be279ace-96d6-497e-97ab-ce4e123930a4\" (UID: \"be279ace-96d6-497e-97ab-ce4e123930a4\") " Jul 09 16:43:41.680557 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:43:41.680528 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be279ace-96d6-497e-97ab-ce4e123930a4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "be279ace-96d6-497e-97ab-ce4e123930a4" (UID: "be279ace-96d6-497e-97ab-ce4e123930a4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 16:43:41.682437 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:43:41.682403 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be279ace-96d6-497e-97ab-ce4e123930a4-kube-api-access-hgsww" (OuterVolumeSpecName: "kube-api-access-hgsww") pod "be279ace-96d6-497e-97ab-ce4e123930a4" (UID: "be279ace-96d6-497e-97ab-ce4e123930a4"). InnerVolumeSpecName "kube-api-access-hgsww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 16:43:41.682437 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:43:41.682419 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be279ace-96d6-497e-97ab-ce4e123930a4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "be279ace-96d6-497e-97ab-ce4e123930a4" (UID: "be279ace-96d6-497e-97ab-ce4e123930a4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 16:43:41.780915 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:43:41.780835 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgsww\" (UniqueName: \"kubernetes.io/projected/be279ace-96d6-497e-97ab-ce4e123930a4-kube-api-access-hgsww\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:43:41.780915 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:43:41.780864 2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be279ace-96d6-497e-97ab-ce4e123930a4-openshift-service-ca\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:43:41.780915 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:43:41.780904 2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be279ace-96d6-497e-97ab-ce4e123930a4-maas-api-tls\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:43:42.634410 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:43:42.634378 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-840481d3-776f8fdcb8-hz9sw" Jul 09 16:43:42.658725 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:43:42.658691 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-840481d3-776f8fdcb8-hz9sw"] Jul 09 16:43:42.661083 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:43:42.661060 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-840481d3-776f8fdcb8-hz9sw"] Jul 09 16:43:43.389085 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:43:43.389049 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be279ace-96d6-497e-97ab-ce4e123930a4" path="/var/lib/kubelet/pods/be279ace-96d6-497e-97ab-ce4e123930a4/volumes" Jul 09 16:43:44.578547 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:43:44.578500 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-840481d3-776f8fdcb8-hz9sw" podUID="be279ace-96d6-497e-97ab-ce4e123930a4" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.52:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Jul 09 16:44:10.585391 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:44:10.585343 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7b08359f-7b6d85656d-drr4q"] Jul 09 16:44:10.586032 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:44:10.585856 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be279ace-96d6-497e-97ab-ce4e123930a4" containerName="maas-api" Jul 09 16:44:10.586032 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:44:10.585888 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="be279ace-96d6-497e-97ab-ce4e123930a4" containerName="maas-api" Jul 09 16:44:10.586032 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:44:10.585973 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="be279ace-96d6-497e-97ab-ce4e123930a4" containerName="maas-api" Jul 09 16:44:10.589248 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:44:10.589227 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b08359f-7b6d85656d-drr4q" Jul 09 16:44:10.592287 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:44:10.592261 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-7b08359f-serving-cert\"" Jul 09 16:44:10.604117 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:44:10.604085 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7b08359f-7b6d85656d-drr4q"] Jul 09 16:44:10.628171 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:44:10.628134 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f6361d8-9c8f-4b8f-aa8d-2c1747227e83-maas-api-tls\") pod \"maas-api-e2e-mt-7b08359f-7b6d85656d-drr4q\" (UID: \"0f6361d8-9c8f-4b8f-aa8d-2c1747227e83\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b08359f-7b6d85656d-drr4q" Jul 09 16:44:10.628349 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:44:10.628189 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f6361d8-9c8f-4b8f-aa8d-2c1747227e83-openshift-service-ca\") pod \"maas-api-e2e-mt-7b08359f-7b6d85656d-drr4q\" (UID: \"0f6361d8-9c8f-4b8f-aa8d-2c1747227e83\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b08359f-7b6d85656d-drr4q" Jul 09 16:44:10.628349 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:44:10.628224 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-645mm\" (UniqueName: \"kubernetes.io/projected/0f6361d8-9c8f-4b8f-aa8d-2c1747227e83-kube-api-access-645mm\") pod \"maas-api-e2e-mt-7b08359f-7b6d85656d-drr4q\" (UID: \"0f6361d8-9c8f-4b8f-aa8d-2c1747227e83\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b08359f-7b6d85656d-drr4q" Jul 09 16:44:10.729460 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:44:10.729416 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f6361d8-9c8f-4b8f-aa8d-2c1747227e83-maas-api-tls\") pod \"maas-api-e2e-mt-7b08359f-7b6d85656d-drr4q\" (UID: \"0f6361d8-9c8f-4b8f-aa8d-2c1747227e83\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b08359f-7b6d85656d-drr4q" Jul 09 16:44:10.729656 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:44:10.729501 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f6361d8-9c8f-4b8f-aa8d-2c1747227e83-openshift-service-ca\") pod \"maas-api-e2e-mt-7b08359f-7b6d85656d-drr4q\" (UID: \"0f6361d8-9c8f-4b8f-aa8d-2c1747227e83\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b08359f-7b6d85656d-drr4q" Jul 09 16:44:10.729656 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:44:10.729556 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-645mm\" (UniqueName: \"kubernetes.io/projected/0f6361d8-9c8f-4b8f-aa8d-2c1747227e83-kube-api-access-645mm\") pod \"maas-api-e2e-mt-7b08359f-7b6d85656d-drr4q\" (UID: \"0f6361d8-9c8f-4b8f-aa8d-2c1747227e83\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b08359f-7b6d85656d-drr4q" Jul 09 16:44:10.730287 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:44:10.730261 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f6361d8-9c8f-4b8f-aa8d-2c1747227e83-openshift-service-ca\") pod \"maas-api-e2e-mt-7b08359f-7b6d85656d-drr4q\" (UID: \"0f6361d8-9c8f-4b8f-aa8d-2c1747227e83\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b08359f-7b6d85656d-drr4q" Jul 09 16:44:10.732084 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:44:10.732059 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f6361d8-9c8f-4b8f-aa8d-2c1747227e83-maas-api-tls\") pod \"maas-api-e2e-mt-7b08359f-7b6d85656d-drr4q\" (UID: \"0f6361d8-9c8f-4b8f-aa8d-2c1747227e83\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b08359f-7b6d85656d-drr4q" Jul 09 16:44:10.739766 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:44:10.739737 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-645mm\" (UniqueName: \"kubernetes.io/projected/0f6361d8-9c8f-4b8f-aa8d-2c1747227e83-kube-api-access-645mm\") pod \"maas-api-e2e-mt-7b08359f-7b6d85656d-drr4q\" (UID: \"0f6361d8-9c8f-4b8f-aa8d-2c1747227e83\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b08359f-7b6d85656d-drr4q" Jul 09 16:44:10.900409 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:44:10.900366 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b08359f-7b6d85656d-drr4q" Jul 09 16:44:11.043816 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:44:11.043786 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7b08359f-7b6d85656d-drr4q"] Jul 09 16:44:11.046050 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:44:11.046025 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f6361d8_9c8f_4b8f_aa8d_2c1747227e83.slice/crio-13cc74b4512c13205b2c41e9190fa1780f00c8d7dd0b31841bfdff4fbd4323d8 WatchSource:0}: Error finding container 13cc74b4512c13205b2c41e9190fa1780f00c8d7dd0b31841bfdff4fbd4323d8: Status 404 returned error can't find the container with id 13cc74b4512c13205b2c41e9190fa1780f00c8d7dd0b31841bfdff4fbd4323d8 Jul 09 16:44:11.779087 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:44:11.778965 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b08359f-7b6d85656d-drr4q" event={"ID":"0f6361d8-9c8f-4b8f-aa8d-2c1747227e83","Type":"ContainerStarted","Data":"a139ce0781bfd9173f4f3ffa438360a9eb3e13c42e0497d13920b06c2812362b"} Jul 09 16:44:11.779087 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:44:11.779023 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b08359f-7b6d85656d-drr4q" event={"ID":"0f6361d8-9c8f-4b8f-aa8d-2c1747227e83","Type":"ContainerStarted","Data":"13cc74b4512c13205b2c41e9190fa1780f00c8d7dd0b31841bfdff4fbd4323d8"} Jul 09 16:44:11.779528 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:44:11.779150 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b08359f-7b6d85656d-drr4q" Jul 09 16:44:11.798151 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:44:11.798097 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b08359f-7b6d85656d-drr4q" podStartSLOduration=1.356360373 podStartE2EDuration="1.798076791s" totalImagesPullingTime="441.716418ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:44:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:44:11.797302639 +0000 UTC m=+1736.990409019" watchObservedRunningTime="2026-07-09 16:44:11.798076791 +0000 UTC m=+1736.991183174" Jul 09 16:44:17.790291 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:44:17.790260 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b08359f-7b6d85656d-drr4q" Jul 09 16:44:40.871350 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:44:40.871307 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-7b08359f/e2e-lifecycle-model-7b08359f-kserve-844f957dff-2jjnk"] Jul 09 16:44:40.875576 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:44:40.875550 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7b08359f/e2e-lifecycle-model-7b08359f-kserve-844f957dff-2jjnk" Jul 09 16:44:40.878479 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:44:40.878446 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-7b08359f\"/\"openshift-service-ca.crt\"" Jul 09 16:44:40.878618 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:44:40.878457 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-7b08359f\"/\"kube-root-ca.crt\"" Jul 09 16:44:40.879700 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:44:40.879678 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-7b08359f\"/\"e2e-lifecycle-model-7b08359f-kserve-self-signed-certs\"" Jul 09 16:44:40.879815 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:44:40.879730 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-7b08359f\"/\"default-dockercfg-2jw4f\"" Jul 09 16:44:40.884947 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:44:40.884863 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-7b08359f/e2e-lifecycle-model-7b08359f-kserve-844f957dff-2jjnk"] Jul 09 16:44:40.913481 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:44:40.913449 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b860675-4baa-4676-85c3-e0a0cd690ac9-dshm\") pod \"e2e-lifecycle-model-7b08359f-kserve-844f957dff-2jjnk\" (UID: \"6b860675-4baa-4676-85c3-e0a0cd690ac9\") " pod="ai-tenant-e2e-mt-7b08359f/e2e-lifecycle-model-7b08359f-kserve-844f957dff-2jjnk" Jul 09 16:44:40.913481 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:44:40.913486 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b860675-4baa-4676-85c3-e0a0cd690ac9-tls-certs\") pod \"e2e-lifecycle-model-7b08359f-kserve-844f957dff-2jjnk\" (UID: \"6b860675-4baa-4676-85c3-e0a0cd690ac9\") " pod="ai-tenant-e2e-mt-7b08359f/e2e-lifecycle-model-7b08359f-kserve-844f957dff-2jjnk" Jul 09 16:44:40.913718 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:44:40.913520 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b860675-4baa-4676-85c3-e0a0cd690ac9-home\") pod \"e2e-lifecycle-model-7b08359f-kserve-844f957dff-2jjnk\" (UID: \"6b860675-4baa-4676-85c3-e0a0cd690ac9\") " pod="ai-tenant-e2e-mt-7b08359f/e2e-lifecycle-model-7b08359f-kserve-844f957dff-2jjnk" Jul 09 16:44:40.913718 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:44:40.913580 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b860675-4baa-4676-85c3-e0a0cd690ac9-model-cache\") pod \"e2e-lifecycle-model-7b08359f-kserve-844f957dff-2jjnk\" (UID: \"6b860675-4baa-4676-85c3-e0a0cd690ac9\") " pod="ai-tenant-e2e-mt-7b08359f/e2e-lifecycle-model-7b08359f-kserve-844f957dff-2jjnk" Jul 09 16:44:40.913718 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:44:40.913635 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mt6l\" (UniqueName: \"kubernetes.io/projected/6b860675-4baa-4676-85c3-e0a0cd690ac9-kube-api-access-8mt6l\") pod \"e2e-lifecycle-model-7b08359f-kserve-844f957dff-2jjnk\" (UID: \"6b860675-4baa-4676-85c3-e0a0cd690ac9\") " pod="ai-tenant-e2e-mt-7b08359f/e2e-lifecycle-model-7b08359f-kserve-844f957dff-2jjnk" Jul 09 16:44:40.913718 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:44:40.913697 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b860675-4baa-4676-85c3-e0a0cd690ac9-tmp-dir\") pod \"e2e-lifecycle-model-7b08359f-kserve-844f957dff-2jjnk\" (UID: \"6b860675-4baa-4676-85c3-e0a0cd690ac9\") " pod="ai-tenant-e2e-mt-7b08359f/e2e-lifecycle-model-7b08359f-kserve-844f957dff-2jjnk" Jul 09 16:44:41.015118 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:44:41.015085 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b860675-4baa-4676-85c3-e0a0cd690ac9-model-cache\") pod \"e2e-lifecycle-model-7b08359f-kserve-844f957dff-2jjnk\" (UID: \"6b860675-4baa-4676-85c3-e0a0cd690ac9\") " pod="ai-tenant-e2e-mt-7b08359f/e2e-lifecycle-model-7b08359f-kserve-844f957dff-2jjnk" Jul 09 16:44:41.015118 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:44:41.015120 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mt6l\" (UniqueName: \"kubernetes.io/projected/6b860675-4baa-4676-85c3-e0a0cd690ac9-kube-api-access-8mt6l\") pod \"e2e-lifecycle-model-7b08359f-kserve-844f957dff-2jjnk\" (UID: \"6b860675-4baa-4676-85c3-e0a0cd690ac9\") " pod="ai-tenant-e2e-mt-7b08359f/e2e-lifecycle-model-7b08359f-kserve-844f957dff-2jjnk" Jul 09 16:44:41.015351 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:44:41.015185 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b860675-4baa-4676-85c3-e0a0cd690ac9-tmp-dir\") pod \"e2e-lifecycle-model-7b08359f-kserve-844f957dff-2jjnk\" (UID: \"6b860675-4baa-4676-85c3-e0a0cd690ac9\") " pod="ai-tenant-e2e-mt-7b08359f/e2e-lifecycle-model-7b08359f-kserve-844f957dff-2jjnk" Jul 09 16:44:41.015351 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:44:41.015237 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b860675-4baa-4676-85c3-e0a0cd690ac9-dshm\") pod \"e2e-lifecycle-model-7b08359f-kserve-844f957dff-2jjnk\" (UID: \"6b860675-4baa-4676-85c3-e0a0cd690ac9\") " pod="ai-tenant-e2e-mt-7b08359f/e2e-lifecycle-model-7b08359f-kserve-844f957dff-2jjnk" Jul 09 16:44:41.015351 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:44:41.015256 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b860675-4baa-4676-85c3-e0a0cd690ac9-tls-certs\") pod \"e2e-lifecycle-model-7b08359f-kserve-844f957dff-2jjnk\" (UID: \"6b860675-4baa-4676-85c3-e0a0cd690ac9\") " pod="ai-tenant-e2e-mt-7b08359f/e2e-lifecycle-model-7b08359f-kserve-844f957dff-2jjnk" Jul 09 16:44:41.015351 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:44:41.015284 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b860675-4baa-4676-85c3-e0a0cd690ac9-home\") pod \"e2e-lifecycle-model-7b08359f-kserve-844f957dff-2jjnk\" (UID: \"6b860675-4baa-4676-85c3-e0a0cd690ac9\") " pod="ai-tenant-e2e-mt-7b08359f/e2e-lifecycle-model-7b08359f-kserve-844f957dff-2jjnk" Jul 09 16:44:41.015635 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:44:41.015569 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b860675-4baa-4676-85c3-e0a0cd690ac9-home\") pod \"e2e-lifecycle-model-7b08359f-kserve-844f957dff-2jjnk\" (UID: \"6b860675-4baa-4676-85c3-e0a0cd690ac9\") " pod="ai-tenant-e2e-mt-7b08359f/e2e-lifecycle-model-7b08359f-kserve-844f957dff-2jjnk" Jul 09 16:44:41.015635 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:44:41.015553 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b860675-4baa-4676-85c3-e0a0cd690ac9-model-cache\") pod \"e2e-lifecycle-model-7b08359f-kserve-844f957dff-2jjnk\" (UID: \"6b860675-4baa-4676-85c3-e0a0cd690ac9\") " pod="ai-tenant-e2e-mt-7b08359f/e2e-lifecycle-model-7b08359f-kserve-844f957dff-2jjnk" Jul 09 16:44:41.015750 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:44:41.015656 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b860675-4baa-4676-85c3-e0a0cd690ac9-tmp-dir\") pod \"e2e-lifecycle-model-7b08359f-kserve-844f957dff-2jjnk\" (UID: \"6b860675-4baa-4676-85c3-e0a0cd690ac9\") " pod="ai-tenant-e2e-mt-7b08359f/e2e-lifecycle-model-7b08359f-kserve-844f957dff-2jjnk" Jul 09 16:44:41.017654 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:44:41.017631 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b860675-4baa-4676-85c3-e0a0cd690ac9-dshm\") pod \"e2e-lifecycle-model-7b08359f-kserve-844f957dff-2jjnk\" (UID: \"6b860675-4baa-4676-85c3-e0a0cd690ac9\") " pod="ai-tenant-e2e-mt-7b08359f/e2e-lifecycle-model-7b08359f-kserve-844f957dff-2jjnk" Jul 09 16:44:41.017757 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:44:41.017738 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b860675-4baa-4676-85c3-e0a0cd690ac9-tls-certs\") pod \"e2e-lifecycle-model-7b08359f-kserve-844f957dff-2jjnk\" (UID: \"6b860675-4baa-4676-85c3-e0a0cd690ac9\") " pod="ai-tenant-e2e-mt-7b08359f/e2e-lifecycle-model-7b08359f-kserve-844f957dff-2jjnk" Jul 09 16:44:41.023640 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:44:41.023619 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mt6l\" (UniqueName: \"kubernetes.io/projected/6b860675-4baa-4676-85c3-e0a0cd690ac9-kube-api-access-8mt6l\") pod \"e2e-lifecycle-model-7b08359f-kserve-844f957dff-2jjnk\" (UID: \"6b860675-4baa-4676-85c3-e0a0cd690ac9\") " pod="ai-tenant-e2e-mt-7b08359f/e2e-lifecycle-model-7b08359f-kserve-844f957dff-2jjnk" Jul 09 16:44:41.187218 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:44:41.187124 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7b08359f/e2e-lifecycle-model-7b08359f-kserve-844f957dff-2jjnk" Jul 09 16:44:41.336665 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:44:41.336643 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-7b08359f/e2e-lifecycle-model-7b08359f-kserve-844f957dff-2jjnk"] Jul 09 16:44:41.339293 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:44:41.339262 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6b860675_4baa_4676_85c3_e0a0cd690ac9.slice/crio-f277380f607d5041f7f16cf5111a37e2768f532f3b0f933f08ad50f1c0655920 WatchSource:0}: Error finding container f277380f607d5041f7f16cf5111a37e2768f532f3b0f933f08ad50f1c0655920: Status 404 returned error can't find the container with id f277380f607d5041f7f16cf5111a37e2768f532f3b0f933f08ad50f1c0655920 Jul 09 16:44:41.929178 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:44:41.929136 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7b08359f/e2e-lifecycle-model-7b08359f-kserve-844f957dff-2jjnk" event={"ID":"6b860675-4baa-4676-85c3-e0a0cd690ac9","Type":"ContainerStarted","Data":"02da63610c37053c760235ec585ea71bf86440159d1a07ff015c5520e2d28eba"} Jul 09 16:44:41.929178 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:44:41.929185 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7b08359f/e2e-lifecycle-model-7b08359f-kserve-844f957dff-2jjnk" event={"ID":"6b860675-4baa-4676-85c3-e0a0cd690ac9","Type":"ContainerStarted","Data":"f277380f607d5041f7f16cf5111a37e2768f532f3b0f933f08ad50f1c0655920"} Jul 09 16:44:41.962716 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:44:41.962658 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-7b08359f/e2e-lifecycle-model-7b08359f-kserve-844f957dff-2jjnk" podStartSLOduration=1.962642565 podStartE2EDuration="1.962642565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:44:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:44:41.961733879 +0000 UTC m=+1767.154840257" watchObservedRunningTime="2026-07-09 16:44:41.962642565 +0000 UTC m=+1767.155748945" Jul 09 16:44:51.188112 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:44:51.188070 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-7b08359f/e2e-lifecycle-model-7b08359f-kserve-844f957dff-2jjnk" Jul 09 16:44:51.188608 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:44:51.188242 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-7b08359f/e2e-lifecycle-model-7b08359f-kserve-844f957dff-2jjnk" Jul 09 16:44:51.201008 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:44:51.200979 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-7b08359f/e2e-lifecycle-model-7b08359f-kserve-844f957dff-2jjnk" Jul 09 16:44:51.989453 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:44:51.989423 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-7b08359f/e2e-lifecycle-model-7b08359f-kserve-844f957dff-2jjnk" Jul 09 16:45:00.137273 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:45:00.137229 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-7b08359f-29726925-tsrjh"] Jul 09 16:45:00.140839 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:45:00.140818 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-7b08359f-29726925-tsrjh" Jul 09 16:45:00.151373 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:45:00.151346 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-7b08359f-29726925-tsrjh"] Jul 09 16:45:00.289197 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:45:00.289158 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x6c6\" (UniqueName: \"kubernetes.io/projected/c85fa5f6-bc39-48c3-97f8-c0126ae019f9-kube-api-access-7x6c6\") pod \"maas-api-key-cleanup-e2e-mt-7b08359f-29726925-tsrjh\" (UID: \"c85fa5f6-bc39-48c3-97f8-c0126ae019f9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-7b08359f-29726925-tsrjh" Jul 09 16:45:00.390528 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:45:00.390424 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x6c6\" (UniqueName: \"kubernetes.io/projected/c85fa5f6-bc39-48c3-97f8-c0126ae019f9-kube-api-access-7x6c6\") pod \"maas-api-key-cleanup-e2e-mt-7b08359f-29726925-tsrjh\" (UID: \"c85fa5f6-bc39-48c3-97f8-c0126ae019f9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-7b08359f-29726925-tsrjh" Jul 09 16:45:00.402066 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:45:00.402026 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x6c6\" (UniqueName: \"kubernetes.io/projected/c85fa5f6-bc39-48c3-97f8-c0126ae019f9-kube-api-access-7x6c6\") pod \"maas-api-key-cleanup-e2e-mt-7b08359f-29726925-tsrjh\" (UID: \"c85fa5f6-bc39-48c3-97f8-c0126ae019f9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-7b08359f-29726925-tsrjh" Jul 09 16:45:00.453947 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:45:00.453902 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-7b08359f-29726925-tsrjh" Jul 09 16:45:00.593726 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:45:00.593692 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-7b08359f-29726925-tsrjh"] Jul 09 16:45:00.595812 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:45:00.595783 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc85fa5f6_bc39_48c3_97f8_c0126ae019f9.slice/crio-34e0d3d59952ed0cff7244e59e98595b84db9af0f344607b7f17d0b8594010f1 WatchSource:0}: Error finding container 34e0d3d59952ed0cff7244e59e98595b84db9af0f344607b7f17d0b8594010f1: Status 404 returned error can't find the container with id 34e0d3d59952ed0cff7244e59e98595b84db9af0f344607b7f17d0b8594010f1 Jul 09 16:45:01.023790 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:45:01.023698 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-7b08359f-29726925-tsrjh" event={"ID":"c85fa5f6-bc39-48c3-97f8-c0126ae019f9","Type":"ContainerStarted","Data":"6864633339e7cd45128335e9fdb927b5c5909f2badab992d9e701d1d72f7eddd"} Jul 09 16:45:01.023790 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:45:01.023752 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-7b08359f-29726925-tsrjh" event={"ID":"c85fa5f6-bc39-48c3-97f8-c0126ae019f9","Type":"ContainerStarted","Data":"34e0d3d59952ed0cff7244e59e98595b84db9af0f344607b7f17d0b8594010f1"} Jul 09 16:45:01.040640 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:45:01.040581 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-7b08359f-29726925-tsrjh" podStartSLOduration=1.040558786 podStartE2EDuration="1.040558786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:45:01.040251823 +0000 UTC m=+1786.233358208" watchObservedRunningTime="2026-07-09 16:45:01.040558786 +0000 UTC m=+1786.233665168" Jul 09 16:45:08.339635 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:45:08.339541 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7b08359f-7b6d85656d-drr4q"] Jul 09 16:45:08.340130 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:45:08.339774 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b08359f-7b6d85656d-drr4q" podUID="0f6361d8-9c8f-4b8f-aa8d-2c1747227e83" containerName="maas-api" containerID="cri-o://a139ce0781bfd9173f4f3ffa438360a9eb3e13c42e0497d13920b06c2812362b" gracePeriod=30 Jul 09 16:45:08.375111 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:45:08.375074 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-7b08359f-29726925-tsrjh"] Jul 09 16:45:08.375315 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:45:08.375283 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-7b08359f-29726925-tsrjh" podUID="c85fa5f6-bc39-48c3-97f8-c0126ae019f9" containerName="cleanup" containerID="cri-o://6864633339e7cd45128335e9fdb927b5c5909f2badab992d9e701d1d72f7eddd" gracePeriod=30 Jul 09 16:45:08.599062 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:45:08.598992 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b08359f-7b6d85656d-drr4q" Jul 09 16:45:08.768833 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:45:08.768800 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f6361d8-9c8f-4b8f-aa8d-2c1747227e83-openshift-service-ca\") pod \"0f6361d8-9c8f-4b8f-aa8d-2c1747227e83\" (UID: \"0f6361d8-9c8f-4b8f-aa8d-2c1747227e83\") " Jul 09 16:45:08.769011 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:45:08.768936 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-645mm\" (UniqueName: \"kubernetes.io/projected/0f6361d8-9c8f-4b8f-aa8d-2c1747227e83-kube-api-access-645mm\") pod \"0f6361d8-9c8f-4b8f-aa8d-2c1747227e83\" (UID: \"0f6361d8-9c8f-4b8f-aa8d-2c1747227e83\") " Jul 09 16:45:08.769011 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:45:08.768981 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f6361d8-9c8f-4b8f-aa8d-2c1747227e83-maas-api-tls\") pod \"0f6361d8-9c8f-4b8f-aa8d-2c1747227e83\" (UID: \"0f6361d8-9c8f-4b8f-aa8d-2c1747227e83\") " Jul 09 16:45:08.769247 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:45:08.769226 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f6361d8-9c8f-4b8f-aa8d-2c1747227e83-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0f6361d8-9c8f-4b8f-aa8d-2c1747227e83" (UID: "0f6361d8-9c8f-4b8f-aa8d-2c1747227e83"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 16:45:08.771187 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:45:08.771161 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f6361d8-9c8f-4b8f-aa8d-2c1747227e83-kube-api-access-645mm" (OuterVolumeSpecName: "kube-api-access-645mm") pod "0f6361d8-9c8f-4b8f-aa8d-2c1747227e83" (UID: "0f6361d8-9c8f-4b8f-aa8d-2c1747227e83"). InnerVolumeSpecName "kube-api-access-645mm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 16:45:08.771258 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:45:08.771161 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f6361d8-9c8f-4b8f-aa8d-2c1747227e83-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0f6361d8-9c8f-4b8f-aa8d-2c1747227e83" (UID: "0f6361d8-9c8f-4b8f-aa8d-2c1747227e83"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 16:45:08.870117 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:45:08.870016 2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f6361d8-9c8f-4b8f-aa8d-2c1747227e83-openshift-service-ca\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:45:08.870117 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:45:08.870054 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-645mm\" (UniqueName: \"kubernetes.io/projected/0f6361d8-9c8f-4b8f-aa8d-2c1747227e83-kube-api-access-645mm\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:45:08.870117 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:45:08.870064 2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f6361d8-9c8f-4b8f-aa8d-2c1747227e83-maas-api-tls\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:45:09.065151 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:45:09.065113 2569 generic.go:356] "Generic (PLEG): container finished" podID="0f6361d8-9c8f-4b8f-aa8d-2c1747227e83" containerID="a139ce0781bfd9173f4f3ffa438360a9eb3e13c42e0497d13920b06c2812362b" exitCode=0 Jul 09 16:45:09.065358 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:45:09.065161 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b08359f-7b6d85656d-drr4q" Jul 09 16:45:09.065358 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:45:09.065196 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b08359f-7b6d85656d-drr4q" event={"ID":"0f6361d8-9c8f-4b8f-aa8d-2c1747227e83","Type":"ContainerDied","Data":"a139ce0781bfd9173f4f3ffa438360a9eb3e13c42e0497d13920b06c2812362b"} Jul 09 16:45:09.065358 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:45:09.065253 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7b08359f-7b6d85656d-drr4q" event={"ID":"0f6361d8-9c8f-4b8f-aa8d-2c1747227e83","Type":"ContainerDied","Data":"13cc74b4512c13205b2c41e9190fa1780f00c8d7dd0b31841bfdff4fbd4323d8"} Jul 09 16:45:09.065358 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:45:09.065277 2569 scope.go:117] "RemoveContainer" containerID="a139ce0781bfd9173f4f3ffa438360a9eb3e13c42e0497d13920b06c2812362b" Jul 09 16:45:09.074736 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:45:09.074719 2569 scope.go:117] "RemoveContainer" containerID="a139ce0781bfd9173f4f3ffa438360a9eb3e13c42e0497d13920b06c2812362b" Jul 09 16:45:09.075090 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:45:09.075071 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a139ce0781bfd9173f4f3ffa438360a9eb3e13c42e0497d13920b06c2812362b\": container with ID starting with a139ce0781bfd9173f4f3ffa438360a9eb3e13c42e0497d13920b06c2812362b not found: ID does not exist" containerID="a139ce0781bfd9173f4f3ffa438360a9eb3e13c42e0497d13920b06c2812362b" Jul 09 16:45:09.075156 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:45:09.075100 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a139ce0781bfd9173f4f3ffa438360a9eb3e13c42e0497d13920b06c2812362b"} err="failed to get container status \"a139ce0781bfd9173f4f3ffa438360a9eb3e13c42e0497d13920b06c2812362b\": rpc error: code = NotFound desc = could not find container \"a139ce0781bfd9173f4f3ffa438360a9eb3e13c42e0497d13920b06c2812362b\": container with ID starting with a139ce0781bfd9173f4f3ffa438360a9eb3e13c42e0497d13920b06c2812362b not found: ID does not exist" Jul 09 16:45:09.088090 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:45:09.088059 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7b08359f-7b6d85656d-drr4q"] Jul 09 16:45:09.092264 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:45:09.092239 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7b08359f-7b6d85656d-drr4q"] Jul 09 16:45:09.390292 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:45:09.390247 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f6361d8-9c8f-4b8f-aa8d-2c1747227e83" path="/var/lib/kubelet/pods/0f6361d8-9c8f-4b8f-aa8d-2c1747227e83/volumes" Jul 09 16:45:15.387271 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:45:15.387236 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-nzrk9_a284ea64-610c-40f7-837b-fa0975c0a290/console-operator/2.log" Jul 09 16:45:15.394086 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:45:15.394062 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-nzrk9_a284ea64-610c-40f7-837b-fa0975c0a290/console-operator/2.log" Jul 09 16:45:15.395333 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:45:15.395310 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-49sqd_17313664-7a0f-472b-a597-39f8c5412bc3/ovn-acl-logging/0.log" Jul 09 16:45:15.401288 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:45:15.401269 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-49sqd_17313664-7a0f-472b-a597-39f8c5412bc3/ovn-acl-logging/0.log" Jul 09 16:45:19.036795 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:45:19.036753 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-7b08359f/e2e-lifecycle-model-7b08359f-kserve-844f957dff-2jjnk"] Jul 09 16:45:34.058501 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:45:34.058453 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-7b08359f/e2e-lifecycle-model-7b08359f-kserve-844f957dff-2jjnk" podUID="6b860675-4baa-4676-85c3-e0a0cd690ac9" containerName="main" containerID="cri-o://02da63610c37053c760235ec585ea71bf86440159d1a07ff015c5520e2d28eba" gracePeriod=45 Jul 09 16:45:34.196302 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:45:34.196261 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7b08359f/e2e-lifecycle-model-7b08359f-kserve-844f957dff-2jjnk" event={"ID":"6b860675-4baa-4676-85c3-e0a0cd690ac9","Type":"ContainerDied","Data":"02da63610c37053c760235ec585ea71bf86440159d1a07ff015c5520e2d28eba"} Jul 09 16:45:34.196466 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:45:34.196256 2569 generic.go:356] "Generic (PLEG): container finished" podID="6b860675-4baa-4676-85c3-e0a0cd690ac9" containerID="02da63610c37053c760235ec585ea71bf86440159d1a07ff015c5520e2d28eba" exitCode=0 Jul 09 16:45:34.320246 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:45:34.320183 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7b08359f/e2e-lifecycle-model-7b08359f-kserve-844f957dff-2jjnk" Jul 09 16:45:34.390938 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:45:34.390905 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8mt6l\" (UniqueName: \"kubernetes.io/projected/6b860675-4baa-4676-85c3-e0a0cd690ac9-kube-api-access-8mt6l\") pod \"6b860675-4baa-4676-85c3-e0a0cd690ac9\" (UID: \"6b860675-4baa-4676-85c3-e0a0cd690ac9\") " Jul 09 16:45:34.391126 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:45:34.390964 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b860675-4baa-4676-85c3-e0a0cd690ac9-home\") pod \"6b860675-4baa-4676-85c3-e0a0cd690ac9\" (UID: \"6b860675-4baa-4676-85c3-e0a0cd690ac9\") " Jul 09 16:45:34.391126 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:45:34.391003 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b860675-4baa-4676-85c3-e0a0cd690ac9-dshm\") pod \"6b860675-4baa-4676-85c3-e0a0cd690ac9\" (UID: \"6b860675-4baa-4676-85c3-e0a0cd690ac9\") " Jul 09 16:45:34.391126 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:45:34.391033 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b860675-4baa-4676-85c3-e0a0cd690ac9-model-cache\") pod \"6b860675-4baa-4676-85c3-e0a0cd690ac9\" (UID: \"6b860675-4baa-4676-85c3-e0a0cd690ac9\") " Jul 09 16:45:34.391126 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:45:34.391061 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b860675-4baa-4676-85c3-e0a0cd690ac9-tmp-dir\") pod \"6b860675-4baa-4676-85c3-e0a0cd690ac9\" (UID: \"6b860675-4baa-4676-85c3-e0a0cd690ac9\") " Jul 09 16:45:34.391339 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:45:34.391128 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b860675-4baa-4676-85c3-e0a0cd690ac9-tls-certs\") pod \"6b860675-4baa-4676-85c3-e0a0cd690ac9\" (UID: \"6b860675-4baa-4676-85c3-e0a0cd690ac9\") " Jul 09 16:45:34.391339 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:45:34.391312 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b860675-4baa-4676-85c3-e0a0cd690ac9-model-cache" (OuterVolumeSpecName: "model-cache") pod "6b860675-4baa-4676-85c3-e0a0cd690ac9" (UID: "6b860675-4baa-4676-85c3-e0a0cd690ac9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:45:34.391444 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:45:34.391332 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b860675-4baa-4676-85c3-e0a0cd690ac9-home" (OuterVolumeSpecName: "home") pod "6b860675-4baa-4676-85c3-e0a0cd690ac9" (UID: "6b860675-4baa-4676-85c3-e0a0cd690ac9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:45:34.391444 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:45:34.391351 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b860675-4baa-4676-85c3-e0a0cd690ac9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6b860675-4baa-4676-85c3-e0a0cd690ac9" (UID: "6b860675-4baa-4676-85c3-e0a0cd690ac9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:45:34.391546 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:45:34.391472 2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b860675-4baa-4676-85c3-e0a0cd690ac9-model-cache\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:45:34.391546 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:45:34.391485 2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b860675-4baa-4676-85c3-e0a0cd690ac9-tmp-dir\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:45:34.391546 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:45:34.391493 2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b860675-4baa-4676-85c3-e0a0cd690ac9-home\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:45:34.393266 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:45:34.393237 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b860675-4baa-4676-85c3-e0a0cd690ac9-dshm" (OuterVolumeSpecName: "dshm") pod "6b860675-4baa-4676-85c3-e0a0cd690ac9" (UID: "6b860675-4baa-4676-85c3-e0a0cd690ac9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:45:34.393396 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:45:34.393239 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b860675-4baa-4676-85c3-e0a0cd690ac9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6b860675-4baa-4676-85c3-e0a0cd690ac9" (UID: "6b860675-4baa-4676-85c3-e0a0cd690ac9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 16:45:34.393396 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:45:34.393297 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b860675-4baa-4676-85c3-e0a0cd690ac9-kube-api-access-8mt6l" (OuterVolumeSpecName: "kube-api-access-8mt6l") pod "6b860675-4baa-4676-85c3-e0a0cd690ac9" (UID: "6b860675-4baa-4676-85c3-e0a0cd690ac9"). InnerVolumeSpecName "kube-api-access-8mt6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 16:45:34.492577 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:45:34.492542 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mt6l\" (UniqueName: \"kubernetes.io/projected/6b860675-4baa-4676-85c3-e0a0cd690ac9-kube-api-access-8mt6l\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:45:34.492577 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:45:34.492571 2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b860675-4baa-4676-85c3-e0a0cd690ac9-dshm\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:45:34.492577 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:45:34.492582 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b860675-4baa-4676-85c3-e0a0cd690ac9-tls-certs\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:45:35.202930 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:45:35.202899 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7b08359f/e2e-lifecycle-model-7b08359f-kserve-844f957dff-2jjnk" Jul 09 16:45:35.203375 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:45:35.202904 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7b08359f/e2e-lifecycle-model-7b08359f-kserve-844f957dff-2jjnk" event={"ID":"6b860675-4baa-4676-85c3-e0a0cd690ac9","Type":"ContainerDied","Data":"f277380f607d5041f7f16cf5111a37e2768f532f3b0f933f08ad50f1c0655920"} Jul 09 16:45:35.203375 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:45:35.203024 2569 scope.go:117] "RemoveContainer" containerID="02da63610c37053c760235ec585ea71bf86440159d1a07ff015c5520e2d28eba" Jul 09 16:45:35.223464 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:45:35.223442 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-7b08359f/e2e-lifecycle-model-7b08359f-kserve-844f957dff-2jjnk"] Jul 09 16:45:35.226788 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:45:35.226763 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-7b08359f/e2e-lifecycle-model-7b08359f-kserve-844f957dff-2jjnk"] Jul 09 16:45:35.389026 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:45:35.388994 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b860675-4baa-4676-85c3-e0a0cd690ac9" path="/var/lib/kubelet/pods/6b860675-4baa-4676-85c3-e0a0cd690ac9/volumes" Jul 09 16:45:38.561007 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:45:38.560984 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-7b08359f-29726925-tsrjh" Jul 09 16:45:38.631322 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:45:38.631235 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7x6c6\" (UniqueName: \"kubernetes.io/projected/c85fa5f6-bc39-48c3-97f8-c0126ae019f9-kube-api-access-7x6c6\") pod \"c85fa5f6-bc39-48c3-97f8-c0126ae019f9\" (UID: \"c85fa5f6-bc39-48c3-97f8-c0126ae019f9\") " Jul 09 16:45:38.633331 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:45:38.633306 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c85fa5f6-bc39-48c3-97f8-c0126ae019f9-kube-api-access-7x6c6" (OuterVolumeSpecName: "kube-api-access-7x6c6") pod "c85fa5f6-bc39-48c3-97f8-c0126ae019f9" (UID: "c85fa5f6-bc39-48c3-97f8-c0126ae019f9"). InnerVolumeSpecName "kube-api-access-7x6c6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 16:45:38.732530 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:45:38.732494 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7x6c6\" (UniqueName: \"kubernetes.io/projected/c85fa5f6-bc39-48c3-97f8-c0126ae019f9-kube-api-access-7x6c6\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:45:39.228514 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:45:39.228472 2569 generic.go:356] "Generic (PLEG): container finished" podID="c85fa5f6-bc39-48c3-97f8-c0126ae019f9" containerID="6864633339e7cd45128335e9fdb927b5c5909f2badab992d9e701d1d72f7eddd" exitCode=137 Jul 09 16:45:39.228745 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:45:39.228528 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-7b08359f-29726925-tsrjh" Jul 09 16:45:39.228745 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:45:39.228550 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-7b08359f-29726925-tsrjh" event={"ID":"c85fa5f6-bc39-48c3-97f8-c0126ae019f9","Type":"ContainerDied","Data":"6864633339e7cd45128335e9fdb927b5c5909f2badab992d9e701d1d72f7eddd"} Jul 09 16:45:39.228745 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:45:39.228631 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-7b08359f-29726925-tsrjh" event={"ID":"c85fa5f6-bc39-48c3-97f8-c0126ae019f9","Type":"ContainerDied","Data":"34e0d3d59952ed0cff7244e59e98595b84db9af0f344607b7f17d0b8594010f1"} Jul 09 16:45:39.228745 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:45:39.228666 2569 scope.go:117] "RemoveContainer" containerID="6864633339e7cd45128335e9fdb927b5c5909f2badab992d9e701d1d72f7eddd" Jul 09 16:45:39.245930 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:45:39.245887 2569 scope.go:117] "RemoveContainer" containerID="6864633339e7cd45128335e9fdb927b5c5909f2badab992d9e701d1d72f7eddd" Jul 09 16:45:39.246349 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:45:39.246327 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6864633339e7cd45128335e9fdb927b5c5909f2badab992d9e701d1d72f7eddd\": container with ID starting with 6864633339e7cd45128335e9fdb927b5c5909f2badab992d9e701d1d72f7eddd not found: ID does not exist" containerID="6864633339e7cd45128335e9fdb927b5c5909f2badab992d9e701d1d72f7eddd" Jul 09 16:45:39.246407 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:45:39.246358 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6864633339e7cd45128335e9fdb927b5c5909f2badab992d9e701d1d72f7eddd"} err="failed to get container status \"6864633339e7cd45128335e9fdb927b5c5909f2badab992d9e701d1d72f7eddd\": rpc error: code = NotFound desc = could not find container \"6864633339e7cd45128335e9fdb927b5c5909f2badab992d9e701d1d72f7eddd\": container with ID starting with 6864633339e7cd45128335e9fdb927b5c5909f2badab992d9e701d1d72f7eddd not found: ID does not exist" Jul 09 16:45:39.256131 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:45:39.256101 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-7b08359f-29726925-tsrjh"] Jul 09 16:45:39.259991 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:45:39.259967 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-7b08359f-29726925-tsrjh"] Jul 09 16:45:39.389226 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:45:39.389193 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c85fa5f6-bc39-48c3-97f8-c0126ae019f9" path="/var/lib/kubelet/pods/c85fa5f6-bc39-48c3-97f8-c0126ae019f9/volumes" Jul 09 16:47:00.391440 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:00.391352 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6c0f57-c8dd8f45-c7gb5"] Jul 09 16:47:00.391994 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:00.391971 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f6361d8-9c8f-4b8f-aa8d-2c1747227e83" containerName="maas-api" Jul 09 16:47:00.392088 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:00.391998 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f6361d8-9c8f-4b8f-aa8d-2c1747227e83" containerName="maas-api" Jul 09 16:47:00.392088 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:00.392017 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c85fa5f6-bc39-48c3-97f8-c0126ae019f9" containerName="cleanup" Jul 09 16:47:00.392088 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:00.392027 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c85fa5f6-bc39-48c3-97f8-c0126ae019f9" containerName="cleanup" Jul 09 16:47:00.392088 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:00.392047 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b860675-4baa-4676-85c3-e0a0cd690ac9" containerName="main" Jul 09 16:47:00.392088 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:00.392056 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b860675-4baa-4676-85c3-e0a0cd690ac9" containerName="main" Jul 09 16:47:00.392265 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:00.392169 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="c85fa5f6-bc39-48c3-97f8-c0126ae019f9" containerName="cleanup" Jul 09 16:47:00.392265 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:00.392186 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b860675-4baa-4676-85c3-e0a0cd690ac9" containerName="main" Jul 09 16:47:00.392265 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:00.392197 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f6361d8-9c8f-4b8f-aa8d-2c1747227e83" containerName="maas-api" Jul 09 16:47:00.395864 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:00.395841 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6c0f57-c8dd8f45-c7gb5" Jul 09 16:47:00.398474 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:00.398450 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-6c0f57-serving-cert\"" Jul 09 16:47:00.411242 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:00.411216 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6c0f57-c8dd8f45-c7gb5"] Jul 09 16:47:00.546450 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:00.546418 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0d03792-498a-4114-bda5-b4eb572570af-maas-api-tls\") pod \"maas-api-e2e-inf-a-6c0f57-c8dd8f45-c7gb5\" (UID: \"d0d03792-498a-4114-bda5-b4eb572570af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6c0f57-c8dd8f45-c7gb5" Jul 09 16:47:00.546635 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:00.546468 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kgrz\" (UniqueName: \"kubernetes.io/projected/d0d03792-498a-4114-bda5-b4eb572570af-kube-api-access-8kgrz\") pod \"maas-api-e2e-inf-a-6c0f57-c8dd8f45-c7gb5\" (UID: \"d0d03792-498a-4114-bda5-b4eb572570af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6c0f57-c8dd8f45-c7gb5" Jul 09 16:47:00.546635 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:00.546554 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0d03792-498a-4114-bda5-b4eb572570af-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6c0f57-c8dd8f45-c7gb5\" (UID: \"d0d03792-498a-4114-bda5-b4eb572570af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6c0f57-c8dd8f45-c7gb5" Jul 09 16:47:00.647099 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:00.647010 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0d03792-498a-4114-bda5-b4eb572570af-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6c0f57-c8dd8f45-c7gb5\" (UID: \"d0d03792-498a-4114-bda5-b4eb572570af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6c0f57-c8dd8f45-c7gb5" Jul 09 16:47:00.647258 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:00.647104 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0d03792-498a-4114-bda5-b4eb572570af-maas-api-tls\") pod \"maas-api-e2e-inf-a-6c0f57-c8dd8f45-c7gb5\" (UID: \"d0d03792-498a-4114-bda5-b4eb572570af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6c0f57-c8dd8f45-c7gb5" Jul 09 16:47:00.647258 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:00.647155 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kgrz\" (UniqueName: \"kubernetes.io/projected/d0d03792-498a-4114-bda5-b4eb572570af-kube-api-access-8kgrz\") pod \"maas-api-e2e-inf-a-6c0f57-c8dd8f45-c7gb5\" (UID: \"d0d03792-498a-4114-bda5-b4eb572570af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6c0f57-c8dd8f45-c7gb5" Jul 09 16:47:00.647357 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:47:00.647276 2569 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-6c0f57-serving-cert: secret "maas-api-e2e-inf-a-6c0f57-serving-cert" not found Jul 09 16:47:00.647398 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:47:00.647356 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0d03792-498a-4114-bda5-b4eb572570af-maas-api-tls podName:d0d03792-498a-4114-bda5-b4eb572570af nodeName:}" failed. No retries permitted until 2026-07-09 16:47:01.147333799 +0000 UTC m=+1906.340440161 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d0d03792-498a-4114-bda5-b4eb572570af-maas-api-tls") pod "maas-api-e2e-inf-a-6c0f57-c8dd8f45-c7gb5" (UID: "d0d03792-498a-4114-bda5-b4eb572570af") : secret "maas-api-e2e-inf-a-6c0f57-serving-cert" not found Jul 09 16:47:00.647746 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:00.647726 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0d03792-498a-4114-bda5-b4eb572570af-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6c0f57-c8dd8f45-c7gb5\" (UID: \"d0d03792-498a-4114-bda5-b4eb572570af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6c0f57-c8dd8f45-c7gb5" Jul 09 16:47:00.656629 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:00.656597 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kgrz\" (UniqueName: \"kubernetes.io/projected/d0d03792-498a-4114-bda5-b4eb572570af-kube-api-access-8kgrz\") pod \"maas-api-e2e-inf-a-6c0f57-c8dd8f45-c7gb5\" (UID: \"d0d03792-498a-4114-bda5-b4eb572570af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6c0f57-c8dd8f45-c7gb5" Jul 09 16:47:01.153166 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:01.153120 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0d03792-498a-4114-bda5-b4eb572570af-maas-api-tls\") pod \"maas-api-e2e-inf-a-6c0f57-c8dd8f45-c7gb5\" (UID: \"d0d03792-498a-4114-bda5-b4eb572570af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6c0f57-c8dd8f45-c7gb5" Jul 09 16:47:01.155600 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:01.155577 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0d03792-498a-4114-bda5-b4eb572570af-maas-api-tls\") pod \"maas-api-e2e-inf-a-6c0f57-c8dd8f45-c7gb5\" (UID: \"d0d03792-498a-4114-bda5-b4eb572570af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6c0f57-c8dd8f45-c7gb5" Jul 09 16:47:01.307975 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:01.307935 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6c0f57-c8dd8f45-c7gb5" Jul 09 16:47:01.458488 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:01.458458 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6c0f57-c8dd8f45-c7gb5"] Jul 09 16:47:01.461480 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:47:01.461452 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0d03792_498a_4114_bda5_b4eb572570af.slice/crio-adb804deba35dd37db4eb8ef221ea096be6d9681c0a068804c9d8cd3bdba8a6d WatchSource:0}: Error finding container adb804deba35dd37db4eb8ef221ea096be6d9681c0a068804c9d8cd3bdba8a6d: Status 404 returned error can't find the container with id adb804deba35dd37db4eb8ef221ea096be6d9681c0a068804c9d8cd3bdba8a6d Jul 09 16:47:01.462851 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:01.462832 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 16:47:01.626776 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:01.626739 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6c0f57-c8dd8f45-c7gb5" event={"ID":"d0d03792-498a-4114-bda5-b4eb572570af","Type":"ContainerStarted","Data":"adb804deba35dd37db4eb8ef221ea096be6d9681c0a068804c9d8cd3bdba8a6d"} Jul 09 16:47:02.633407 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:02.633373 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6c0f57-c8dd8f45-c7gb5" event={"ID":"d0d03792-498a-4114-bda5-b4eb572570af","Type":"ContainerStarted","Data":"1a050e1e15fc78e16d94ea3dd858ca2341f433ac00e50c2bd525d70d0915efd0"} Jul 09 16:47:02.633847 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:02.633500 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6c0f57-c8dd8f45-c7gb5" Jul 09 16:47:02.653152 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:02.653094 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6c0f57-c8dd8f45-c7gb5" podStartSLOduration=2.176409836 podStartE2EDuration="2.653074658s" totalImagesPullingTime="476.664822ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:47:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:47:02.651333351 +0000 UTC m=+1907.844439747" watchObservedRunningTime="2026-07-09 16:47:02.653074658 +0000 UTC m=+1907.846181038" Jul 09 16:47:08.450602 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:08.450567 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-aa22b3-67565ff7b8-v5p4r"] Jul 09 16:47:08.454446 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:08.454421 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aa22b3-67565ff7b8-v5p4r" Jul 09 16:47:08.457512 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:08.457481 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-aa22b3-serving-cert\"" Jul 09 16:47:08.465709 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:08.465679 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-aa22b3-67565ff7b8-v5p4r"] Jul 09 16:47:08.622991 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:08.622950 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvmvf\" (UniqueName: \"kubernetes.io/projected/40b30bda-77d8-4c7a-8ca9-97affc6e4181-kube-api-access-pvmvf\") pod \"maas-api-e2e-inf-b-aa22b3-67565ff7b8-v5p4r\" (UID: \"40b30bda-77d8-4c7a-8ca9-97affc6e4181\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aa22b3-67565ff7b8-v5p4r" Jul 09 16:47:08.622991 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:08.622991 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40b30bda-77d8-4c7a-8ca9-97affc6e4181-openshift-service-ca\") pod \"maas-api-e2e-inf-b-aa22b3-67565ff7b8-v5p4r\" (UID: \"40b30bda-77d8-4c7a-8ca9-97affc6e4181\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aa22b3-67565ff7b8-v5p4r" Jul 09 16:47:08.623241 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:08.623008 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40b30bda-77d8-4c7a-8ca9-97affc6e4181-maas-api-tls\") pod \"maas-api-e2e-inf-b-aa22b3-67565ff7b8-v5p4r\" (UID: \"40b30bda-77d8-4c7a-8ca9-97affc6e4181\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aa22b3-67565ff7b8-v5p4r" Jul 09 16:47:08.644033 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:08.644001 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6c0f57-c8dd8f45-c7gb5" Jul 09 16:47:08.723954 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:08.723845 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvmvf\" (UniqueName: \"kubernetes.io/projected/40b30bda-77d8-4c7a-8ca9-97affc6e4181-kube-api-access-pvmvf\") pod \"maas-api-e2e-inf-b-aa22b3-67565ff7b8-v5p4r\" (UID: \"40b30bda-77d8-4c7a-8ca9-97affc6e4181\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aa22b3-67565ff7b8-v5p4r" Jul 09 16:47:08.723954 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:08.723906 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40b30bda-77d8-4c7a-8ca9-97affc6e4181-openshift-service-ca\") pod \"maas-api-e2e-inf-b-aa22b3-67565ff7b8-v5p4r\" (UID: \"40b30bda-77d8-4c7a-8ca9-97affc6e4181\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aa22b3-67565ff7b8-v5p4r" Jul 09 16:47:08.723954 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:08.723934 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40b30bda-77d8-4c7a-8ca9-97affc6e4181-maas-api-tls\") pod \"maas-api-e2e-inf-b-aa22b3-67565ff7b8-v5p4r\" (UID: \"40b30bda-77d8-4c7a-8ca9-97affc6e4181\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aa22b3-67565ff7b8-v5p4r" Jul 09 16:47:08.724230 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:47:08.724077 2569 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-aa22b3-serving-cert: secret "maas-api-e2e-inf-b-aa22b3-serving-cert" not found Jul 09 16:47:08.724230 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:47:08.724157 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40b30bda-77d8-4c7a-8ca9-97affc6e4181-maas-api-tls podName:40b30bda-77d8-4c7a-8ca9-97affc6e4181 nodeName:}" failed. No retries permitted until 2026-07-09 16:47:09.22413447 +0000 UTC m=+1914.417240831 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/40b30bda-77d8-4c7a-8ca9-97affc6e4181-maas-api-tls") pod "maas-api-e2e-inf-b-aa22b3-67565ff7b8-v5p4r" (UID: "40b30bda-77d8-4c7a-8ca9-97affc6e4181") : secret "maas-api-e2e-inf-b-aa22b3-serving-cert" not found Jul 09 16:47:08.724653 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:08.724631 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40b30bda-77d8-4c7a-8ca9-97affc6e4181-openshift-service-ca\") pod \"maas-api-e2e-inf-b-aa22b3-67565ff7b8-v5p4r\" (UID: \"40b30bda-77d8-4c7a-8ca9-97affc6e4181\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aa22b3-67565ff7b8-v5p4r" Jul 09 16:47:08.733358 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:08.733327 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvmvf\" (UniqueName: \"kubernetes.io/projected/40b30bda-77d8-4c7a-8ca9-97affc6e4181-kube-api-access-pvmvf\") pod \"maas-api-e2e-inf-b-aa22b3-67565ff7b8-v5p4r\" (UID: \"40b30bda-77d8-4c7a-8ca9-97affc6e4181\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aa22b3-67565ff7b8-v5p4r" Jul 09 16:47:09.229091 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:09.229053 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40b30bda-77d8-4c7a-8ca9-97affc6e4181-maas-api-tls\") pod \"maas-api-e2e-inf-b-aa22b3-67565ff7b8-v5p4r\" (UID: \"40b30bda-77d8-4c7a-8ca9-97affc6e4181\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aa22b3-67565ff7b8-v5p4r" Jul 09 16:47:09.231576 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:09.231538 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40b30bda-77d8-4c7a-8ca9-97affc6e4181-maas-api-tls\") pod \"maas-api-e2e-inf-b-aa22b3-67565ff7b8-v5p4r\" (UID: \"40b30bda-77d8-4c7a-8ca9-97affc6e4181\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aa22b3-67565ff7b8-v5p4r" Jul 09 16:47:09.366822 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:09.366781 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aa22b3-67565ff7b8-v5p4r" Jul 09 16:47:09.512665 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:09.512607 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-aa22b3-67565ff7b8-v5p4r"] Jul 09 16:47:09.515223 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:47:09.515196 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40b30bda_77d8_4c7a_8ca9_97affc6e4181.slice/crio-bceb87afe7161a0f70b3531d3d2f5ce5bffd91776a099381ce685c9348f911d9 WatchSource:0}: Error finding container bceb87afe7161a0f70b3531d3d2f5ce5bffd91776a099381ce685c9348f911d9: Status 404 returned error can't find the container with id bceb87afe7161a0f70b3531d3d2f5ce5bffd91776a099381ce685c9348f911d9 Jul 09 16:47:09.669949 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:09.669907 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aa22b3-67565ff7b8-v5p4r" event={"ID":"40b30bda-77d8-4c7a-8ca9-97affc6e4181","Type":"ContainerStarted","Data":"bceb87afe7161a0f70b3531d3d2f5ce5bffd91776a099381ce685c9348f911d9"} Jul 09 16:47:10.677028 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:10.676986 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aa22b3-67565ff7b8-v5p4r" event={"ID":"40b30bda-77d8-4c7a-8ca9-97affc6e4181","Type":"ContainerStarted","Data":"94d0ca90383060b885497f45ee53e5a79c1cf4720e3400b8f236b9a2b981fa95"} Jul 09 16:47:10.677411 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:10.677114 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aa22b3-67565ff7b8-v5p4r" Jul 09 16:47:10.695956 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:10.695900 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aa22b3-67565ff7b8-v5p4r" podStartSLOduration=1.97214915 podStartE2EDuration="2.695868809s" totalImagesPullingTime="723.719659ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:47:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:47:10.693823801 +0000 UTC m=+1915.886930195" watchObservedRunningTime="2026-07-09 16:47:10.695868809 +0000 UTC m=+1915.888975189" Jul 09 16:47:16.687629 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:16.687595 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aa22b3-67565ff7b8-v5p4r" Jul 09 16:47:22.057934 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:22.057900 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-6c0f57/test-model-6c0f57-kserve-7d9f6f8475-bczh9"] Jul 09 16:47:22.061487 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:22.061467 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6c0f57/test-model-6c0f57-kserve-7d9f6f8475-bczh9" Jul 09 16:47:22.066280 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:22.066253 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-6c0f57\"/\"test-model-6c0f57-kserve-self-signed-certs\"" Jul 09 16:47:22.066280 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:22.066275 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-6c0f57\"/\"kube-root-ca.crt\"" Jul 09 16:47:22.066487 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:22.066320 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-6c0f57\"/\"default-dockercfg-zv9j8\"" Jul 09 16:47:22.066487 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:22.066320 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-6c0f57\"/\"openshift-service-ca.crt\"" Jul 09 16:47:22.074260 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:22.074234 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-6c0f57/test-model-6c0f57-kserve-7d9f6f8475-bczh9"] Jul 09 16:47:22.149374 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:22.149336 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e90de64e-44bd-4ed6-8254-084c6990147e-home\") pod \"test-model-6c0f57-kserve-7d9f6f8475-bczh9\" (UID: \"e90de64e-44bd-4ed6-8254-084c6990147e\") " pod="ai-tenant-e2e-inf-a-6c0f57/test-model-6c0f57-kserve-7d9f6f8475-bczh9" Jul 09 16:47:22.149594 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:22.149381 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e90de64e-44bd-4ed6-8254-084c6990147e-tls-certs\") pod \"test-model-6c0f57-kserve-7d9f6f8475-bczh9\" (UID: \"e90de64e-44bd-4ed6-8254-084c6990147e\") " pod="ai-tenant-e2e-inf-a-6c0f57/test-model-6c0f57-kserve-7d9f6f8475-bczh9" Jul 09 16:47:22.149594 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:22.149405 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e90de64e-44bd-4ed6-8254-084c6990147e-dshm\") pod \"test-model-6c0f57-kserve-7d9f6f8475-bczh9\" (UID: \"e90de64e-44bd-4ed6-8254-084c6990147e\") " pod="ai-tenant-e2e-inf-a-6c0f57/test-model-6c0f57-kserve-7d9f6f8475-bczh9" Jul 09 16:47:22.149594 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:22.149526 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e90de64e-44bd-4ed6-8254-084c6990147e-model-cache\") pod \"test-model-6c0f57-kserve-7d9f6f8475-bczh9\" (UID: \"e90de64e-44bd-4ed6-8254-084c6990147e\") " pod="ai-tenant-e2e-inf-a-6c0f57/test-model-6c0f57-kserve-7d9f6f8475-bczh9" Jul 09 16:47:22.149594 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:22.149562 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e90de64e-44bd-4ed6-8254-084c6990147e-tmp-dir\") pod \"test-model-6c0f57-kserve-7d9f6f8475-bczh9\" (UID: \"e90de64e-44bd-4ed6-8254-084c6990147e\") " pod="ai-tenant-e2e-inf-a-6c0f57/test-model-6c0f57-kserve-7d9f6f8475-bczh9" Jul 09 16:47:22.149783 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:22.149647 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9mj4\" (UniqueName: \"kubernetes.io/projected/e90de64e-44bd-4ed6-8254-084c6990147e-kube-api-access-b9mj4\") pod \"test-model-6c0f57-kserve-7d9f6f8475-bczh9\" (UID: \"e90de64e-44bd-4ed6-8254-084c6990147e\") " pod="ai-tenant-e2e-inf-a-6c0f57/test-model-6c0f57-kserve-7d9f6f8475-bczh9" Jul 09 16:47:22.251129 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:22.251090 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9mj4\" (UniqueName: \"kubernetes.io/projected/e90de64e-44bd-4ed6-8254-084c6990147e-kube-api-access-b9mj4\") pod \"test-model-6c0f57-kserve-7d9f6f8475-bczh9\" (UID: \"e90de64e-44bd-4ed6-8254-084c6990147e\") " pod="ai-tenant-e2e-inf-a-6c0f57/test-model-6c0f57-kserve-7d9f6f8475-bczh9" Jul 09 16:47:22.251337 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:22.251144 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e90de64e-44bd-4ed6-8254-084c6990147e-home\") pod \"test-model-6c0f57-kserve-7d9f6f8475-bczh9\" (UID: \"e90de64e-44bd-4ed6-8254-084c6990147e\") " pod="ai-tenant-e2e-inf-a-6c0f57/test-model-6c0f57-kserve-7d9f6f8475-bczh9" Jul 09 16:47:22.251337 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:22.251167 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e90de64e-44bd-4ed6-8254-084c6990147e-tls-certs\") pod \"test-model-6c0f57-kserve-7d9f6f8475-bczh9\" (UID: \"e90de64e-44bd-4ed6-8254-084c6990147e\") " pod="ai-tenant-e2e-inf-a-6c0f57/test-model-6c0f57-kserve-7d9f6f8475-bczh9" Jul 09 16:47:22.251337 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:22.251185 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e90de64e-44bd-4ed6-8254-084c6990147e-dshm\") pod \"test-model-6c0f57-kserve-7d9f6f8475-bczh9\" (UID: \"e90de64e-44bd-4ed6-8254-084c6990147e\") " pod="ai-tenant-e2e-inf-a-6c0f57/test-model-6c0f57-kserve-7d9f6f8475-bczh9" Jul 09 16:47:22.251337 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:22.251230 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e90de64e-44bd-4ed6-8254-084c6990147e-model-cache\") pod \"test-model-6c0f57-kserve-7d9f6f8475-bczh9\" (UID: \"e90de64e-44bd-4ed6-8254-084c6990147e\") " pod="ai-tenant-e2e-inf-a-6c0f57/test-model-6c0f57-kserve-7d9f6f8475-bczh9" Jul 09 16:47:22.251337 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:22.251247 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e90de64e-44bd-4ed6-8254-084c6990147e-tmp-dir\") pod \"test-model-6c0f57-kserve-7d9f6f8475-bczh9\" (UID: \"e90de64e-44bd-4ed6-8254-084c6990147e\") " pod="ai-tenant-e2e-inf-a-6c0f57/test-model-6c0f57-kserve-7d9f6f8475-bczh9" Jul 09 16:47:22.251641 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:22.251600 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e90de64e-44bd-4ed6-8254-084c6990147e-home\") pod \"test-model-6c0f57-kserve-7d9f6f8475-bczh9\" (UID: \"e90de64e-44bd-4ed6-8254-084c6990147e\") " pod="ai-tenant-e2e-inf-a-6c0f57/test-model-6c0f57-kserve-7d9f6f8475-bczh9" Jul 09 16:47:22.251697 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:22.251647 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e90de64e-44bd-4ed6-8254-084c6990147e-model-cache\") pod \"test-model-6c0f57-kserve-7d9f6f8475-bczh9\" (UID: \"e90de64e-44bd-4ed6-8254-084c6990147e\") " pod="ai-tenant-e2e-inf-a-6c0f57/test-model-6c0f57-kserve-7d9f6f8475-bczh9" Jul 09 16:47:22.251760 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:22.251748 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e90de64e-44bd-4ed6-8254-084c6990147e-tmp-dir\") pod \"test-model-6c0f57-kserve-7d9f6f8475-bczh9\" (UID: \"e90de64e-44bd-4ed6-8254-084c6990147e\") " pod="ai-tenant-e2e-inf-a-6c0f57/test-model-6c0f57-kserve-7d9f6f8475-bczh9" Jul 09 16:47:22.253632 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:22.253600 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e90de64e-44bd-4ed6-8254-084c6990147e-dshm\") pod \"test-model-6c0f57-kserve-7d9f6f8475-bczh9\" (UID: \"e90de64e-44bd-4ed6-8254-084c6990147e\") " pod="ai-tenant-e2e-inf-a-6c0f57/test-model-6c0f57-kserve-7d9f6f8475-bczh9" Jul 09 16:47:22.253862 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:22.253845 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e90de64e-44bd-4ed6-8254-084c6990147e-tls-certs\") pod \"test-model-6c0f57-kserve-7d9f6f8475-bczh9\" (UID: \"e90de64e-44bd-4ed6-8254-084c6990147e\") " pod="ai-tenant-e2e-inf-a-6c0f57/test-model-6c0f57-kserve-7d9f6f8475-bczh9" Jul 09 16:47:22.260687 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:22.260665 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9mj4\" (UniqueName: \"kubernetes.io/projected/e90de64e-44bd-4ed6-8254-084c6990147e-kube-api-access-b9mj4\") pod \"test-model-6c0f57-kserve-7d9f6f8475-bczh9\" (UID: \"e90de64e-44bd-4ed6-8254-084c6990147e\") " pod="ai-tenant-e2e-inf-a-6c0f57/test-model-6c0f57-kserve-7d9f6f8475-bczh9" Jul 09 16:47:22.370951 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:22.370912 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6c0f57/test-model-6c0f57-kserve-7d9f6f8475-bczh9" Jul 09 16:47:22.520943 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:22.520915 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-6c0f57/test-model-6c0f57-kserve-7d9f6f8475-bczh9"] Jul 09 16:47:22.523189 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:47:22.523161 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode90de64e_44bd_4ed6_8254_084c6990147e.slice/crio-ec5b6fad5ab352ded66b32b05ae6416149ed18ca79231f6b7184e3625b185c8c WatchSource:0}: Error finding container ec5b6fad5ab352ded66b32b05ae6416149ed18ca79231f6b7184e3625b185c8c: Status 404 returned error can't find the container with id ec5b6fad5ab352ded66b32b05ae6416149ed18ca79231f6b7184e3625b185c8c Jul 09 16:47:22.737009 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:22.736912 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6c0f57/test-model-6c0f57-kserve-7d9f6f8475-bczh9" event={"ID":"e90de64e-44bd-4ed6-8254-084c6990147e","Type":"ContainerStarted","Data":"4a4213a8ccaf684e521c36472d0b7d62effcf2a9867ef09ed5cf97ea8523e221"} Jul 09 16:47:22.737009 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:22.736952 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6c0f57/test-model-6c0f57-kserve-7d9f6f8475-bczh9" event={"ID":"e90de64e-44bd-4ed6-8254-084c6990147e","Type":"ContainerStarted","Data":"ec5b6fad5ab352ded66b32b05ae6416149ed18ca79231f6b7184e3625b185c8c"} Jul 09 16:47:22.767904 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:22.767818 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-6c0f57/test-model-6c0f57-kserve-7d9f6f8475-bczh9" podStartSLOduration=0.767797736 podStartE2EDuration="767.797736ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:47:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:47:22.767705823 +0000 UTC m=+1927.960812198" watchObservedRunningTime="2026-07-09 16:47:22.767797736 +0000 UTC m=+1927.960904118" Jul 09 16:47:32.371558 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:32.371516 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-6c0f57/test-model-6c0f57-kserve-7d9f6f8475-bczh9" Jul 09 16:47:32.372066 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:32.371580 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-6c0f57/test-model-6c0f57-kserve-7d9f6f8475-bczh9" Jul 09 16:47:32.384304 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:32.384271 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-6c0f57/test-model-6c0f57-kserve-7d9f6f8475-bczh9" Jul 09 16:47:32.804512 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:32.804416 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-6c0f57/test-model-6c0f57-kserve-7d9f6f8475-bczh9" Jul 09 16:47:52.486848 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:52.486813 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-aa22b3/test-model-aa22b3-kserve-7db58cc5bc-p8l98"] Jul 09 16:47:52.490401 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:52.490380 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-aa22b3/test-model-aa22b3-kserve-7db58cc5bc-p8l98" Jul 09 16:47:52.494850 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:52.494824 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-aa22b3\"/\"kube-root-ca.crt\"" Jul 09 16:47:52.495019 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:52.494823 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-aa22b3\"/\"default-dockercfg-9n8tb\"" Jul 09 16:47:52.495019 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:52.494823 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-aa22b3\"/\"test-model-aa22b3-kserve-self-signed-certs\"" Jul 09 16:47:52.495019 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:52.494823 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-aa22b3\"/\"openshift-service-ca.crt\"" Jul 09 16:47:52.501510 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:52.501487 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-aa22b3/test-model-aa22b3-kserve-7db58cc5bc-p8l98"] Jul 09 16:47:52.634256 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:52.634217 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d81fe02-4239-49ab-951e-c5b46e76d03b-home\") pod \"test-model-aa22b3-kserve-7db58cc5bc-p8l98\" (UID: \"6d81fe02-4239-49ab-951e-c5b46e76d03b\") " pod="ai-tenant-e2e-inf-b-aa22b3/test-model-aa22b3-kserve-7db58cc5bc-p8l98" Jul 09 16:47:52.634468 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:52.634279 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d81fe02-4239-49ab-951e-c5b46e76d03b-tmp-dir\") pod \"test-model-aa22b3-kserve-7db58cc5bc-p8l98\" (UID: \"6d81fe02-4239-49ab-951e-c5b46e76d03b\") " pod="ai-tenant-e2e-inf-b-aa22b3/test-model-aa22b3-kserve-7db58cc5bc-p8l98" Jul 09 16:47:52.634468 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:52.634316 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw6gh\" (UniqueName: \"kubernetes.io/projected/6d81fe02-4239-49ab-951e-c5b46e76d03b-kube-api-access-fw6gh\") pod \"test-model-aa22b3-kserve-7db58cc5bc-p8l98\" (UID: \"6d81fe02-4239-49ab-951e-c5b46e76d03b\") " pod="ai-tenant-e2e-inf-b-aa22b3/test-model-aa22b3-kserve-7db58cc5bc-p8l98" Jul 09 16:47:52.634468 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:52.634348 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d81fe02-4239-49ab-951e-c5b46e76d03b-dshm\") pod \"test-model-aa22b3-kserve-7db58cc5bc-p8l98\" (UID: \"6d81fe02-4239-49ab-951e-c5b46e76d03b\") " pod="ai-tenant-e2e-inf-b-aa22b3/test-model-aa22b3-kserve-7db58cc5bc-p8l98" Jul 09 16:47:52.634468 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:52.634369 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d81fe02-4239-49ab-951e-c5b46e76d03b-tls-certs\") pod \"test-model-aa22b3-kserve-7db58cc5bc-p8l98\" (UID: \"6d81fe02-4239-49ab-951e-c5b46e76d03b\") " pod="ai-tenant-e2e-inf-b-aa22b3/test-model-aa22b3-kserve-7db58cc5bc-p8l98" Jul 09 16:47:52.634468 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:52.634449 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d81fe02-4239-49ab-951e-c5b46e76d03b-model-cache\") pod \"test-model-aa22b3-kserve-7db58cc5bc-p8l98\" (UID: \"6d81fe02-4239-49ab-951e-c5b46e76d03b\") " pod="ai-tenant-e2e-inf-b-aa22b3/test-model-aa22b3-kserve-7db58cc5bc-p8l98" Jul 09 16:47:52.735175 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:52.735143 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d81fe02-4239-49ab-951e-c5b46e76d03b-home\") pod \"test-model-aa22b3-kserve-7db58cc5bc-p8l98\" (UID: \"6d81fe02-4239-49ab-951e-c5b46e76d03b\") " pod="ai-tenant-e2e-inf-b-aa22b3/test-model-aa22b3-kserve-7db58cc5bc-p8l98" Jul 09 16:47:52.735362 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:52.735198 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d81fe02-4239-49ab-951e-c5b46e76d03b-tmp-dir\") pod \"test-model-aa22b3-kserve-7db58cc5bc-p8l98\" (UID: \"6d81fe02-4239-49ab-951e-c5b46e76d03b\") " pod="ai-tenant-e2e-inf-b-aa22b3/test-model-aa22b3-kserve-7db58cc5bc-p8l98" Jul 09 16:47:52.735362 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:52.735234 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw6gh\" (UniqueName: \"kubernetes.io/projected/6d81fe02-4239-49ab-951e-c5b46e76d03b-kube-api-access-fw6gh\") pod \"test-model-aa22b3-kserve-7db58cc5bc-p8l98\" (UID: \"6d81fe02-4239-49ab-951e-c5b46e76d03b\") " pod="ai-tenant-e2e-inf-b-aa22b3/test-model-aa22b3-kserve-7db58cc5bc-p8l98" Jul 09 16:47:52.735362 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:52.735258 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d81fe02-4239-49ab-951e-c5b46e76d03b-dshm\") pod \"test-model-aa22b3-kserve-7db58cc5bc-p8l98\" (UID: \"6d81fe02-4239-49ab-951e-c5b46e76d03b\") " pod="ai-tenant-e2e-inf-b-aa22b3/test-model-aa22b3-kserve-7db58cc5bc-p8l98" Jul 09 16:47:52.735362 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:52.735273 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d81fe02-4239-49ab-951e-c5b46e76d03b-tls-certs\") pod \"test-model-aa22b3-kserve-7db58cc5bc-p8l98\" (UID: \"6d81fe02-4239-49ab-951e-c5b46e76d03b\") " pod="ai-tenant-e2e-inf-b-aa22b3/test-model-aa22b3-kserve-7db58cc5bc-p8l98" Jul 09 16:47:52.735362 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:52.735338 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d81fe02-4239-49ab-951e-c5b46e76d03b-model-cache\") pod \"test-model-aa22b3-kserve-7db58cc5bc-p8l98\" (UID: \"6d81fe02-4239-49ab-951e-c5b46e76d03b\") " pod="ai-tenant-e2e-inf-b-aa22b3/test-model-aa22b3-kserve-7db58cc5bc-p8l98" Jul 09 16:47:52.735642 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:52.735618 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d81fe02-4239-49ab-951e-c5b46e76d03b-home\") pod \"test-model-aa22b3-kserve-7db58cc5bc-p8l98\" (UID: \"6d81fe02-4239-49ab-951e-c5b46e76d03b\") " pod="ai-tenant-e2e-inf-b-aa22b3/test-model-aa22b3-kserve-7db58cc5bc-p8l98" Jul 09 16:47:52.735728 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:52.735701 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d81fe02-4239-49ab-951e-c5b46e76d03b-tmp-dir\") pod \"test-model-aa22b3-kserve-7db58cc5bc-p8l98\" (UID: \"6d81fe02-4239-49ab-951e-c5b46e76d03b\") " pod="ai-tenant-e2e-inf-b-aa22b3/test-model-aa22b3-kserve-7db58cc5bc-p8l98" Jul 09 16:47:52.735853 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:52.735734 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d81fe02-4239-49ab-951e-c5b46e76d03b-model-cache\") pod \"test-model-aa22b3-kserve-7db58cc5bc-p8l98\" (UID: \"6d81fe02-4239-49ab-951e-c5b46e76d03b\") " pod="ai-tenant-e2e-inf-b-aa22b3/test-model-aa22b3-kserve-7db58cc5bc-p8l98" Jul 09 16:47:52.737706 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:52.737641 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d81fe02-4239-49ab-951e-c5b46e76d03b-dshm\") pod \"test-model-aa22b3-kserve-7db58cc5bc-p8l98\" (UID: \"6d81fe02-4239-49ab-951e-c5b46e76d03b\") " pod="ai-tenant-e2e-inf-b-aa22b3/test-model-aa22b3-kserve-7db58cc5bc-p8l98" Jul 09 16:47:52.737816 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:52.737804 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d81fe02-4239-49ab-951e-c5b46e76d03b-tls-certs\") pod \"test-model-aa22b3-kserve-7db58cc5bc-p8l98\" (UID: \"6d81fe02-4239-49ab-951e-c5b46e76d03b\") " pod="ai-tenant-e2e-inf-b-aa22b3/test-model-aa22b3-kserve-7db58cc5bc-p8l98" Jul 09 16:47:52.743776 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:52.743754 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw6gh\" (UniqueName: \"kubernetes.io/projected/6d81fe02-4239-49ab-951e-c5b46e76d03b-kube-api-access-fw6gh\") pod \"test-model-aa22b3-kserve-7db58cc5bc-p8l98\" (UID: \"6d81fe02-4239-49ab-951e-c5b46e76d03b\") " pod="ai-tenant-e2e-inf-b-aa22b3/test-model-aa22b3-kserve-7db58cc5bc-p8l98" Jul 09 16:47:52.801938 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:52.801902 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-aa22b3/test-model-aa22b3-kserve-7db58cc5bc-p8l98" Jul 09 16:47:52.945926 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:52.945901 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-aa22b3/test-model-aa22b3-kserve-7db58cc5bc-p8l98"] Jul 09 16:47:52.947825 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:47:52.947784 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6d81fe02_4239_49ab_951e_c5b46e76d03b.slice/crio-412604b1450fe0ff6c44221d7558f4809f0f7041827fb20eeed9dfbff7cb6fdb WatchSource:0}: Error finding container 412604b1450fe0ff6c44221d7558f4809f0f7041827fb20eeed9dfbff7cb6fdb: Status 404 returned error can't find the container with id 412604b1450fe0ff6c44221d7558f4809f0f7041827fb20eeed9dfbff7cb6fdb Jul 09 16:47:53.897377 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:53.897340 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-aa22b3/test-model-aa22b3-kserve-7db58cc5bc-p8l98" event={"ID":"6d81fe02-4239-49ab-951e-c5b46e76d03b","Type":"ContainerStarted","Data":"6818682d8693077f719f3651d3fcc41acb26959ac7084c024ecff49cf89be5f2"} Jul 09 16:47:53.897377 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:53.897382 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-aa22b3/test-model-aa22b3-kserve-7db58cc5bc-p8l98" event={"ID":"6d81fe02-4239-49ab-951e-c5b46e76d03b","Type":"ContainerStarted","Data":"412604b1450fe0ff6c44221d7558f4809f0f7041827fb20eeed9dfbff7cb6fdb"} Jul 09 16:47:53.915667 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:47:53.915614 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-aa22b3/test-model-aa22b3-kserve-7db58cc5bc-p8l98" podStartSLOduration=1.915596195 podStartE2EDuration="1.915596195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:47:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:47:53.914394068 +0000 UTC m=+1959.107500448" watchObservedRunningTime="2026-07-09 16:47:53.915596195 +0000 UTC m=+1959.108702635" Jul 09 16:48:02.802275 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:48:02.802236 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-aa22b3/test-model-aa22b3-kserve-7db58cc5bc-p8l98" Jul 09 16:48:02.802772 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:48:02.802296 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-aa22b3/test-model-aa22b3-kserve-7db58cc5bc-p8l98" Jul 09 16:48:02.815391 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:48:02.815359 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-aa22b3/test-model-aa22b3-kserve-7db58cc5bc-p8l98" Jul 09 16:48:02.953198 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:48:02.953161 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-aa22b3/test-model-aa22b3-kserve-7db58cc5bc-p8l98" Jul 09 16:48:16.492887 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:48:16.492780 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-6c0f57/test-model-6c0f57-kserve-7d9f6f8475-bczh9"] Jul 09 16:48:17.441038 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:48:17.440999 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-aa22b3/test-model-aa22b3-kserve-7db58cc5bc-p8l98"] Jul 09 16:48:18.343936 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:48:18.343902 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6c0f57-c8dd8f45-c7gb5"] Jul 09 16:48:18.344406 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:48:18.344132 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6c0f57-c8dd8f45-c7gb5" podUID="d0d03792-498a-4114-bda5-b4eb572570af" containerName="maas-api" containerID="cri-o://1a050e1e15fc78e16d94ea3dd858ca2341f433ac00e50c2bd525d70d0915efd0" gracePeriod=30 Jul 09 16:48:18.600585 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:48:18.600513 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6c0f57-c8dd8f45-c7gb5" Jul 09 16:48:18.654299 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:48:18.654259 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0d03792-498a-4114-bda5-b4eb572570af-maas-api-tls\") pod \"d0d03792-498a-4114-bda5-b4eb572570af\" (UID: \"d0d03792-498a-4114-bda5-b4eb572570af\") " Jul 09 16:48:18.654470 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:48:18.654354 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8kgrz\" (UniqueName: \"kubernetes.io/projected/d0d03792-498a-4114-bda5-b4eb572570af-kube-api-access-8kgrz\") pod \"d0d03792-498a-4114-bda5-b4eb572570af\" (UID: \"d0d03792-498a-4114-bda5-b4eb572570af\") " Jul 09 16:48:18.654470 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:48:18.654390 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0d03792-498a-4114-bda5-b4eb572570af-openshift-service-ca\") pod \"d0d03792-498a-4114-bda5-b4eb572570af\" (UID: \"d0d03792-498a-4114-bda5-b4eb572570af\") " Jul 09 16:48:18.654797 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:48:18.654770 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0d03792-498a-4114-bda5-b4eb572570af-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d0d03792-498a-4114-bda5-b4eb572570af" (UID: "d0d03792-498a-4114-bda5-b4eb572570af"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 16:48:18.656471 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:48:18.656446 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0d03792-498a-4114-bda5-b4eb572570af-kube-api-access-8kgrz" (OuterVolumeSpecName: "kube-api-access-8kgrz") pod "d0d03792-498a-4114-bda5-b4eb572570af" (UID: "d0d03792-498a-4114-bda5-b4eb572570af"). InnerVolumeSpecName "kube-api-access-8kgrz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 16:48:18.656554 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:48:18.656520 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0d03792-498a-4114-bda5-b4eb572570af-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d0d03792-498a-4114-bda5-b4eb572570af" (UID: "d0d03792-498a-4114-bda5-b4eb572570af"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 16:48:18.755395 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:48:18.755359 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kgrz\" (UniqueName: \"kubernetes.io/projected/d0d03792-498a-4114-bda5-b4eb572570af-kube-api-access-8kgrz\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:48:18.755395 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:48:18.755388 2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0d03792-498a-4114-bda5-b4eb572570af-openshift-service-ca\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:48:18.755395 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:48:18.755399 2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0d03792-498a-4114-bda5-b4eb572570af-maas-api-tls\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:48:19.022508 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:48:19.022420 2569 generic.go:356] "Generic (PLEG): container finished" podID="d0d03792-498a-4114-bda5-b4eb572570af" containerID="1a050e1e15fc78e16d94ea3dd858ca2341f433ac00e50c2bd525d70d0915efd0" exitCode=0 Jul 09 16:48:19.022508 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:48:19.022483 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6c0f57-c8dd8f45-c7gb5" Jul 09 16:48:19.022689 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:48:19.022506 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6c0f57-c8dd8f45-c7gb5" event={"ID":"d0d03792-498a-4114-bda5-b4eb572570af","Type":"ContainerDied","Data":"1a050e1e15fc78e16d94ea3dd858ca2341f433ac00e50c2bd525d70d0915efd0"} Jul 09 16:48:19.022689 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:48:19.022594 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6c0f57-c8dd8f45-c7gb5" event={"ID":"d0d03792-498a-4114-bda5-b4eb572570af","Type":"ContainerDied","Data":"adb804deba35dd37db4eb8ef221ea096be6d9681c0a068804c9d8cd3bdba8a6d"} Jul 09 16:48:19.022689 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:48:19.022623 2569 scope.go:117] "RemoveContainer" containerID="1a050e1e15fc78e16d94ea3dd858ca2341f433ac00e50c2bd525d70d0915efd0" Jul 09 16:48:19.032400 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:48:19.032381 2569 scope.go:117] "RemoveContainer" containerID="1a050e1e15fc78e16d94ea3dd858ca2341f433ac00e50c2bd525d70d0915efd0" Jul 09 16:48:19.032674 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:48:19.032655 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a050e1e15fc78e16d94ea3dd858ca2341f433ac00e50c2bd525d70d0915efd0\": container with ID starting with 1a050e1e15fc78e16d94ea3dd858ca2341f433ac00e50c2bd525d70d0915efd0 not found: ID does not exist" containerID="1a050e1e15fc78e16d94ea3dd858ca2341f433ac00e50c2bd525d70d0915efd0" Jul 09 16:48:19.032751 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:48:19.032681 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a050e1e15fc78e16d94ea3dd858ca2341f433ac00e50c2bd525d70d0915efd0"} err="failed to get container status \"1a050e1e15fc78e16d94ea3dd858ca2341f433ac00e50c2bd525d70d0915efd0\": rpc error: code = NotFound desc = could not find container \"1a050e1e15fc78e16d94ea3dd858ca2341f433ac00e50c2bd525d70d0915efd0\": container with ID starting with 1a050e1e15fc78e16d94ea3dd858ca2341f433ac00e50c2bd525d70d0915efd0 not found: ID does not exist" Jul 09 16:48:19.045172 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:48:19.045106 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6c0f57-c8dd8f45-c7gb5"] Jul 09 16:48:19.047010 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:48:19.046989 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6c0f57-c8dd8f45-c7gb5"] Jul 09 16:48:19.389443 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:48:19.389406 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0d03792-498a-4114-bda5-b4eb572570af" path="/var/lib/kubelet/pods/d0d03792-498a-4114-bda5-b4eb572570af/volumes" Jul 09 16:48:31.513578 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:48:31.513539 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-6c0f57/test-model-6c0f57-kserve-7d9f6f8475-bczh9" podUID="e90de64e-44bd-4ed6-8254-084c6990147e" containerName="main" containerID="cri-o://4a4213a8ccaf684e521c36472d0b7d62effcf2a9867ef09ed5cf97ea8523e221" gracePeriod=45 Jul 09 16:48:31.766833 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:48:31.766773 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6c0f57/test-model-6c0f57-kserve-7d9f6f8475-bczh9" Jul 09 16:48:31.876611 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:48:31.876577 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e90de64e-44bd-4ed6-8254-084c6990147e-tmp-dir\") pod \"e90de64e-44bd-4ed6-8254-084c6990147e\" (UID: \"e90de64e-44bd-4ed6-8254-084c6990147e\") " Jul 09 16:48:31.876812 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:48:31.876651 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b9mj4\" (UniqueName: \"kubernetes.io/projected/e90de64e-44bd-4ed6-8254-084c6990147e-kube-api-access-b9mj4\") pod \"e90de64e-44bd-4ed6-8254-084c6990147e\" (UID: \"e90de64e-44bd-4ed6-8254-084c6990147e\") " Jul 09 16:48:31.876812 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:48:31.876683 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e90de64e-44bd-4ed6-8254-084c6990147e-dshm\") pod \"e90de64e-44bd-4ed6-8254-084c6990147e\" (UID: \"e90de64e-44bd-4ed6-8254-084c6990147e\") " Jul 09 16:48:31.876812 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:48:31.876706 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e90de64e-44bd-4ed6-8254-084c6990147e-model-cache\") pod \"e90de64e-44bd-4ed6-8254-084c6990147e\" (UID: \"e90de64e-44bd-4ed6-8254-084c6990147e\") " Jul 09 16:48:31.876812 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:48:31.876730 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e90de64e-44bd-4ed6-8254-084c6990147e-tls-certs\") pod \"e90de64e-44bd-4ed6-8254-084c6990147e\" (UID: \"e90de64e-44bd-4ed6-8254-084c6990147e\") " Jul 09 16:48:31.876812 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:48:31.876780 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e90de64e-44bd-4ed6-8254-084c6990147e-home\") pod \"e90de64e-44bd-4ed6-8254-084c6990147e\" (UID: \"e90de64e-44bd-4ed6-8254-084c6990147e\") " Jul 09 16:48:31.877116 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:48:31.876947 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e90de64e-44bd-4ed6-8254-084c6990147e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e90de64e-44bd-4ed6-8254-084c6990147e" (UID: "e90de64e-44bd-4ed6-8254-084c6990147e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:48:31.877116 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:48:31.877069 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e90de64e-44bd-4ed6-8254-084c6990147e-model-cache" (OuterVolumeSpecName: "model-cache") pod "e90de64e-44bd-4ed6-8254-084c6990147e" (UID: "e90de64e-44bd-4ed6-8254-084c6990147e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:48:31.877201 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:48:31.877165 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e90de64e-44bd-4ed6-8254-084c6990147e-home" (OuterVolumeSpecName: "home") pod "e90de64e-44bd-4ed6-8254-084c6990147e" (UID: "e90de64e-44bd-4ed6-8254-084c6990147e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:48:31.877317 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:48:31.877297 2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e90de64e-44bd-4ed6-8254-084c6990147e-home\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:48:31.877317 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:48:31.877317 2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e90de64e-44bd-4ed6-8254-084c6990147e-tmp-dir\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:48:31.877468 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:48:31.877326 2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e90de64e-44bd-4ed6-8254-084c6990147e-model-cache\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:48:31.879143 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:48:31.879107 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e90de64e-44bd-4ed6-8254-084c6990147e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e90de64e-44bd-4ed6-8254-084c6990147e" (UID: "e90de64e-44bd-4ed6-8254-084c6990147e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 16:48:31.879337 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:48:31.879316 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e90de64e-44bd-4ed6-8254-084c6990147e-dshm" (OuterVolumeSpecName: "dshm") pod "e90de64e-44bd-4ed6-8254-084c6990147e" (UID: "e90de64e-44bd-4ed6-8254-084c6990147e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:48:31.879400 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:48:31.879318 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e90de64e-44bd-4ed6-8254-084c6990147e-kube-api-access-b9mj4" (OuterVolumeSpecName: "kube-api-access-b9mj4") pod "e90de64e-44bd-4ed6-8254-084c6990147e" (UID: "e90de64e-44bd-4ed6-8254-084c6990147e"). InnerVolumeSpecName "kube-api-access-b9mj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 16:48:31.978523 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:48:31.978471 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9mj4\" (UniqueName: \"kubernetes.io/projected/e90de64e-44bd-4ed6-8254-084c6990147e-kube-api-access-b9mj4\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:48:31.978523 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:48:31.978517 2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e90de64e-44bd-4ed6-8254-084c6990147e-dshm\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:48:31.978523 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:48:31.978529 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e90de64e-44bd-4ed6-8254-084c6990147e-tls-certs\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:48:32.092591 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:48:32.092563 2569 generic.go:356] "Generic (PLEG): container finished" podID="e90de64e-44bd-4ed6-8254-084c6990147e" containerID="4a4213a8ccaf684e521c36472d0b7d62effcf2a9867ef09ed5cf97ea8523e221" exitCode=0 Jul 09 16:48:32.092792 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:48:32.092604 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6c0f57/test-model-6c0f57-kserve-7d9f6f8475-bczh9" Jul 09 16:48:32.092792 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:48:32.092656 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6c0f57/test-model-6c0f57-kserve-7d9f6f8475-bczh9" event={"ID":"e90de64e-44bd-4ed6-8254-084c6990147e","Type":"ContainerDied","Data":"4a4213a8ccaf684e521c36472d0b7d62effcf2a9867ef09ed5cf97ea8523e221"} Jul 09 16:48:32.092792 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:48:32.092704 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6c0f57/test-model-6c0f57-kserve-7d9f6f8475-bczh9" event={"ID":"e90de64e-44bd-4ed6-8254-084c6990147e","Type":"ContainerDied","Data":"ec5b6fad5ab352ded66b32b05ae6416149ed18ca79231f6b7184e3625b185c8c"} Jul 09 16:48:32.092792 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:48:32.092725 2569 scope.go:117] "RemoveContainer" containerID="4a4213a8ccaf684e521c36472d0b7d62effcf2a9867ef09ed5cf97ea8523e221" Jul 09 16:48:32.102297 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:48:32.102280 2569 scope.go:117] "RemoveContainer" containerID="4a4213a8ccaf684e521c36472d0b7d62effcf2a9867ef09ed5cf97ea8523e221" Jul 09 16:48:32.102549 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:48:32.102532 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a4213a8ccaf684e521c36472d0b7d62effcf2a9867ef09ed5cf97ea8523e221\": container with ID starting with 4a4213a8ccaf684e521c36472d0b7d62effcf2a9867ef09ed5cf97ea8523e221 not found: ID does not exist" containerID="4a4213a8ccaf684e521c36472d0b7d62effcf2a9867ef09ed5cf97ea8523e221" Jul 09 16:48:32.102590 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:48:32.102559 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a4213a8ccaf684e521c36472d0b7d62effcf2a9867ef09ed5cf97ea8523e221"} err="failed to get container status \"4a4213a8ccaf684e521c36472d0b7d62effcf2a9867ef09ed5cf97ea8523e221\": rpc error: code = NotFound desc = could not find container \"4a4213a8ccaf684e521c36472d0b7d62effcf2a9867ef09ed5cf97ea8523e221\": container with ID starting with 4a4213a8ccaf684e521c36472d0b7d62effcf2a9867ef09ed5cf97ea8523e221 not found: ID does not exist" Jul 09 16:48:32.113470 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:48:32.113446 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-6c0f57/test-model-6c0f57-kserve-7d9f6f8475-bczh9"] Jul 09 16:48:32.117507 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:48:32.117483 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-6c0f57/test-model-6c0f57-kserve-7d9f6f8475-bczh9"] Jul 09 16:48:32.462173 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:48:32.462071 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-aa22b3/test-model-aa22b3-kserve-7db58cc5bc-p8l98" podUID="6d81fe02-4239-49ab-951e-c5b46e76d03b" containerName="main" containerID="cri-o://6818682d8693077f719f3651d3fcc41acb26959ac7084c024ecff49cf89be5f2" gracePeriod=45 Jul 09 16:48:32.717290 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:48:32.717226 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-aa22b3/test-model-aa22b3-kserve-7db58cc5bc-p8l98" Jul 09 16:48:32.786104 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:48:32.786073 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d81fe02-4239-49ab-951e-c5b46e76d03b-model-cache\") pod \"6d81fe02-4239-49ab-951e-c5b46e76d03b\" (UID: \"6d81fe02-4239-49ab-951e-c5b46e76d03b\") " Jul 09 16:48:32.786266 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:48:32.786126 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d81fe02-4239-49ab-951e-c5b46e76d03b-home\") pod \"6d81fe02-4239-49ab-951e-c5b46e76d03b\" (UID: \"6d81fe02-4239-49ab-951e-c5b46e76d03b\") " Jul 09 16:48:32.786266 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:48:32.786164 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fw6gh\" (UniqueName: \"kubernetes.io/projected/6d81fe02-4239-49ab-951e-c5b46e76d03b-kube-api-access-fw6gh\") pod \"6d81fe02-4239-49ab-951e-c5b46e76d03b\" (UID: \"6d81fe02-4239-49ab-951e-c5b46e76d03b\") " Jul 09 16:48:32.786266 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:48:32.786241 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d81fe02-4239-49ab-951e-c5b46e76d03b-tmp-dir\") pod \"6d81fe02-4239-49ab-951e-c5b46e76d03b\" (UID: \"6d81fe02-4239-49ab-951e-c5b46e76d03b\") " Jul 09 16:48:32.786407 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:48:32.786279 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d81fe02-4239-49ab-951e-c5b46e76d03b-tls-certs\") pod \"6d81fe02-4239-49ab-951e-c5b46e76d03b\" (UID: \"6d81fe02-4239-49ab-951e-c5b46e76d03b\") " Jul 09 16:48:32.786407 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:48:32.786314 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d81fe02-4239-49ab-951e-c5b46e76d03b-dshm\") pod \"6d81fe02-4239-49ab-951e-c5b46e76d03b\" (UID: \"6d81fe02-4239-49ab-951e-c5b46e76d03b\") " Jul 09 16:48:32.786407 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:48:32.786361 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d81fe02-4239-49ab-951e-c5b46e76d03b-model-cache" (OuterVolumeSpecName: "model-cache") pod "6d81fe02-4239-49ab-951e-c5b46e76d03b" (UID: "6d81fe02-4239-49ab-951e-c5b46e76d03b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:48:32.786565 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:48:32.786442 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d81fe02-4239-49ab-951e-c5b46e76d03b-home" (OuterVolumeSpecName: "home") pod "6d81fe02-4239-49ab-951e-c5b46e76d03b" (UID: "6d81fe02-4239-49ab-951e-c5b46e76d03b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:48:32.786565 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:48:32.786556 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d81fe02-4239-49ab-951e-c5b46e76d03b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6d81fe02-4239-49ab-951e-c5b46e76d03b" (UID: "6d81fe02-4239-49ab-951e-c5b46e76d03b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:48:32.786675 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:48:32.786623 2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d81fe02-4239-49ab-951e-c5b46e76d03b-tmp-dir\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:48:32.786675 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:48:32.786640 2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d81fe02-4239-49ab-951e-c5b46e76d03b-model-cache\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:48:32.786675 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:48:32.786650 2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d81fe02-4239-49ab-951e-c5b46e76d03b-home\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:48:32.788563 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:48:32.788529 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d81fe02-4239-49ab-951e-c5b46e76d03b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6d81fe02-4239-49ab-951e-c5b46e76d03b" (UID: "6d81fe02-4239-49ab-951e-c5b46e76d03b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 16:48:32.788563 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:48:32.788538 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d81fe02-4239-49ab-951e-c5b46e76d03b-kube-api-access-fw6gh" (OuterVolumeSpecName: "kube-api-access-fw6gh") pod "6d81fe02-4239-49ab-951e-c5b46e76d03b" (UID: "6d81fe02-4239-49ab-951e-c5b46e76d03b"). InnerVolumeSpecName "kube-api-access-fw6gh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 16:48:32.788723 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:48:32.788590 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d81fe02-4239-49ab-951e-c5b46e76d03b-dshm" (OuterVolumeSpecName: "dshm") pod "6d81fe02-4239-49ab-951e-c5b46e76d03b" (UID: "6d81fe02-4239-49ab-951e-c5b46e76d03b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:48:32.887618 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:48:32.887583 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d81fe02-4239-49ab-951e-c5b46e76d03b-tls-certs\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:48:32.887618 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:48:32.887614 2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d81fe02-4239-49ab-951e-c5b46e76d03b-dshm\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:48:32.887618 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:48:32.887624 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fw6gh\" (UniqueName: \"kubernetes.io/projected/6d81fe02-4239-49ab-951e-c5b46e76d03b-kube-api-access-fw6gh\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:48:33.100125 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:48:33.100081 2569 generic.go:356] "Generic (PLEG): container finished" podID="6d81fe02-4239-49ab-951e-c5b46e76d03b" containerID="6818682d8693077f719f3651d3fcc41acb26959ac7084c024ecff49cf89be5f2" exitCode=0 Jul 09 16:48:33.100322 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:48:33.100147 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-aa22b3/test-model-aa22b3-kserve-7db58cc5bc-p8l98" Jul 09 16:48:33.100322 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:48:33.100167 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-aa22b3/test-model-aa22b3-kserve-7db58cc5bc-p8l98" event={"ID":"6d81fe02-4239-49ab-951e-c5b46e76d03b","Type":"ContainerDied","Data":"6818682d8693077f719f3651d3fcc41acb26959ac7084c024ecff49cf89be5f2"} Jul 09 16:48:33.100322 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:48:33.100215 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-aa22b3/test-model-aa22b3-kserve-7db58cc5bc-p8l98" event={"ID":"6d81fe02-4239-49ab-951e-c5b46e76d03b","Type":"ContainerDied","Data":"412604b1450fe0ff6c44221d7558f4809f0f7041827fb20eeed9dfbff7cb6fdb"} Jul 09 16:48:33.100322 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:48:33.100233 2569 scope.go:117] "RemoveContainer" containerID="6818682d8693077f719f3651d3fcc41acb26959ac7084c024ecff49cf89be5f2" Jul 09 16:48:33.112169 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:48:33.112143 2569 scope.go:117] "RemoveContainer" containerID="6818682d8693077f719f3651d3fcc41acb26959ac7084c024ecff49cf89be5f2" Jul 09 16:48:33.112619 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:48:33.112596 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6818682d8693077f719f3651d3fcc41acb26959ac7084c024ecff49cf89be5f2\": container with ID starting with 6818682d8693077f719f3651d3fcc41acb26959ac7084c024ecff49cf89be5f2 not found: ID does not exist" containerID="6818682d8693077f719f3651d3fcc41acb26959ac7084c024ecff49cf89be5f2" Jul 09 16:48:33.112701 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:48:33.112628 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6818682d8693077f719f3651d3fcc41acb26959ac7084c024ecff49cf89be5f2"} err="failed to get container status \"6818682d8693077f719f3651d3fcc41acb26959ac7084c024ecff49cf89be5f2\": rpc error: code = NotFound desc = could not find container \"6818682d8693077f719f3651d3fcc41acb26959ac7084c024ecff49cf89be5f2\": container with ID starting with 6818682d8693077f719f3651d3fcc41acb26959ac7084c024ecff49cf89be5f2 not found: ID does not exist" Jul 09 16:48:33.124456 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:48:33.124422 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-aa22b3/test-model-aa22b3-kserve-7db58cc5bc-p8l98"] Jul 09 16:48:33.128530 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:48:33.128505 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-aa22b3/test-model-aa22b3-kserve-7db58cc5bc-p8l98"] Jul 09 16:48:33.389805 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:48:33.389721 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d81fe02-4239-49ab-951e-c5b46e76d03b" path="/var/lib/kubelet/pods/6d81fe02-4239-49ab-951e-c5b46e76d03b/volumes" Jul 09 16:48:33.390174 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:48:33.390160 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e90de64e-44bd-4ed6-8254-084c6990147e" path="/var/lib/kubelet/pods/e90de64e-44bd-4ed6-8254-084c6990147e/volumes" Jul 09 16:49:01.476745 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:01.476712 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-aa22b3-67565ff7b8-v5p4r"] Jul 09 16:49:01.477373 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:01.476958 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aa22b3-67565ff7b8-v5p4r" podUID="40b30bda-77d8-4c7a-8ca9-97affc6e4181" containerName="maas-api" containerID="cri-o://94d0ca90383060b885497f45ee53e5a79c1cf4720e3400b8f236b9a2b981fa95" gracePeriod=30 Jul 09 16:49:01.733379 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:01.733314 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aa22b3-67565ff7b8-v5p4r" Jul 09 16:49:01.741783 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:01.741757 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40b30bda-77d8-4c7a-8ca9-97affc6e4181-maas-api-tls\") pod \"40b30bda-77d8-4c7a-8ca9-97affc6e4181\" (UID: \"40b30bda-77d8-4c7a-8ca9-97affc6e4181\") " Jul 09 16:49:01.741960 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:01.741810 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pvmvf\" (UniqueName: \"kubernetes.io/projected/40b30bda-77d8-4c7a-8ca9-97affc6e4181-kube-api-access-pvmvf\") pod \"40b30bda-77d8-4c7a-8ca9-97affc6e4181\" (UID: \"40b30bda-77d8-4c7a-8ca9-97affc6e4181\") " Jul 09 16:49:01.741960 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:01.741922 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40b30bda-77d8-4c7a-8ca9-97affc6e4181-openshift-service-ca\") pod \"40b30bda-77d8-4c7a-8ca9-97affc6e4181\" (UID: \"40b30bda-77d8-4c7a-8ca9-97affc6e4181\") " Jul 09 16:49:01.742279 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:01.742253 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40b30bda-77d8-4c7a-8ca9-97affc6e4181-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "40b30bda-77d8-4c7a-8ca9-97affc6e4181" (UID: "40b30bda-77d8-4c7a-8ca9-97affc6e4181"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 16:49:01.743803 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:01.743783 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40b30bda-77d8-4c7a-8ca9-97affc6e4181-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "40b30bda-77d8-4c7a-8ca9-97affc6e4181" (UID: "40b30bda-77d8-4c7a-8ca9-97affc6e4181"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 16:49:01.743943 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:01.743926 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40b30bda-77d8-4c7a-8ca9-97affc6e4181-kube-api-access-pvmvf" (OuterVolumeSpecName: "kube-api-access-pvmvf") pod "40b30bda-77d8-4c7a-8ca9-97affc6e4181" (UID: "40b30bda-77d8-4c7a-8ca9-97affc6e4181"). InnerVolumeSpecName "kube-api-access-pvmvf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 16:49:01.843547 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:01.843509 2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40b30bda-77d8-4c7a-8ca9-97affc6e4181-openshift-service-ca\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:49:01.843547 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:01.843541 2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40b30bda-77d8-4c7a-8ca9-97affc6e4181-maas-api-tls\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:49:01.843547 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:01.843550 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvmvf\" (UniqueName: \"kubernetes.io/projected/40b30bda-77d8-4c7a-8ca9-97affc6e4181-kube-api-access-pvmvf\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:49:02.245503 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:02.245469 2569 generic.go:356] "Generic (PLEG): container finished" podID="40b30bda-77d8-4c7a-8ca9-97affc6e4181" containerID="94d0ca90383060b885497f45ee53e5a79c1cf4720e3400b8f236b9a2b981fa95" exitCode=0 Jul 09 16:49:02.245738 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:02.245514 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aa22b3-67565ff7b8-v5p4r" Jul 09 16:49:02.245738 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:02.245558 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aa22b3-67565ff7b8-v5p4r" event={"ID":"40b30bda-77d8-4c7a-8ca9-97affc6e4181","Type":"ContainerDied","Data":"94d0ca90383060b885497f45ee53e5a79c1cf4720e3400b8f236b9a2b981fa95"} Jul 09 16:49:02.245738 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:02.245626 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aa22b3-67565ff7b8-v5p4r" event={"ID":"40b30bda-77d8-4c7a-8ca9-97affc6e4181","Type":"ContainerDied","Data":"bceb87afe7161a0f70b3531d3d2f5ce5bffd91776a099381ce685c9348f911d9"} Jul 09 16:49:02.245738 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:02.245657 2569 scope.go:117] "RemoveContainer" containerID="94d0ca90383060b885497f45ee53e5a79c1cf4720e3400b8f236b9a2b981fa95" Jul 09 16:49:02.255646 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:02.255627 2569 scope.go:117] "RemoveContainer" containerID="94d0ca90383060b885497f45ee53e5a79c1cf4720e3400b8f236b9a2b981fa95" Jul 09 16:49:02.255945 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:49:02.255923 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94d0ca90383060b885497f45ee53e5a79c1cf4720e3400b8f236b9a2b981fa95\": container with ID starting with 94d0ca90383060b885497f45ee53e5a79c1cf4720e3400b8f236b9a2b981fa95 not found: ID does not exist" containerID="94d0ca90383060b885497f45ee53e5a79c1cf4720e3400b8f236b9a2b981fa95" Jul 09 16:49:02.256005 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:02.255955 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94d0ca90383060b885497f45ee53e5a79c1cf4720e3400b8f236b9a2b981fa95"} err="failed to get container status \"94d0ca90383060b885497f45ee53e5a79c1cf4720e3400b8f236b9a2b981fa95\": rpc error: code = NotFound desc = could not find container \"94d0ca90383060b885497f45ee53e5a79c1cf4720e3400b8f236b9a2b981fa95\": container with ID starting with 94d0ca90383060b885497f45ee53e5a79c1cf4720e3400b8f236b9a2b981fa95 not found: ID does not exist" Jul 09 16:49:02.268179 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:02.268151 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-aa22b3-67565ff7b8-v5p4r"] Jul 09 16:49:02.273373 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:02.273349 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-aa22b3-67565ff7b8-v5p4r"] Jul 09 16:49:03.389293 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:03.389254 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40b30bda-77d8-4c7a-8ca9-97affc6e4181" path="/var/lib/kubelet/pods/40b30bda-77d8-4c7a-8ca9-97affc6e4181/volumes" Jul 09 16:49:04.684453 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:04.684407 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aa22b3-67565ff7b8-v5p4r" podUID="40b30bda-77d8-4c7a-8ca9-97affc6e4181" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.57:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Jul 09 16:49:11.085521 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:11.085483 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6d03d1-cb4b45889-vcgrk"] Jul 09 16:49:11.085983 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:11.085864 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40b30bda-77d8-4c7a-8ca9-97affc6e4181" containerName="maas-api" Jul 09 16:49:11.085983 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:11.085889 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="40b30bda-77d8-4c7a-8ca9-97affc6e4181" containerName="maas-api" Jul 09 16:49:11.085983 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:11.085900 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d81fe02-4239-49ab-951e-c5b46e76d03b" containerName="main" Jul 09 16:49:11.085983 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:11.085906 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d81fe02-4239-49ab-951e-c5b46e76d03b" containerName="main" Jul 09 16:49:11.085983 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:11.085920 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0d03792-498a-4114-bda5-b4eb572570af" containerName="maas-api" Jul 09 16:49:11.085983 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:11.085928 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0d03792-498a-4114-bda5-b4eb572570af" containerName="maas-api" Jul 09 16:49:11.085983 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:11.085934 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e90de64e-44bd-4ed6-8254-084c6990147e" containerName="main" Jul 09 16:49:11.085983 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:11.085939 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e90de64e-44bd-4ed6-8254-084c6990147e" containerName="main" Jul 09 16:49:11.086273 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:11.086003 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0d03792-498a-4114-bda5-b4eb572570af" containerName="maas-api" Jul 09 16:49:11.086273 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:11.086012 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="40b30bda-77d8-4c7a-8ca9-97affc6e4181" containerName="maas-api" Jul 09 16:49:11.086273 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:11.086019 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="e90de64e-44bd-4ed6-8254-084c6990147e" containerName="main" Jul 09 16:49:11.086273 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:11.086025 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d81fe02-4239-49ab-951e-c5b46e76d03b" containerName="main" Jul 09 16:49:11.090348 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:11.090325 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6d03d1-cb4b45889-vcgrk" Jul 09 16:49:11.093596 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:11.093562 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-6d03d1-serving-cert\"" Jul 09 16:49:11.100702 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:11.100677 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6d03d1-cb4b45889-vcgrk"] Jul 09 16:49:11.122454 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:11.122418 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vptbr\" (UniqueName: \"kubernetes.io/projected/5d7948c2-3acb-4ba8-a6d0-7d584196b74e-kube-api-access-vptbr\") pod \"maas-api-e2e-api-a-6d03d1-cb4b45889-vcgrk\" (UID: \"5d7948c2-3acb-4ba8-a6d0-7d584196b74e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6d03d1-cb4b45889-vcgrk" Jul 09 16:49:11.122616 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:11.122461 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d7948c2-3acb-4ba8-a6d0-7d584196b74e-openshift-service-ca\") pod \"maas-api-e2e-api-a-6d03d1-cb4b45889-vcgrk\" (UID: \"5d7948c2-3acb-4ba8-a6d0-7d584196b74e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6d03d1-cb4b45889-vcgrk" Jul 09 16:49:11.122616 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:11.122556 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d7948c2-3acb-4ba8-a6d0-7d584196b74e-maas-api-tls\") pod \"maas-api-e2e-api-a-6d03d1-cb4b45889-vcgrk\" (UID: \"5d7948c2-3acb-4ba8-a6d0-7d584196b74e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6d03d1-cb4b45889-vcgrk" Jul 09 16:49:11.223747 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:11.223715 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d7948c2-3acb-4ba8-a6d0-7d584196b74e-maas-api-tls\") pod \"maas-api-e2e-api-a-6d03d1-cb4b45889-vcgrk\" (UID: \"5d7948c2-3acb-4ba8-a6d0-7d584196b74e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6d03d1-cb4b45889-vcgrk" Jul 09 16:49:11.223991 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:11.223782 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vptbr\" (UniqueName: \"kubernetes.io/projected/5d7948c2-3acb-4ba8-a6d0-7d584196b74e-kube-api-access-vptbr\") pod \"maas-api-e2e-api-a-6d03d1-cb4b45889-vcgrk\" (UID: \"5d7948c2-3acb-4ba8-a6d0-7d584196b74e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6d03d1-cb4b45889-vcgrk" Jul 09 16:49:11.223991 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:11.223816 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d7948c2-3acb-4ba8-a6d0-7d584196b74e-openshift-service-ca\") pod \"maas-api-e2e-api-a-6d03d1-cb4b45889-vcgrk\" (UID: \"5d7948c2-3acb-4ba8-a6d0-7d584196b74e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6d03d1-cb4b45889-vcgrk" Jul 09 16:49:11.223991 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:49:11.223953 2569 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-6d03d1-serving-cert: secret "maas-api-e2e-api-a-6d03d1-serving-cert" not found Jul 09 16:49:11.224167 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:49:11.224012 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d7948c2-3acb-4ba8-a6d0-7d584196b74e-maas-api-tls podName:5d7948c2-3acb-4ba8-a6d0-7d584196b74e nodeName:}" failed. No retries permitted until 2026-07-09 16:49:11.723996238 +0000 UTC m=+2036.917102597 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5d7948c2-3acb-4ba8-a6d0-7d584196b74e-maas-api-tls") pod "maas-api-e2e-api-a-6d03d1-cb4b45889-vcgrk" (UID: "5d7948c2-3acb-4ba8-a6d0-7d584196b74e") : secret "maas-api-e2e-api-a-6d03d1-serving-cert" not found Jul 09 16:49:11.224469 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:11.224451 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d7948c2-3acb-4ba8-a6d0-7d584196b74e-openshift-service-ca\") pod \"maas-api-e2e-api-a-6d03d1-cb4b45889-vcgrk\" (UID: \"5d7948c2-3acb-4ba8-a6d0-7d584196b74e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6d03d1-cb4b45889-vcgrk" Jul 09 16:49:11.232539 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:11.232520 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vptbr\" (UniqueName: \"kubernetes.io/projected/5d7948c2-3acb-4ba8-a6d0-7d584196b74e-kube-api-access-vptbr\") pod \"maas-api-e2e-api-a-6d03d1-cb4b45889-vcgrk\" (UID: \"5d7948c2-3acb-4ba8-a6d0-7d584196b74e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6d03d1-cb4b45889-vcgrk" Jul 09 16:49:11.730002 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:11.729953 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d7948c2-3acb-4ba8-a6d0-7d584196b74e-maas-api-tls\") pod \"maas-api-e2e-api-a-6d03d1-cb4b45889-vcgrk\" (UID: \"5d7948c2-3acb-4ba8-a6d0-7d584196b74e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6d03d1-cb4b45889-vcgrk" Jul 09 16:49:11.732414 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:11.732380 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d7948c2-3acb-4ba8-a6d0-7d584196b74e-maas-api-tls\") pod \"maas-api-e2e-api-a-6d03d1-cb4b45889-vcgrk\" (UID: \"5d7948c2-3acb-4ba8-a6d0-7d584196b74e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6d03d1-cb4b45889-vcgrk" Jul 09 16:49:12.001855 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:12.001752 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6d03d1-cb4b45889-vcgrk" Jul 09 16:49:12.147908 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:12.147866 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6d03d1-cb4b45889-vcgrk"] Jul 09 16:49:12.150319 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:49:12.150286 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d7948c2_3acb_4ba8_a6d0_7d584196b74e.slice/crio-ea0217a5d643a21b137127a10bacf6fe9fd1061404becf1205e14c0d13c69dbf WatchSource:0}: Error finding container ea0217a5d643a21b137127a10bacf6fe9fd1061404becf1205e14c0d13c69dbf: Status 404 returned error can't find the container with id ea0217a5d643a21b137127a10bacf6fe9fd1061404becf1205e14c0d13c69dbf Jul 09 16:49:12.297232 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:12.297132 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6d03d1-cb4b45889-vcgrk" event={"ID":"5d7948c2-3acb-4ba8-a6d0-7d584196b74e","Type":"ContainerStarted","Data":"ea0217a5d643a21b137127a10bacf6fe9fd1061404becf1205e14c0d13c69dbf"} Jul 09 16:49:13.304155 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:13.304122 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6d03d1-cb4b45889-vcgrk" event={"ID":"5d7948c2-3acb-4ba8-a6d0-7d584196b74e","Type":"ContainerStarted","Data":"14e4be06218cee86e9eaba043eea9944f8484ee955bc00c6b8e5808abdea1859"} Jul 09 16:49:13.304548 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:13.304253 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6d03d1-cb4b45889-vcgrk" Jul 09 16:49:13.321266 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:13.321212 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6d03d1-cb4b45889-vcgrk" podStartSLOduration=1.828204065 podStartE2EDuration="2.32119297s" totalImagesPullingTime="492.988905ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:49:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:49:13.319910988 +0000 UTC m=+2038.513017369" watchObservedRunningTime="2026-07-09 16:49:13.32119297 +0000 UTC m=+2038.514299351" Jul 09 16:49:19.161220 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:19.161187 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-554cc9-78bc8894f7-cr2rx"] Jul 09 16:49:19.166355 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:19.166329 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-554cc9-78bc8894f7-cr2rx" Jul 09 16:49:19.174373 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:19.174328 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-554cc9-serving-cert\"" Jul 09 16:49:19.196106 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:19.196075 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-554cc9-78bc8894f7-cr2rx"] Jul 09 16:49:19.298345 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:19.298308 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8lsk\" (UniqueName: \"kubernetes.io/projected/e3a1f12b-b9d5-4717-b781-7d86071e6700-kube-api-access-g8lsk\") pod \"maas-api-e2e-api-b-554cc9-78bc8894f7-cr2rx\" (UID: \"e3a1f12b-b9d5-4717-b781-7d86071e6700\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-554cc9-78bc8894f7-cr2rx" Jul 09 16:49:19.298345 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:19.298348 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3a1f12b-b9d5-4717-b781-7d86071e6700-maas-api-tls\") pod \"maas-api-e2e-api-b-554cc9-78bc8894f7-cr2rx\" (UID: \"e3a1f12b-b9d5-4717-b781-7d86071e6700\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-554cc9-78bc8894f7-cr2rx" Jul 09 16:49:19.298555 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:19.298429 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3a1f12b-b9d5-4717-b781-7d86071e6700-openshift-service-ca\") pod \"maas-api-e2e-api-b-554cc9-78bc8894f7-cr2rx\" (UID: \"e3a1f12b-b9d5-4717-b781-7d86071e6700\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-554cc9-78bc8894f7-cr2rx" Jul 09 16:49:19.315273 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:19.315246 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6d03d1-cb4b45889-vcgrk" Jul 09 16:49:19.399163 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:19.399129 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8lsk\" (UniqueName: \"kubernetes.io/projected/e3a1f12b-b9d5-4717-b781-7d86071e6700-kube-api-access-g8lsk\") pod \"maas-api-e2e-api-b-554cc9-78bc8894f7-cr2rx\" (UID: \"e3a1f12b-b9d5-4717-b781-7d86071e6700\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-554cc9-78bc8894f7-cr2rx" Jul 09 16:49:19.399345 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:19.399171 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3a1f12b-b9d5-4717-b781-7d86071e6700-maas-api-tls\") pod \"maas-api-e2e-api-b-554cc9-78bc8894f7-cr2rx\" (UID: \"e3a1f12b-b9d5-4717-b781-7d86071e6700\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-554cc9-78bc8894f7-cr2rx" Jul 09 16:49:19.399345 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:19.399242 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3a1f12b-b9d5-4717-b781-7d86071e6700-openshift-service-ca\") pod \"maas-api-e2e-api-b-554cc9-78bc8894f7-cr2rx\" (UID: \"e3a1f12b-b9d5-4717-b781-7d86071e6700\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-554cc9-78bc8894f7-cr2rx" Jul 09 16:49:19.400263 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:19.400232 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3a1f12b-b9d5-4717-b781-7d86071e6700-openshift-service-ca\") pod \"maas-api-e2e-api-b-554cc9-78bc8894f7-cr2rx\" (UID: \"e3a1f12b-b9d5-4717-b781-7d86071e6700\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-554cc9-78bc8894f7-cr2rx" Jul 09 16:49:19.402449 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:19.402422 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3a1f12b-b9d5-4717-b781-7d86071e6700-maas-api-tls\") pod \"maas-api-e2e-api-b-554cc9-78bc8894f7-cr2rx\" (UID: \"e3a1f12b-b9d5-4717-b781-7d86071e6700\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-554cc9-78bc8894f7-cr2rx" Jul 09 16:49:19.427271 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:19.427205 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8lsk\" (UniqueName: \"kubernetes.io/projected/e3a1f12b-b9d5-4717-b781-7d86071e6700-kube-api-access-g8lsk\") pod \"maas-api-e2e-api-b-554cc9-78bc8894f7-cr2rx\" (UID: \"e3a1f12b-b9d5-4717-b781-7d86071e6700\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-554cc9-78bc8894f7-cr2rx" Jul 09 16:49:19.478937 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:19.478897 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-554cc9-78bc8894f7-cr2rx" Jul 09 16:49:19.631239 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:19.631205 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-554cc9-78bc8894f7-cr2rx"] Jul 09 16:49:19.635454 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:49:19.635427 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3a1f12b_b9d5_4717_b781_7d86071e6700.slice/crio-cef498f13fac76b07ad7dd1b453efeeab98ab7ea6d9f29acb190f81b150f018b WatchSource:0}: Error finding container cef498f13fac76b07ad7dd1b453efeeab98ab7ea6d9f29acb190f81b150f018b: Status 404 returned error can't find the container with id cef498f13fac76b07ad7dd1b453efeeab98ab7ea6d9f29acb190f81b150f018b Jul 09 16:49:20.340785 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:20.340750 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-554cc9-78bc8894f7-cr2rx" event={"ID":"e3a1f12b-b9d5-4717-b781-7d86071e6700","Type":"ContainerStarted","Data":"cef498f13fac76b07ad7dd1b453efeeab98ab7ea6d9f29acb190f81b150f018b"} Jul 09 16:49:21.347514 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:21.347472 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-554cc9-78bc8894f7-cr2rx" event={"ID":"e3a1f12b-b9d5-4717-b781-7d86071e6700","Type":"ContainerStarted","Data":"654b3618a8255b23e80c10290ae7b6e077b2889fb8deb9b459e598ecb4ff7cfa"} Jul 09 16:49:21.348013 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:21.347560 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-554cc9-78bc8894f7-cr2rx" Jul 09 16:49:21.367823 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:21.367770 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-554cc9-78bc8894f7-cr2rx" podStartSLOduration=1.2748697230000001 podStartE2EDuration="2.367751799s" totalImagesPullingTime="1.092882076s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:49:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:49:21.365677992 +0000 UTC m=+2046.558784387" watchObservedRunningTime="2026-07-09 16:49:21.367751799 +0000 UTC m=+2046.560858180" Jul 09 16:49:27.358686 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:27.358655 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-554cc9-78bc8894f7-cr2rx" Jul 09 16:49:32.476565 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:32.476530 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6d03d1-cb4b45889-vcgrk"] Jul 09 16:49:32.476994 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:32.476761 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6d03d1-cb4b45889-vcgrk" podUID="5d7948c2-3acb-4ba8-a6d0-7d584196b74e" containerName="maas-api" containerID="cri-o://14e4be06218cee86e9eaba043eea9944f8484ee955bc00c6b8e5808abdea1859" gracePeriod=30 Jul 09 16:49:32.735024 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:32.734942 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6d03d1-cb4b45889-vcgrk" Jul 09 16:49:32.828130 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:32.828095 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vptbr\" (UniqueName: \"kubernetes.io/projected/5d7948c2-3acb-4ba8-a6d0-7d584196b74e-kube-api-access-vptbr\") pod \"5d7948c2-3acb-4ba8-a6d0-7d584196b74e\" (UID: \"5d7948c2-3acb-4ba8-a6d0-7d584196b74e\") " Jul 09 16:49:32.828299 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:32.828165 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d7948c2-3acb-4ba8-a6d0-7d584196b74e-maas-api-tls\") pod \"5d7948c2-3acb-4ba8-a6d0-7d584196b74e\" (UID: \"5d7948c2-3acb-4ba8-a6d0-7d584196b74e\") " Jul 09 16:49:32.828299 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:32.828227 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d7948c2-3acb-4ba8-a6d0-7d584196b74e-openshift-service-ca\") pod \"5d7948c2-3acb-4ba8-a6d0-7d584196b74e\" (UID: \"5d7948c2-3acb-4ba8-a6d0-7d584196b74e\") " Jul 09 16:49:32.828647 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:32.828620 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d7948c2-3acb-4ba8-a6d0-7d584196b74e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5d7948c2-3acb-4ba8-a6d0-7d584196b74e" (UID: "5d7948c2-3acb-4ba8-a6d0-7d584196b74e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 16:49:32.830357 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:32.830333 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d7948c2-3acb-4ba8-a6d0-7d584196b74e-kube-api-access-vptbr" (OuterVolumeSpecName: "kube-api-access-vptbr") pod "5d7948c2-3acb-4ba8-a6d0-7d584196b74e" (UID: "5d7948c2-3acb-4ba8-a6d0-7d584196b74e"). InnerVolumeSpecName "kube-api-access-vptbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 16:49:32.830429 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:32.830366 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d7948c2-3acb-4ba8-a6d0-7d584196b74e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5d7948c2-3acb-4ba8-a6d0-7d584196b74e" (UID: "5d7948c2-3acb-4ba8-a6d0-7d584196b74e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 16:49:32.929066 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:32.929032 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vptbr\" (UniqueName: \"kubernetes.io/projected/5d7948c2-3acb-4ba8-a6d0-7d584196b74e-kube-api-access-vptbr\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:49:32.929066 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:32.929061 2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d7948c2-3acb-4ba8-a6d0-7d584196b74e-maas-api-tls\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:49:32.929066 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:32.929071 2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d7948c2-3acb-4ba8-a6d0-7d584196b74e-openshift-service-ca\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:49:33.407723 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:33.407692 2569 generic.go:356] "Generic (PLEG): container finished" podID="5d7948c2-3acb-4ba8-a6d0-7d584196b74e" containerID="14e4be06218cee86e9eaba043eea9944f8484ee955bc00c6b8e5808abdea1859" exitCode=0 Jul 09 16:49:33.407942 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:33.407739 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6d03d1-cb4b45889-vcgrk" Jul 09 16:49:33.407942 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:33.407773 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6d03d1-cb4b45889-vcgrk" event={"ID":"5d7948c2-3acb-4ba8-a6d0-7d584196b74e","Type":"ContainerDied","Data":"14e4be06218cee86e9eaba043eea9944f8484ee955bc00c6b8e5808abdea1859"} Jul 09 16:49:33.407942 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:33.407835 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6d03d1-cb4b45889-vcgrk" event={"ID":"5d7948c2-3acb-4ba8-a6d0-7d584196b74e","Type":"ContainerDied","Data":"ea0217a5d643a21b137127a10bacf6fe9fd1061404becf1205e14c0d13c69dbf"} Jul 09 16:49:33.407942 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:33.407919 2569 scope.go:117] "RemoveContainer" containerID="14e4be06218cee86e9eaba043eea9944f8484ee955bc00c6b8e5808abdea1859" Jul 09 16:49:33.417033 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:33.417006 2569 scope.go:117] "RemoveContainer" containerID="14e4be06218cee86e9eaba043eea9944f8484ee955bc00c6b8e5808abdea1859" Jul 09 16:49:33.417322 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:49:33.417294 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14e4be06218cee86e9eaba043eea9944f8484ee955bc00c6b8e5808abdea1859\": container with ID starting with 14e4be06218cee86e9eaba043eea9944f8484ee955bc00c6b8e5808abdea1859 not found: ID does not exist" containerID="14e4be06218cee86e9eaba043eea9944f8484ee955bc00c6b8e5808abdea1859" Jul 09 16:49:33.417422 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:33.417333 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14e4be06218cee86e9eaba043eea9944f8484ee955bc00c6b8e5808abdea1859"} err="failed to get container status \"14e4be06218cee86e9eaba043eea9944f8484ee955bc00c6b8e5808abdea1859\": rpc error: code = NotFound desc = could not find container \"14e4be06218cee86e9eaba043eea9944f8484ee955bc00c6b8e5808abdea1859\": container with ID starting with 14e4be06218cee86e9eaba043eea9944f8484ee955bc00c6b8e5808abdea1859 not found: ID does not exist" Jul 09 16:49:33.425155 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:33.425114 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6d03d1-cb4b45889-vcgrk"] Jul 09 16:49:33.429264 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:33.429237 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6d03d1-cb4b45889-vcgrk"] Jul 09 16:49:35.389650 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:35.389617 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d7948c2-3acb-4ba8-a6d0-7d584196b74e" path="/var/lib/kubelet/pods/5d7948c2-3acb-4ba8-a6d0-7d584196b74e/volumes" Jul 09 16:49:39.417482 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:39.417396 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-554cc9-78bc8894f7-cr2rx"] Jul 09 16:49:39.417925 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:39.417657 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-554cc9-78bc8894f7-cr2rx" podUID="e3a1f12b-b9d5-4717-b781-7d86071e6700" containerName="maas-api" containerID="cri-o://654b3618a8255b23e80c10290ae7b6e077b2889fb8deb9b459e598ecb4ff7cfa" gracePeriod=30 Jul 09 16:49:39.677604 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:39.677538 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-554cc9-78bc8894f7-cr2rx" Jul 09 16:49:39.794750 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:39.794707 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g8lsk\" (UniqueName: \"kubernetes.io/projected/e3a1f12b-b9d5-4717-b781-7d86071e6700-kube-api-access-g8lsk\") pod \"e3a1f12b-b9d5-4717-b781-7d86071e6700\" (UID: \"e3a1f12b-b9d5-4717-b781-7d86071e6700\") " Jul 09 16:49:39.794750 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:39.794750 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3a1f12b-b9d5-4717-b781-7d86071e6700-maas-api-tls\") pod \"e3a1f12b-b9d5-4717-b781-7d86071e6700\" (UID: \"e3a1f12b-b9d5-4717-b781-7d86071e6700\") " Jul 09 16:49:39.795052 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:39.794804 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3a1f12b-b9d5-4717-b781-7d86071e6700-openshift-service-ca\") pod \"e3a1f12b-b9d5-4717-b781-7d86071e6700\" (UID: \"e3a1f12b-b9d5-4717-b781-7d86071e6700\") " Jul 09 16:49:39.795231 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:39.795207 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3a1f12b-b9d5-4717-b781-7d86071e6700-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e3a1f12b-b9d5-4717-b781-7d86071e6700" (UID: "e3a1f12b-b9d5-4717-b781-7d86071e6700"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 16:49:39.796930 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:39.796892 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3a1f12b-b9d5-4717-b781-7d86071e6700-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e3a1f12b-b9d5-4717-b781-7d86071e6700" (UID: "e3a1f12b-b9d5-4717-b781-7d86071e6700"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 16:49:39.796930 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:39.796917 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3a1f12b-b9d5-4717-b781-7d86071e6700-kube-api-access-g8lsk" (OuterVolumeSpecName: "kube-api-access-g8lsk") pod "e3a1f12b-b9d5-4717-b781-7d86071e6700" (UID: "e3a1f12b-b9d5-4717-b781-7d86071e6700"). InnerVolumeSpecName "kube-api-access-g8lsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 16:49:39.896500 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:39.896457 2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3a1f12b-b9d5-4717-b781-7d86071e6700-openshift-service-ca\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:49:39.896500 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:39.896491 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8lsk\" (UniqueName: \"kubernetes.io/projected/e3a1f12b-b9d5-4717-b781-7d86071e6700-kube-api-access-g8lsk\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:49:39.896500 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:39.896502 2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3a1f12b-b9d5-4717-b781-7d86071e6700-maas-api-tls\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:49:40.451126 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:40.451089 2569 generic.go:356] "Generic (PLEG): container finished" podID="e3a1f12b-b9d5-4717-b781-7d86071e6700" containerID="654b3618a8255b23e80c10290ae7b6e077b2889fb8deb9b459e598ecb4ff7cfa" exitCode=0 Jul 09 16:49:40.451556 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:40.451135 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-554cc9-78bc8894f7-cr2rx" event={"ID":"e3a1f12b-b9d5-4717-b781-7d86071e6700","Type":"ContainerDied","Data":"654b3618a8255b23e80c10290ae7b6e077b2889fb8deb9b459e598ecb4ff7cfa"} Jul 09 16:49:40.451556 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:40.451145 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-554cc9-78bc8894f7-cr2rx" Jul 09 16:49:40.451556 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:40.451188 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-554cc9-78bc8894f7-cr2rx" event={"ID":"e3a1f12b-b9d5-4717-b781-7d86071e6700","Type":"ContainerDied","Data":"cef498f13fac76b07ad7dd1b453efeeab98ab7ea6d9f29acb190f81b150f018b"} Jul 09 16:49:40.451556 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:40.451218 2569 scope.go:117] "RemoveContainer" containerID="654b3618a8255b23e80c10290ae7b6e077b2889fb8deb9b459e598ecb4ff7cfa" Jul 09 16:49:40.460395 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:40.460377 2569 scope.go:117] "RemoveContainer" containerID="654b3618a8255b23e80c10290ae7b6e077b2889fb8deb9b459e598ecb4ff7cfa" Jul 09 16:49:40.460644 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:49:40.460625 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"654b3618a8255b23e80c10290ae7b6e077b2889fb8deb9b459e598ecb4ff7cfa\": container with ID starting with 654b3618a8255b23e80c10290ae7b6e077b2889fb8deb9b459e598ecb4ff7cfa not found: ID does not exist" containerID="654b3618a8255b23e80c10290ae7b6e077b2889fb8deb9b459e598ecb4ff7cfa" Jul 09 16:49:40.460712 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:40.460656 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"654b3618a8255b23e80c10290ae7b6e077b2889fb8deb9b459e598ecb4ff7cfa"} err="failed to get container status \"654b3618a8255b23e80c10290ae7b6e077b2889fb8deb9b459e598ecb4ff7cfa\": rpc error: code = NotFound desc = could not find container \"654b3618a8255b23e80c10290ae7b6e077b2889fb8deb9b459e598ecb4ff7cfa\": container with ID starting with 654b3618a8255b23e80c10290ae7b6e077b2889fb8deb9b459e598ecb4ff7cfa not found: ID does not exist" Jul 09 16:49:40.493718 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:40.493684 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-554cc9-78bc8894f7-cr2rx"] Jul 09 16:49:40.501535 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:40.501506 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-554cc9-78bc8894f7-cr2rx"] Jul 09 16:49:41.390466 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:41.390428 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3a1f12b-b9d5-4717-b781-7d86071e6700" path="/var/lib/kubelet/pods/e3a1f12b-b9d5-4717-b781-7d86071e6700/volumes" Jul 09 16:49:49.295483 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:49.295447 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-bef56b-57c67b66fb-wqdxl"] Jul 09 16:49:49.296018 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:49.295957 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d7948c2-3acb-4ba8-a6d0-7d584196b74e" containerName="maas-api" Jul 09 16:49:49.296018 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:49.295976 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d7948c2-3acb-4ba8-a6d0-7d584196b74e" containerName="maas-api" Jul 09 16:49:49.296018 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:49.295987 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3a1f12b-b9d5-4717-b781-7d86071e6700" containerName="maas-api" Jul 09 16:49:49.296018 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:49.295994 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3a1f12b-b9d5-4717-b781-7d86071e6700" containerName="maas-api" Jul 09 16:49:49.296255 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:49.296097 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3a1f12b-b9d5-4717-b781-7d86071e6700" containerName="maas-api" Jul 09 16:49:49.296255 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:49.296111 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d7948c2-3acb-4ba8-a6d0-7d584196b74e" containerName="maas-api" Jul 09 16:49:49.299553 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:49.299518 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-bef56b-57c67b66fb-wqdxl" Jul 09 16:49:49.302256 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:49.302232 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-bef56b-serving-cert\"" Jul 09 16:49:49.312560 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:49.312520 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-bef56b-57c67b66fb-wqdxl"] Jul 09 16:49:49.387984 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:49.387946 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq7xd\" (UniqueName: \"kubernetes.io/projected/92b20001-ee0f-43ec-a9ab-8ddb1a6e0cd1-kube-api-access-xq7xd\") pod \"maas-api-e2e-shared-a-bef56b-57c67b66fb-wqdxl\" (UID: \"92b20001-ee0f-43ec-a9ab-8ddb1a6e0cd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-bef56b-57c67b66fb-wqdxl" Jul 09 16:49:49.388184 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:49.387997 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92b20001-ee0f-43ec-a9ab-8ddb1a6e0cd1-maas-api-tls\") pod \"maas-api-e2e-shared-a-bef56b-57c67b66fb-wqdxl\" (UID: \"92b20001-ee0f-43ec-a9ab-8ddb1a6e0cd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-bef56b-57c67b66fb-wqdxl" Jul 09 16:49:49.388184 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:49.388074 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92b20001-ee0f-43ec-a9ab-8ddb1a6e0cd1-openshift-service-ca\") pod \"maas-api-e2e-shared-a-bef56b-57c67b66fb-wqdxl\" (UID: \"92b20001-ee0f-43ec-a9ab-8ddb1a6e0cd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-bef56b-57c67b66fb-wqdxl" Jul 09 16:49:49.489041 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:49.489004 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92b20001-ee0f-43ec-a9ab-8ddb1a6e0cd1-maas-api-tls\") pod \"maas-api-e2e-shared-a-bef56b-57c67b66fb-wqdxl\" (UID: \"92b20001-ee0f-43ec-a9ab-8ddb1a6e0cd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-bef56b-57c67b66fb-wqdxl" Jul 09 16:49:49.489200 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:49.489062 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92b20001-ee0f-43ec-a9ab-8ddb1a6e0cd1-openshift-service-ca\") pod \"maas-api-e2e-shared-a-bef56b-57c67b66fb-wqdxl\" (UID: \"92b20001-ee0f-43ec-a9ab-8ddb1a6e0cd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-bef56b-57c67b66fb-wqdxl" Jul 09 16:49:49.489200 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:49.489132 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq7xd\" (UniqueName: \"kubernetes.io/projected/92b20001-ee0f-43ec-a9ab-8ddb1a6e0cd1-kube-api-access-xq7xd\") pod \"maas-api-e2e-shared-a-bef56b-57c67b66fb-wqdxl\" (UID: \"92b20001-ee0f-43ec-a9ab-8ddb1a6e0cd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-bef56b-57c67b66fb-wqdxl" Jul 09 16:49:49.489800 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:49.489773 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92b20001-ee0f-43ec-a9ab-8ddb1a6e0cd1-openshift-service-ca\") pod \"maas-api-e2e-shared-a-bef56b-57c67b66fb-wqdxl\" (UID: \"92b20001-ee0f-43ec-a9ab-8ddb1a6e0cd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-bef56b-57c67b66fb-wqdxl" Jul 09 16:49:49.491676 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:49.491648 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92b20001-ee0f-43ec-a9ab-8ddb1a6e0cd1-maas-api-tls\") pod \"maas-api-e2e-shared-a-bef56b-57c67b66fb-wqdxl\" (UID: \"92b20001-ee0f-43ec-a9ab-8ddb1a6e0cd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-bef56b-57c67b66fb-wqdxl" Jul 09 16:49:49.498264 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:49.498239 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq7xd\" (UniqueName: \"kubernetes.io/projected/92b20001-ee0f-43ec-a9ab-8ddb1a6e0cd1-kube-api-access-xq7xd\") pod \"maas-api-e2e-shared-a-bef56b-57c67b66fb-wqdxl\" (UID: \"92b20001-ee0f-43ec-a9ab-8ddb1a6e0cd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-bef56b-57c67b66fb-wqdxl" Jul 09 16:49:49.614004 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:49.613964 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-bef56b-57c67b66fb-wqdxl" Jul 09 16:49:49.762043 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:49.762013 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-bef56b-57c67b66fb-wqdxl"] Jul 09 16:49:49.764672 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:49:49.764643 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92b20001_ee0f_43ec_a9ab_8ddb1a6e0cd1.slice/crio-7f73df0bf1ab9168e82350702891f1375b764c64cc36c9994fcf92f04a95bb79 WatchSource:0}: Error finding container 7f73df0bf1ab9168e82350702891f1375b764c64cc36c9994fcf92f04a95bb79: Status 404 returned error can't find the container with id 7f73df0bf1ab9168e82350702891f1375b764c64cc36c9994fcf92f04a95bb79 Jul 09 16:49:50.504573 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:50.504540 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-bef56b-57c67b66fb-wqdxl" event={"ID":"92b20001-ee0f-43ec-a9ab-8ddb1a6e0cd1","Type":"ContainerStarted","Data":"f945ae412f6bb8a4544961b2916d8173a6f75fb56d3d1cfcdf5f96cf459705b9"} Jul 09 16:49:50.505043 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:50.504607 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-bef56b-57c67b66fb-wqdxl" event={"ID":"92b20001-ee0f-43ec-a9ab-8ddb1a6e0cd1","Type":"ContainerStarted","Data":"7f73df0bf1ab9168e82350702891f1375b764c64cc36c9994fcf92f04a95bb79"} Jul 09 16:49:50.505043 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:50.504651 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-bef56b-57c67b66fb-wqdxl" Jul 09 16:49:50.521591 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:50.521532 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-bef56b-57c67b66fb-wqdxl" podStartSLOduration=0.989288083 podStartE2EDuration="1.521514563s" totalImagesPullingTime="532.22648ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:49:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:49:50.520388166 +0000 UTC m=+2075.713494548" watchObservedRunningTime="2026-07-09 16:49:50.521514563 +0000 UTC m=+2075.714620943" Jul 09 16:49:56.516223 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:56.516192 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-bef56b-57c67b66fb-wqdxl" Jul 09 16:49:57.479195 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:57.479139 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-6bf011-89fc49d7c-cbp7x"] Jul 09 16:49:57.483229 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:57.483201 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6bf011-89fc49d7c-cbp7x" Jul 09 16:49:57.486001 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:57.485978 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-6bf011-serving-cert\"" Jul 09 16:49:57.493852 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:57.493781 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-6bf011-89fc49d7c-cbp7x"] Jul 09 16:49:57.561921 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:57.561863 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1425d20c-1dde-4d40-b44e-490af34d6f0c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-6bf011-89fc49d7c-cbp7x\" (UID: \"1425d20c-1dde-4d40-b44e-490af34d6f0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6bf011-89fc49d7c-cbp7x" Jul 09 16:49:57.562338 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:57.561933 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1425d20c-1dde-4d40-b44e-490af34d6f0c-maas-api-tls\") pod \"maas-api-e2e-shared-b-6bf011-89fc49d7c-cbp7x\" (UID: \"1425d20c-1dde-4d40-b44e-490af34d6f0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6bf011-89fc49d7c-cbp7x" Jul 09 16:49:57.562338 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:57.561957 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qg5l\" (UniqueName: \"kubernetes.io/projected/1425d20c-1dde-4d40-b44e-490af34d6f0c-kube-api-access-7qg5l\") pod \"maas-api-e2e-shared-b-6bf011-89fc49d7c-cbp7x\" (UID: \"1425d20c-1dde-4d40-b44e-490af34d6f0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6bf011-89fc49d7c-cbp7x" Jul 09 16:49:57.663334 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:57.663299 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1425d20c-1dde-4d40-b44e-490af34d6f0c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-6bf011-89fc49d7c-cbp7x\" (UID: \"1425d20c-1dde-4d40-b44e-490af34d6f0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6bf011-89fc49d7c-cbp7x" Jul 09 16:49:57.663545 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:57.663351 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1425d20c-1dde-4d40-b44e-490af34d6f0c-maas-api-tls\") pod \"maas-api-e2e-shared-b-6bf011-89fc49d7c-cbp7x\" (UID: \"1425d20c-1dde-4d40-b44e-490af34d6f0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6bf011-89fc49d7c-cbp7x" Jul 09 16:49:57.663545 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:57.663466 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qg5l\" (UniqueName: \"kubernetes.io/projected/1425d20c-1dde-4d40-b44e-490af34d6f0c-kube-api-access-7qg5l\") pod \"maas-api-e2e-shared-b-6bf011-89fc49d7c-cbp7x\" (UID: \"1425d20c-1dde-4d40-b44e-490af34d6f0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6bf011-89fc49d7c-cbp7x" Jul 09 16:49:57.664160 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:57.664134 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1425d20c-1dde-4d40-b44e-490af34d6f0c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-6bf011-89fc49d7c-cbp7x\" (UID: \"1425d20c-1dde-4d40-b44e-490af34d6f0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6bf011-89fc49d7c-cbp7x" Jul 09 16:49:57.665924 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:57.665900 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1425d20c-1dde-4d40-b44e-490af34d6f0c-maas-api-tls\") pod \"maas-api-e2e-shared-b-6bf011-89fc49d7c-cbp7x\" (UID: \"1425d20c-1dde-4d40-b44e-490af34d6f0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6bf011-89fc49d7c-cbp7x" Jul 09 16:49:57.671577 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:57.671549 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qg5l\" (UniqueName: \"kubernetes.io/projected/1425d20c-1dde-4d40-b44e-490af34d6f0c-kube-api-access-7qg5l\") pod \"maas-api-e2e-shared-b-6bf011-89fc49d7c-cbp7x\" (UID: \"1425d20c-1dde-4d40-b44e-490af34d6f0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6bf011-89fc49d7c-cbp7x" Jul 09 16:49:57.798343 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:57.798237 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6bf011-89fc49d7c-cbp7x" Jul 09 16:49:57.942786 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:57.942752 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-6bf011-89fc49d7c-cbp7x"] Jul 09 16:49:57.944810 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:49:57.944781 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1425d20c_1dde_4d40_b44e_490af34d6f0c.slice/crio-6b31cc4f3c016b5d89d43489929136cbf3e741a440d4c5be433062fe41662760 WatchSource:0}: Error finding container 6b31cc4f3c016b5d89d43489929136cbf3e741a440d4c5be433062fe41662760: Status 404 returned error can't find the container with id 6b31cc4f3c016b5d89d43489929136cbf3e741a440d4c5be433062fe41662760 Jul 09 16:49:58.547115 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:58.547077 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6bf011-89fc49d7c-cbp7x" event={"ID":"1425d20c-1dde-4d40-b44e-490af34d6f0c","Type":"ContainerStarted","Data":"ebb7bf06497de4907fb130168be50d3f2eded0370052aa202e3f88cbd1cd5a77"} Jul 09 16:49:58.547338 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:58.547142 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6bf011-89fc49d7c-cbp7x" event={"ID":"1425d20c-1dde-4d40-b44e-490af34d6f0c","Type":"ContainerStarted","Data":"6b31cc4f3c016b5d89d43489929136cbf3e741a440d4c5be433062fe41662760"} Jul 09 16:49:58.547338 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:58.547212 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6bf011-89fc49d7c-cbp7x" Jul 09 16:49:58.565088 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:49:58.565031 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6bf011-89fc49d7c-cbp7x" podStartSLOduration=1.13156262 podStartE2EDuration="1.565015s" totalImagesPullingTime="433.45238ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:49:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:49:58.563779431 +0000 UTC m=+2083.756885824" watchObservedRunningTime="2026-07-09 16:49:58.565015 +0000 UTC m=+2083.758121378" Jul 09 16:50:04.558755 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:50:04.558723 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6bf011-89fc49d7c-cbp7x" Jul 09 16:50:15.417426 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:50:15.417397 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-nzrk9_a284ea64-610c-40f7-837b-fa0975c0a290/console-operator/2.log" Jul 09 16:50:15.424824 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:50:15.424799 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-nzrk9_a284ea64-610c-40f7-837b-fa0975c0a290/console-operator/2.log" Jul 09 16:50:15.425195 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:50:15.425175 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-49sqd_17313664-7a0f-472b-a597-39f8c5412bc3/ovn-acl-logging/0.log" Jul 09 16:50:15.432580 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:50:15.432560 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-49sqd_17313664-7a0f-472b-a597-39f8c5412bc3/ovn-acl-logging/0.log" Jul 09 16:50:15.782307 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:50:15.782232 2569 scope.go:117] "RemoveContainer" containerID="e976c9c42ee7d02373ab69f136fb50e4f13cf9bc22ee0926fc41937f18fe4fc4" Jul 09 16:50:27.586695 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:50:27.586649 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-bef56b/auth-test-model-bef56b-kserve-86c6cbfbc-46snn"] Jul 09 16:50:27.590401 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:50:27.590377 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-bef56b/auth-test-model-bef56b-kserve-86c6cbfbc-46snn" Jul 09 16:50:27.594390 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:50:27.594361 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-bef56b\"/\"default-dockercfg-vj8gl\"" Jul 09 16:50:27.594521 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:50:27.594434 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-bef56b\"/\"auth-test-model-bef56b-kserve-self-signed-certs\"" Jul 09 16:50:27.594521 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:50:27.594369 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-bef56b\"/\"openshift-service-ca.crt\"" Jul 09 16:50:27.594521 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:50:27.594361 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-bef56b\"/\"kube-root-ca.crt\"" Jul 09 16:50:27.602292 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:50:27.602267 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-bef56b/auth-test-model-bef56b-kserve-86c6cbfbc-46snn"] Jul 09 16:50:27.637908 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:50:27.637844 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31120de2-7376-4532-8a95-b2c8f1e2eddb-tls-certs\") pod \"auth-test-model-bef56b-kserve-86c6cbfbc-46snn\" (UID: \"31120de2-7376-4532-8a95-b2c8f1e2eddb\") " pod="ai-tenant-e2e-shared-a-bef56b/auth-test-model-bef56b-kserve-86c6cbfbc-46snn" Jul 09 16:50:27.637908 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:50:27.637913 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31120de2-7376-4532-8a95-b2c8f1e2eddb-model-cache\") pod \"auth-test-model-bef56b-kserve-86c6cbfbc-46snn\" (UID: \"31120de2-7376-4532-8a95-b2c8f1e2eddb\") " pod="ai-tenant-e2e-shared-a-bef56b/auth-test-model-bef56b-kserve-86c6cbfbc-46snn" Jul 09 16:50:27.638181 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:50:27.637938 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31120de2-7376-4532-8a95-b2c8f1e2eddb-tmp-dir\") pod \"auth-test-model-bef56b-kserve-86c6cbfbc-46snn\" (UID: \"31120de2-7376-4532-8a95-b2c8f1e2eddb\") " pod="ai-tenant-e2e-shared-a-bef56b/auth-test-model-bef56b-kserve-86c6cbfbc-46snn" Jul 09 16:50:27.638181 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:50:27.637967 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31120de2-7376-4532-8a95-b2c8f1e2eddb-home\") pod \"auth-test-model-bef56b-kserve-86c6cbfbc-46snn\" (UID: \"31120de2-7376-4532-8a95-b2c8f1e2eddb\") " pod="ai-tenant-e2e-shared-a-bef56b/auth-test-model-bef56b-kserve-86c6cbfbc-46snn" Jul 09 16:50:27.638181 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:50:27.637996 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmfgt\" (UniqueName: \"kubernetes.io/projected/31120de2-7376-4532-8a95-b2c8f1e2eddb-kube-api-access-bmfgt\") pod \"auth-test-model-bef56b-kserve-86c6cbfbc-46snn\" (UID: \"31120de2-7376-4532-8a95-b2c8f1e2eddb\") " pod="ai-tenant-e2e-shared-a-bef56b/auth-test-model-bef56b-kserve-86c6cbfbc-46snn" Jul 09 16:50:27.638181 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:50:27.638120 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31120de2-7376-4532-8a95-b2c8f1e2eddb-dshm\") pod \"auth-test-model-bef56b-kserve-86c6cbfbc-46snn\" (UID: \"31120de2-7376-4532-8a95-b2c8f1e2eddb\") " pod="ai-tenant-e2e-shared-a-bef56b/auth-test-model-bef56b-kserve-86c6cbfbc-46snn" Jul 09 16:50:27.739133 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:50:27.739101 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31120de2-7376-4532-8a95-b2c8f1e2eddb-dshm\") pod \"auth-test-model-bef56b-kserve-86c6cbfbc-46snn\" (UID: \"31120de2-7376-4532-8a95-b2c8f1e2eddb\") " pod="ai-tenant-e2e-shared-a-bef56b/auth-test-model-bef56b-kserve-86c6cbfbc-46snn" Jul 09 16:50:27.739344 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:50:27.739163 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31120de2-7376-4532-8a95-b2c8f1e2eddb-tls-certs\") pod \"auth-test-model-bef56b-kserve-86c6cbfbc-46snn\" (UID: \"31120de2-7376-4532-8a95-b2c8f1e2eddb\") " pod="ai-tenant-e2e-shared-a-bef56b/auth-test-model-bef56b-kserve-86c6cbfbc-46snn" Jul 09 16:50:27.739344 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:50:27.739189 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31120de2-7376-4532-8a95-b2c8f1e2eddb-model-cache\") pod \"auth-test-model-bef56b-kserve-86c6cbfbc-46snn\" (UID: \"31120de2-7376-4532-8a95-b2c8f1e2eddb\") " pod="ai-tenant-e2e-shared-a-bef56b/auth-test-model-bef56b-kserve-86c6cbfbc-46snn" Jul 09 16:50:27.739344 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:50:27.739218 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31120de2-7376-4532-8a95-b2c8f1e2eddb-tmp-dir\") pod \"auth-test-model-bef56b-kserve-86c6cbfbc-46snn\" (UID: \"31120de2-7376-4532-8a95-b2c8f1e2eddb\") " pod="ai-tenant-e2e-shared-a-bef56b/auth-test-model-bef56b-kserve-86c6cbfbc-46snn" Jul 09 16:50:27.739344 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:50:27.739260 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31120de2-7376-4532-8a95-b2c8f1e2eddb-home\") pod \"auth-test-model-bef56b-kserve-86c6cbfbc-46snn\" (UID: \"31120de2-7376-4532-8a95-b2c8f1e2eddb\") " pod="ai-tenant-e2e-shared-a-bef56b/auth-test-model-bef56b-kserve-86c6cbfbc-46snn" Jul 09 16:50:27.739344 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:50:27.739299 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmfgt\" (UniqueName: \"kubernetes.io/projected/31120de2-7376-4532-8a95-b2c8f1e2eddb-kube-api-access-bmfgt\") pod \"auth-test-model-bef56b-kserve-86c6cbfbc-46snn\" (UID: \"31120de2-7376-4532-8a95-b2c8f1e2eddb\") " pod="ai-tenant-e2e-shared-a-bef56b/auth-test-model-bef56b-kserve-86c6cbfbc-46snn" Jul 09 16:50:27.739638 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:50:27.739610 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31120de2-7376-4532-8a95-b2c8f1e2eddb-model-cache\") pod \"auth-test-model-bef56b-kserve-86c6cbfbc-46snn\" (UID: \"31120de2-7376-4532-8a95-b2c8f1e2eddb\") " pod="ai-tenant-e2e-shared-a-bef56b/auth-test-model-bef56b-kserve-86c6cbfbc-46snn" Jul 09 16:50:27.739698 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:50:27.739644 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31120de2-7376-4532-8a95-b2c8f1e2eddb-tmp-dir\") pod \"auth-test-model-bef56b-kserve-86c6cbfbc-46snn\" (UID: \"31120de2-7376-4532-8a95-b2c8f1e2eddb\") " pod="ai-tenant-e2e-shared-a-bef56b/auth-test-model-bef56b-kserve-86c6cbfbc-46snn" Jul 09 16:50:27.739698 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:50:27.739683 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31120de2-7376-4532-8a95-b2c8f1e2eddb-home\") pod \"auth-test-model-bef56b-kserve-86c6cbfbc-46snn\" (UID: \"31120de2-7376-4532-8a95-b2c8f1e2eddb\") " pod="ai-tenant-e2e-shared-a-bef56b/auth-test-model-bef56b-kserve-86c6cbfbc-46snn" Jul 09 16:50:27.741480 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:50:27.741460 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31120de2-7376-4532-8a95-b2c8f1e2eddb-dshm\") pod \"auth-test-model-bef56b-kserve-86c6cbfbc-46snn\" (UID: \"31120de2-7376-4532-8a95-b2c8f1e2eddb\") " pod="ai-tenant-e2e-shared-a-bef56b/auth-test-model-bef56b-kserve-86c6cbfbc-46snn" Jul 09 16:50:27.741896 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:50:27.741857 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31120de2-7376-4532-8a95-b2c8f1e2eddb-tls-certs\") pod \"auth-test-model-bef56b-kserve-86c6cbfbc-46snn\" (UID: \"31120de2-7376-4532-8a95-b2c8f1e2eddb\") " pod="ai-tenant-e2e-shared-a-bef56b/auth-test-model-bef56b-kserve-86c6cbfbc-46snn" Jul 09 16:50:27.747066 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:50:27.747041 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmfgt\" (UniqueName: \"kubernetes.io/projected/31120de2-7376-4532-8a95-b2c8f1e2eddb-kube-api-access-bmfgt\") pod \"auth-test-model-bef56b-kserve-86c6cbfbc-46snn\" (UID: \"31120de2-7376-4532-8a95-b2c8f1e2eddb\") " pod="ai-tenant-e2e-shared-a-bef56b/auth-test-model-bef56b-kserve-86c6cbfbc-46snn" Jul 09 16:50:27.900462 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:50:27.900418 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-bef56b/auth-test-model-bef56b-kserve-86c6cbfbc-46snn" Jul 09 16:50:28.045927 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:50:28.045897 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-bef56b/auth-test-model-bef56b-kserve-86c6cbfbc-46snn"] Jul 09 16:50:28.047790 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:50:28.047760 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod31120de2_7376_4532_8a95_b2c8f1e2eddb.slice/crio-123d01927f8a63a41bb84842bbf2148ac4f722632d5d35897cdc10cb30ad001e WatchSource:0}: Error finding container 123d01927f8a63a41bb84842bbf2148ac4f722632d5d35897cdc10cb30ad001e: Status 404 returned error can't find the container with id 123d01927f8a63a41bb84842bbf2148ac4f722632d5d35897cdc10cb30ad001e Jul 09 16:50:28.697202 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:50:28.697161 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-bef56b/auth-test-model-bef56b-kserve-86c6cbfbc-46snn" event={"ID":"31120de2-7376-4532-8a95-b2c8f1e2eddb","Type":"ContainerStarted","Data":"4a42dbe679704fe24d4b2841a129e65c2b3b34847552127bc46c274dbb486199"} Jul 09 16:50:28.697672 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:50:28.697212 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-bef56b/auth-test-model-bef56b-kserve-86c6cbfbc-46snn" event={"ID":"31120de2-7376-4532-8a95-b2c8f1e2eddb","Type":"ContainerStarted","Data":"123d01927f8a63a41bb84842bbf2148ac4f722632d5d35897cdc10cb30ad001e"} Jul 09 16:50:28.716365 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:50:28.716304 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-bef56b/auth-test-model-bef56b-kserve-86c6cbfbc-46snn" podStartSLOduration=1.716281757 podStartE2EDuration="1.716281757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:50:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:50:28.714323588 +0000 UTC m=+2113.907429969" watchObservedRunningTime="2026-07-09 16:50:28.716281757 +0000 UTC m=+2113.909388138" Jul 09 16:50:37.900599 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:50:37.900549 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-bef56b/auth-test-model-bef56b-kserve-86c6cbfbc-46snn" Jul 09 16:50:37.901099 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:50:37.900637 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-bef56b/auth-test-model-bef56b-kserve-86c6cbfbc-46snn" Jul 09 16:50:37.913641 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:50:37.913613 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-bef56b/auth-test-model-bef56b-kserve-86c6cbfbc-46snn" Jul 09 16:50:38.766446 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:50:38.766409 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-bef56b/auth-test-model-bef56b-kserve-86c6cbfbc-46snn" Jul 09 16:52:46.779734 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:52:46.779651 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-bef56b/auth-test-model-bef56b-kserve-86c6cbfbc-46snn"] Jul 09 16:52:57.694812 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:52:57.694776 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-bef56b/sub-test-model-bef56b-kserve-84b8cf8cb7-tdsc6"] Jul 09 16:52:57.698427 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:52:57.698410 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-bef56b/sub-test-model-bef56b-kserve-84b8cf8cb7-tdsc6" Jul 09 16:52:57.701223 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:52:57.701198 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-bef56b\"/\"sub-test-model-bef56b-kserve-self-signed-certs\"" Jul 09 16:52:57.708236 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:52:57.708214 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-bef56b/sub-test-model-bef56b-kserve-84b8cf8cb7-tdsc6"] Jul 09 16:52:57.810734 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:52:57.810698 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw4h6\" (UniqueName: \"kubernetes.io/projected/2fe719d7-53dc-45f8-a4e6-3d95d867e0d8-kube-api-access-zw4h6\") pod \"sub-test-model-bef56b-kserve-84b8cf8cb7-tdsc6\" (UID: \"2fe719d7-53dc-45f8-a4e6-3d95d867e0d8\") " pod="ai-tenant-e2e-shared-a-bef56b/sub-test-model-bef56b-kserve-84b8cf8cb7-tdsc6" Jul 09 16:52:57.810951 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:52:57.810782 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fe719d7-53dc-45f8-a4e6-3d95d867e0d8-dshm\") pod \"sub-test-model-bef56b-kserve-84b8cf8cb7-tdsc6\" (UID: \"2fe719d7-53dc-45f8-a4e6-3d95d867e0d8\") " pod="ai-tenant-e2e-shared-a-bef56b/sub-test-model-bef56b-kserve-84b8cf8cb7-tdsc6" Jul 09 16:52:57.810951 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:52:57.810806 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fe719d7-53dc-45f8-a4e6-3d95d867e0d8-model-cache\") pod \"sub-test-model-bef56b-kserve-84b8cf8cb7-tdsc6\" (UID: \"2fe719d7-53dc-45f8-a4e6-3d95d867e0d8\") " pod="ai-tenant-e2e-shared-a-bef56b/sub-test-model-bef56b-kserve-84b8cf8cb7-tdsc6" Jul 09 16:52:57.810951 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:52:57.810827 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fe719d7-53dc-45f8-a4e6-3d95d867e0d8-home\") pod \"sub-test-model-bef56b-kserve-84b8cf8cb7-tdsc6\" (UID: \"2fe719d7-53dc-45f8-a4e6-3d95d867e0d8\") " pod="ai-tenant-e2e-shared-a-bef56b/sub-test-model-bef56b-kserve-84b8cf8cb7-tdsc6" Jul 09 16:52:57.810951 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:52:57.810860 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fe719d7-53dc-45f8-a4e6-3d95d867e0d8-tls-certs\") pod \"sub-test-model-bef56b-kserve-84b8cf8cb7-tdsc6\" (UID: \"2fe719d7-53dc-45f8-a4e6-3d95d867e0d8\") " pod="ai-tenant-e2e-shared-a-bef56b/sub-test-model-bef56b-kserve-84b8cf8cb7-tdsc6" Jul 09 16:52:57.810951 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:52:57.810896 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fe719d7-53dc-45f8-a4e6-3d95d867e0d8-tmp-dir\") pod \"sub-test-model-bef56b-kserve-84b8cf8cb7-tdsc6\" (UID: \"2fe719d7-53dc-45f8-a4e6-3d95d867e0d8\") " pod="ai-tenant-e2e-shared-a-bef56b/sub-test-model-bef56b-kserve-84b8cf8cb7-tdsc6" Jul 09 16:52:57.911583 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:52:57.911544 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fe719d7-53dc-45f8-a4e6-3d95d867e0d8-dshm\") pod \"sub-test-model-bef56b-kserve-84b8cf8cb7-tdsc6\" (UID: \"2fe719d7-53dc-45f8-a4e6-3d95d867e0d8\") " pod="ai-tenant-e2e-shared-a-bef56b/sub-test-model-bef56b-kserve-84b8cf8cb7-tdsc6" Jul 09 16:52:57.911763 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:52:57.911589 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fe719d7-53dc-45f8-a4e6-3d95d867e0d8-model-cache\") pod \"sub-test-model-bef56b-kserve-84b8cf8cb7-tdsc6\" (UID: \"2fe719d7-53dc-45f8-a4e6-3d95d867e0d8\") " pod="ai-tenant-e2e-shared-a-bef56b/sub-test-model-bef56b-kserve-84b8cf8cb7-tdsc6" Jul 09 16:52:57.911763 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:52:57.911629 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fe719d7-53dc-45f8-a4e6-3d95d867e0d8-home\") pod \"sub-test-model-bef56b-kserve-84b8cf8cb7-tdsc6\" (UID: \"2fe719d7-53dc-45f8-a4e6-3d95d867e0d8\") " pod="ai-tenant-e2e-shared-a-bef56b/sub-test-model-bef56b-kserve-84b8cf8cb7-tdsc6" Jul 09 16:52:57.911763 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:52:57.911669 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fe719d7-53dc-45f8-a4e6-3d95d867e0d8-tls-certs\") pod \"sub-test-model-bef56b-kserve-84b8cf8cb7-tdsc6\" (UID: \"2fe719d7-53dc-45f8-a4e6-3d95d867e0d8\") " pod="ai-tenant-e2e-shared-a-bef56b/sub-test-model-bef56b-kserve-84b8cf8cb7-tdsc6" Jul 09 16:52:57.911763 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:52:57.911703 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fe719d7-53dc-45f8-a4e6-3d95d867e0d8-tmp-dir\") pod \"sub-test-model-bef56b-kserve-84b8cf8cb7-tdsc6\" (UID: \"2fe719d7-53dc-45f8-a4e6-3d95d867e0d8\") " pod="ai-tenant-e2e-shared-a-bef56b/sub-test-model-bef56b-kserve-84b8cf8cb7-tdsc6" Jul 09 16:52:57.911763 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:52:57.911752 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw4h6\" (UniqueName: \"kubernetes.io/projected/2fe719d7-53dc-45f8-a4e6-3d95d867e0d8-kube-api-access-zw4h6\") pod \"sub-test-model-bef56b-kserve-84b8cf8cb7-tdsc6\" (UID: \"2fe719d7-53dc-45f8-a4e6-3d95d867e0d8\") " pod="ai-tenant-e2e-shared-a-bef56b/sub-test-model-bef56b-kserve-84b8cf8cb7-tdsc6" Jul 09 16:52:57.912157 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:52:57.912131 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fe719d7-53dc-45f8-a4e6-3d95d867e0d8-model-cache\") pod \"sub-test-model-bef56b-kserve-84b8cf8cb7-tdsc6\" (UID: \"2fe719d7-53dc-45f8-a4e6-3d95d867e0d8\") " pod="ai-tenant-e2e-shared-a-bef56b/sub-test-model-bef56b-kserve-84b8cf8cb7-tdsc6" Jul 09 16:52:57.912267 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:52:57.912183 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fe719d7-53dc-45f8-a4e6-3d95d867e0d8-tmp-dir\") pod \"sub-test-model-bef56b-kserve-84b8cf8cb7-tdsc6\" (UID: \"2fe719d7-53dc-45f8-a4e6-3d95d867e0d8\") " pod="ai-tenant-e2e-shared-a-bef56b/sub-test-model-bef56b-kserve-84b8cf8cb7-tdsc6" Jul 09 16:52:57.912267 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:52:57.912140 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fe719d7-53dc-45f8-a4e6-3d95d867e0d8-home\") pod \"sub-test-model-bef56b-kserve-84b8cf8cb7-tdsc6\" (UID: \"2fe719d7-53dc-45f8-a4e6-3d95d867e0d8\") " pod="ai-tenant-e2e-shared-a-bef56b/sub-test-model-bef56b-kserve-84b8cf8cb7-tdsc6" Jul 09 16:52:57.914112 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:52:57.914083 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fe719d7-53dc-45f8-a4e6-3d95d867e0d8-dshm\") pod \"sub-test-model-bef56b-kserve-84b8cf8cb7-tdsc6\" (UID: \"2fe719d7-53dc-45f8-a4e6-3d95d867e0d8\") " pod="ai-tenant-e2e-shared-a-bef56b/sub-test-model-bef56b-kserve-84b8cf8cb7-tdsc6" Jul 09 16:52:57.914250 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:52:57.914190 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fe719d7-53dc-45f8-a4e6-3d95d867e0d8-tls-certs\") pod \"sub-test-model-bef56b-kserve-84b8cf8cb7-tdsc6\" (UID: \"2fe719d7-53dc-45f8-a4e6-3d95d867e0d8\") " pod="ai-tenant-e2e-shared-a-bef56b/sub-test-model-bef56b-kserve-84b8cf8cb7-tdsc6" Jul 09 16:52:57.920451 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:52:57.920427 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw4h6\" (UniqueName: \"kubernetes.io/projected/2fe719d7-53dc-45f8-a4e6-3d95d867e0d8-kube-api-access-zw4h6\") pod \"sub-test-model-bef56b-kserve-84b8cf8cb7-tdsc6\" (UID: \"2fe719d7-53dc-45f8-a4e6-3d95d867e0d8\") " pod="ai-tenant-e2e-shared-a-bef56b/sub-test-model-bef56b-kserve-84b8cf8cb7-tdsc6" Jul 09 16:52:58.008862 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:52:58.008760 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-bef56b/sub-test-model-bef56b-kserve-84b8cf8cb7-tdsc6" Jul 09 16:52:58.149702 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:52:58.149674 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-bef56b/sub-test-model-bef56b-kserve-84b8cf8cb7-tdsc6"] Jul 09 16:52:58.151989 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:52:58.151957 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2fe719d7_53dc_45f8_a4e6_3d95d867e0d8.slice/crio-bc989e1cdbe6797f5771d5225e25d16beecdf7e82abb0d72ea81e539b10dc3e3 WatchSource:0}: Error finding container bc989e1cdbe6797f5771d5225e25d16beecdf7e82abb0d72ea81e539b10dc3e3: Status 404 returned error can't find the container with id bc989e1cdbe6797f5771d5225e25d16beecdf7e82abb0d72ea81e539b10dc3e3 Jul 09 16:52:58.457242 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:52:58.457209 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-bef56b/sub-test-model-bef56b-kserve-84b8cf8cb7-tdsc6" event={"ID":"2fe719d7-53dc-45f8-a4e6-3d95d867e0d8","Type":"ContainerStarted","Data":"293c90de113dbbc8c81404489abffb787fe59b2cf21659ba80a9b5a90f9d53eb"} Jul 09 16:52:58.457436 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:52:58.457254 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-bef56b/sub-test-model-bef56b-kserve-84b8cf8cb7-tdsc6" event={"ID":"2fe719d7-53dc-45f8-a4e6-3d95d867e0d8","Type":"ContainerStarted","Data":"bc989e1cdbe6797f5771d5225e25d16beecdf7e82abb0d72ea81e539b10dc3e3"} Jul 09 16:52:58.474316 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:52:58.474262 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-bef56b/sub-test-model-bef56b-kserve-84b8cf8cb7-tdsc6" podStartSLOduration=1.474245587 podStartE2EDuration="1.474245587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:52:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:52:58.473618312 +0000 UTC m=+2263.666724695" watchObservedRunningTime="2026-07-09 16:52:58.474245587 +0000 UTC m=+2263.667351968" Jul 09 16:53:01.800283 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:53:01.800231 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-bef56b/auth-test-model-bef56b-kserve-86c6cbfbc-46snn" podUID="31120de2-7376-4532-8a95-b2c8f1e2eddb" containerName="main" containerID="cri-o://4a42dbe679704fe24d4b2841a129e65c2b3b34847552127bc46c274dbb486199" gracePeriod=45 Jul 09 16:53:02.064098 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:53:02.064022 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-bef56b/auth-test-model-bef56b-kserve-86c6cbfbc-46snn" Jul 09 16:53:02.147553 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:53:02.147517 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31120de2-7376-4532-8a95-b2c8f1e2eddb-home\") pod \"31120de2-7376-4532-8a95-b2c8f1e2eddb\" (UID: \"31120de2-7376-4532-8a95-b2c8f1e2eddb\") " Jul 09 16:53:02.147757 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:53:02.147570 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31120de2-7376-4532-8a95-b2c8f1e2eddb-tls-certs\") pod \"31120de2-7376-4532-8a95-b2c8f1e2eddb\" (UID: \"31120de2-7376-4532-8a95-b2c8f1e2eddb\") " Jul 09 16:53:02.147757 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:53:02.147609 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31120de2-7376-4532-8a95-b2c8f1e2eddb-dshm\") pod \"31120de2-7376-4532-8a95-b2c8f1e2eddb\" (UID: \"31120de2-7376-4532-8a95-b2c8f1e2eddb\") " Jul 09 16:53:02.147757 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:53:02.147641 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bmfgt\" (UniqueName: \"kubernetes.io/projected/31120de2-7376-4532-8a95-b2c8f1e2eddb-kube-api-access-bmfgt\") pod \"31120de2-7376-4532-8a95-b2c8f1e2eddb\" (UID: \"31120de2-7376-4532-8a95-b2c8f1e2eddb\") " Jul 09 16:53:02.147757 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:53:02.147670 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31120de2-7376-4532-8a95-b2c8f1e2eddb-model-cache\") pod \"31120de2-7376-4532-8a95-b2c8f1e2eddb\" (UID: \"31120de2-7376-4532-8a95-b2c8f1e2eddb\") " Jul 09 16:53:02.147757 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:53:02.147693 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31120de2-7376-4532-8a95-b2c8f1e2eddb-tmp-dir\") pod \"31120de2-7376-4532-8a95-b2c8f1e2eddb\" (UID: \"31120de2-7376-4532-8a95-b2c8f1e2eddb\") " Jul 09 16:53:02.148101 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:53:02.147844 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31120de2-7376-4532-8a95-b2c8f1e2eddb-home" (OuterVolumeSpecName: "home") pod "31120de2-7376-4532-8a95-b2c8f1e2eddb" (UID: "31120de2-7376-4532-8a95-b2c8f1e2eddb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:53:02.148101 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:53:02.147961 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31120de2-7376-4532-8a95-b2c8f1e2eddb-model-cache" (OuterVolumeSpecName: "model-cache") pod "31120de2-7376-4532-8a95-b2c8f1e2eddb" (UID: "31120de2-7376-4532-8a95-b2c8f1e2eddb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:53:02.148101 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:53:02.148046 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31120de2-7376-4532-8a95-b2c8f1e2eddb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "31120de2-7376-4532-8a95-b2c8f1e2eddb" (UID: "31120de2-7376-4532-8a95-b2c8f1e2eddb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:53:02.148101 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:53:02.148077 2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31120de2-7376-4532-8a95-b2c8f1e2eddb-home\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:53:02.148101 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:53:02.148102 2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31120de2-7376-4532-8a95-b2c8f1e2eddb-model-cache\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:53:02.150043 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:53:02.150009 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31120de2-7376-4532-8a95-b2c8f1e2eddb-kube-api-access-bmfgt" (OuterVolumeSpecName: "kube-api-access-bmfgt") pod "31120de2-7376-4532-8a95-b2c8f1e2eddb" (UID: "31120de2-7376-4532-8a95-b2c8f1e2eddb"). InnerVolumeSpecName "kube-api-access-bmfgt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 16:53:02.150193 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:53:02.150060 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31120de2-7376-4532-8a95-b2c8f1e2eddb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "31120de2-7376-4532-8a95-b2c8f1e2eddb" (UID: "31120de2-7376-4532-8a95-b2c8f1e2eddb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 16:53:02.150193 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:53:02.150071 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31120de2-7376-4532-8a95-b2c8f1e2eddb-dshm" (OuterVolumeSpecName: "dshm") pod "31120de2-7376-4532-8a95-b2c8f1e2eddb" (UID: "31120de2-7376-4532-8a95-b2c8f1e2eddb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:53:02.249074 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:53:02.249021 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31120de2-7376-4532-8a95-b2c8f1e2eddb-tls-certs\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:53:02.249074 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:53:02.249066 2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31120de2-7376-4532-8a95-b2c8f1e2eddb-dshm\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:53:02.249074 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:53:02.249076 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmfgt\" (UniqueName: \"kubernetes.io/projected/31120de2-7376-4532-8a95-b2c8f1e2eddb-kube-api-access-bmfgt\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:53:02.249074 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:53:02.249087 2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31120de2-7376-4532-8a95-b2c8f1e2eddb-tmp-dir\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:53:02.479041 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:53:02.479001 2569 generic.go:356] "Generic (PLEG): container finished" podID="31120de2-7376-4532-8a95-b2c8f1e2eddb" containerID="4a42dbe679704fe24d4b2841a129e65c2b3b34847552127bc46c274dbb486199" exitCode=0 Jul 09 16:53:02.479219 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:53:02.479044 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-bef56b/auth-test-model-bef56b-kserve-86c6cbfbc-46snn" Jul 09 16:53:02.479219 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:53:02.479096 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-bef56b/auth-test-model-bef56b-kserve-86c6cbfbc-46snn" event={"ID":"31120de2-7376-4532-8a95-b2c8f1e2eddb","Type":"ContainerDied","Data":"4a42dbe679704fe24d4b2841a129e65c2b3b34847552127bc46c274dbb486199"} Jul 09 16:53:02.479219 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:53:02.479139 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-bef56b/auth-test-model-bef56b-kserve-86c6cbfbc-46snn" event={"ID":"31120de2-7376-4532-8a95-b2c8f1e2eddb","Type":"ContainerDied","Data":"123d01927f8a63a41bb84842bbf2148ac4f722632d5d35897cdc10cb30ad001e"} Jul 09 16:53:02.479219 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:53:02.479161 2569 scope.go:117] "RemoveContainer" containerID="4a42dbe679704fe24d4b2841a129e65c2b3b34847552127bc46c274dbb486199" Jul 09 16:53:02.491495 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:53:02.491475 2569 scope.go:117] "RemoveContainer" containerID="4a42dbe679704fe24d4b2841a129e65c2b3b34847552127bc46c274dbb486199" Jul 09 16:53:02.491841 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:53:02.491822 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a42dbe679704fe24d4b2841a129e65c2b3b34847552127bc46c274dbb486199\": container with ID starting with 4a42dbe679704fe24d4b2841a129e65c2b3b34847552127bc46c274dbb486199 not found: ID does not exist" containerID="4a42dbe679704fe24d4b2841a129e65c2b3b34847552127bc46c274dbb486199" Jul 09 16:53:02.491959 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:53:02.491851 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a42dbe679704fe24d4b2841a129e65c2b3b34847552127bc46c274dbb486199"} err="failed to get container status \"4a42dbe679704fe24d4b2841a129e65c2b3b34847552127bc46c274dbb486199\": rpc error: code = NotFound desc = could not find container \"4a42dbe679704fe24d4b2841a129e65c2b3b34847552127bc46c274dbb486199\": container with ID starting with 4a42dbe679704fe24d4b2841a129e65c2b3b34847552127bc46c274dbb486199 not found: ID does not exist" Jul 09 16:53:02.501678 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:53:02.501646 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-bef56b/auth-test-model-bef56b-kserve-86c6cbfbc-46snn"] Jul 09 16:53:02.505156 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:53:02.505131 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-bef56b/auth-test-model-bef56b-kserve-86c6cbfbc-46snn"] Jul 09 16:53:03.390582 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:53:03.390546 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31120de2-7376-4532-8a95-b2c8f1e2eddb" path="/var/lib/kubelet/pods/31120de2-7376-4532-8a95-b2c8f1e2eddb/volumes" Jul 09 16:53:08.009173 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:53:08.009124 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-bef56b/sub-test-model-bef56b-kserve-84b8cf8cb7-tdsc6" Jul 09 16:53:08.009173 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:53:08.009183 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-bef56b/sub-test-model-bef56b-kserve-84b8cf8cb7-tdsc6" Jul 09 16:53:08.021970 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:53:08.021942 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-bef56b/sub-test-model-bef56b-kserve-84b8cf8cb7-tdsc6" Jul 09 16:53:08.520980 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:53:08.520955 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-bef56b/sub-test-model-bef56b-kserve-84b8cf8cb7-tdsc6" Jul 09 16:53:38.085802 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:53:38.085759 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-6bf011/sub-test-model-6bf011-kserve-6cbb7fcf95-ztkbk"] Jul 09 16:53:38.086464 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:53:38.086438 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31120de2-7376-4532-8a95-b2c8f1e2eddb" containerName="main" Jul 09 16:53:38.086464 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:53:38.086466 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="31120de2-7376-4532-8a95-b2c8f1e2eddb" containerName="main" Jul 09 16:53:38.086585 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:53:38.086573 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="31120de2-7376-4532-8a95-b2c8f1e2eddb" containerName="main" Jul 09 16:53:38.090260 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:53:38.090236 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-6bf011/sub-test-model-6bf011-kserve-6cbb7fcf95-ztkbk" Jul 09 16:53:38.092920 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:53:38.092890 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-6bf011\"/\"openshift-service-ca.crt\"" Jul 09 16:53:38.094345 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:53:38.094323 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-6bf011\"/\"default-dockercfg-ljp6q\"" Jul 09 16:53:38.094476 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:53:38.094401 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-6bf011\"/\"sub-test-model-6bf011-kserve-self-signed-certs\"" Jul 09 16:53:38.094476 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:53:38.094446 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-6bf011\"/\"kube-root-ca.crt\"" Jul 09 16:53:38.097733 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:53:38.097707 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-6bf011/sub-test-model-6bf011-kserve-6cbb7fcf95-ztkbk"] Jul 09 16:53:38.178503 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:53:38.178465 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pt4t\" (UniqueName: \"kubernetes.io/projected/6641aeb0-90c9-4902-b469-9a87552c5e79-kube-api-access-2pt4t\") pod \"sub-test-model-6bf011-kserve-6cbb7fcf95-ztkbk\" (UID: \"6641aeb0-90c9-4902-b469-9a87552c5e79\") " pod="ai-tenant-e2e-shared-b-6bf011/sub-test-model-6bf011-kserve-6cbb7fcf95-ztkbk" Jul 09 16:53:38.178690 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:53:38.178519 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6641aeb0-90c9-4902-b469-9a87552c5e79-tmp-dir\") pod \"sub-test-model-6bf011-kserve-6cbb7fcf95-ztkbk\" (UID: \"6641aeb0-90c9-4902-b469-9a87552c5e79\") " pod="ai-tenant-e2e-shared-b-6bf011/sub-test-model-6bf011-kserve-6cbb7fcf95-ztkbk" Jul 09 16:53:38.178690 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:53:38.178605 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6641aeb0-90c9-4902-b469-9a87552c5e79-model-cache\") pod \"sub-test-model-6bf011-kserve-6cbb7fcf95-ztkbk\" (UID: \"6641aeb0-90c9-4902-b469-9a87552c5e79\") " pod="ai-tenant-e2e-shared-b-6bf011/sub-test-model-6bf011-kserve-6cbb7fcf95-ztkbk" Jul 09 16:53:38.178690 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:53:38.178648 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6641aeb0-90c9-4902-b469-9a87552c5e79-dshm\") pod \"sub-test-model-6bf011-kserve-6cbb7fcf95-ztkbk\" (UID: \"6641aeb0-90c9-4902-b469-9a87552c5e79\") " pod="ai-tenant-e2e-shared-b-6bf011/sub-test-model-6bf011-kserve-6cbb7fcf95-ztkbk" Jul 09 16:53:38.178797 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:53:38.178696 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6641aeb0-90c9-4902-b469-9a87552c5e79-home\") pod \"sub-test-model-6bf011-kserve-6cbb7fcf95-ztkbk\" (UID: \"6641aeb0-90c9-4902-b469-9a87552c5e79\") " pod="ai-tenant-e2e-shared-b-6bf011/sub-test-model-6bf011-kserve-6cbb7fcf95-ztkbk" Jul 09 16:53:38.178797 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:53:38.178750 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6641aeb0-90c9-4902-b469-9a87552c5e79-tls-certs\") pod \"sub-test-model-6bf011-kserve-6cbb7fcf95-ztkbk\" (UID: \"6641aeb0-90c9-4902-b469-9a87552c5e79\") " pod="ai-tenant-e2e-shared-b-6bf011/sub-test-model-6bf011-kserve-6cbb7fcf95-ztkbk" Jul 09 16:53:38.279696 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:53:38.279655 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6641aeb0-90c9-4902-b469-9a87552c5e79-dshm\") pod \"sub-test-model-6bf011-kserve-6cbb7fcf95-ztkbk\" (UID: \"6641aeb0-90c9-4902-b469-9a87552c5e79\") " pod="ai-tenant-e2e-shared-b-6bf011/sub-test-model-6bf011-kserve-6cbb7fcf95-ztkbk" Jul 09 16:53:38.279696 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:53:38.279703 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6641aeb0-90c9-4902-b469-9a87552c5e79-home\") pod \"sub-test-model-6bf011-kserve-6cbb7fcf95-ztkbk\" (UID: \"6641aeb0-90c9-4902-b469-9a87552c5e79\") " pod="ai-tenant-e2e-shared-b-6bf011/sub-test-model-6bf011-kserve-6cbb7fcf95-ztkbk" Jul 09 16:53:38.279984 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:53:38.279741 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6641aeb0-90c9-4902-b469-9a87552c5e79-tls-certs\") pod \"sub-test-model-6bf011-kserve-6cbb7fcf95-ztkbk\" (UID: \"6641aeb0-90c9-4902-b469-9a87552c5e79\") " pod="ai-tenant-e2e-shared-b-6bf011/sub-test-model-6bf011-kserve-6cbb7fcf95-ztkbk" Jul 09 16:53:38.279984 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:53:38.279779 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pt4t\" (UniqueName: \"kubernetes.io/projected/6641aeb0-90c9-4902-b469-9a87552c5e79-kube-api-access-2pt4t\") pod \"sub-test-model-6bf011-kserve-6cbb7fcf95-ztkbk\" (UID: \"6641aeb0-90c9-4902-b469-9a87552c5e79\") " pod="ai-tenant-e2e-shared-b-6bf011/sub-test-model-6bf011-kserve-6cbb7fcf95-ztkbk" Jul 09 16:53:38.279984 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:53:38.279829 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6641aeb0-90c9-4902-b469-9a87552c5e79-tmp-dir\") pod \"sub-test-model-6bf011-kserve-6cbb7fcf95-ztkbk\" (UID: \"6641aeb0-90c9-4902-b469-9a87552c5e79\") " pod="ai-tenant-e2e-shared-b-6bf011/sub-test-model-6bf011-kserve-6cbb7fcf95-ztkbk" Jul 09 16:53:38.279984 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:53:38.279896 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6641aeb0-90c9-4902-b469-9a87552c5e79-model-cache\") pod \"sub-test-model-6bf011-kserve-6cbb7fcf95-ztkbk\" (UID: \"6641aeb0-90c9-4902-b469-9a87552c5e79\") " pod="ai-tenant-e2e-shared-b-6bf011/sub-test-model-6bf011-kserve-6cbb7fcf95-ztkbk" Jul 09 16:53:38.280246 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:53:38.280221 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6641aeb0-90c9-4902-b469-9a87552c5e79-home\") pod \"sub-test-model-6bf011-kserve-6cbb7fcf95-ztkbk\" (UID: \"6641aeb0-90c9-4902-b469-9a87552c5e79\") " pod="ai-tenant-e2e-shared-b-6bf011/sub-test-model-6bf011-kserve-6cbb7fcf95-ztkbk" Jul 09 16:53:38.280317 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:53:38.280253 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6641aeb0-90c9-4902-b469-9a87552c5e79-model-cache\") pod \"sub-test-model-6bf011-kserve-6cbb7fcf95-ztkbk\" (UID: \"6641aeb0-90c9-4902-b469-9a87552c5e79\") " pod="ai-tenant-e2e-shared-b-6bf011/sub-test-model-6bf011-kserve-6cbb7fcf95-ztkbk" Jul 09 16:53:38.280317 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:53:38.280304 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6641aeb0-90c9-4902-b469-9a87552c5e79-tmp-dir\") pod \"sub-test-model-6bf011-kserve-6cbb7fcf95-ztkbk\" (UID: \"6641aeb0-90c9-4902-b469-9a87552c5e79\") " pod="ai-tenant-e2e-shared-b-6bf011/sub-test-model-6bf011-kserve-6cbb7fcf95-ztkbk" Jul 09 16:53:38.281958 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:53:38.281939 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6641aeb0-90c9-4902-b469-9a87552c5e79-dshm\") pod \"sub-test-model-6bf011-kserve-6cbb7fcf95-ztkbk\" (UID: \"6641aeb0-90c9-4902-b469-9a87552c5e79\") " pod="ai-tenant-e2e-shared-b-6bf011/sub-test-model-6bf011-kserve-6cbb7fcf95-ztkbk" Jul 09 16:53:38.282297 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:53:38.282279 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6641aeb0-90c9-4902-b469-9a87552c5e79-tls-certs\") pod \"sub-test-model-6bf011-kserve-6cbb7fcf95-ztkbk\" (UID: \"6641aeb0-90c9-4902-b469-9a87552c5e79\") " pod="ai-tenant-e2e-shared-b-6bf011/sub-test-model-6bf011-kserve-6cbb7fcf95-ztkbk" Jul 09 16:53:38.289031 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:53:38.289004 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pt4t\" (UniqueName: \"kubernetes.io/projected/6641aeb0-90c9-4902-b469-9a87552c5e79-kube-api-access-2pt4t\") pod \"sub-test-model-6bf011-kserve-6cbb7fcf95-ztkbk\" (UID: \"6641aeb0-90c9-4902-b469-9a87552c5e79\") " pod="ai-tenant-e2e-shared-b-6bf011/sub-test-model-6bf011-kserve-6cbb7fcf95-ztkbk" Jul 09 16:53:38.403606 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:53:38.403566 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-6bf011/sub-test-model-6bf011-kserve-6cbb7fcf95-ztkbk" Jul 09 16:53:38.546612 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:53:38.546580 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-6bf011/sub-test-model-6bf011-kserve-6cbb7fcf95-ztkbk"] Jul 09 16:53:38.548037 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:53:38.548010 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6641aeb0_90c9_4902_b469_9a87552c5e79.slice/crio-403687685ff02964beb298110cd469d6da60ef04b1b6583f45315823be5362c8 WatchSource:0}: Error finding container 403687685ff02964beb298110cd469d6da60ef04b1b6583f45315823be5362c8: Status 404 returned error can't find the container with id 403687685ff02964beb298110cd469d6da60ef04b1b6583f45315823be5362c8 Jul 09 16:53:38.657923 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:53:38.657806 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-6bf011/sub-test-model-6bf011-kserve-6cbb7fcf95-ztkbk" event={"ID":"6641aeb0-90c9-4902-b469-9a87552c5e79","Type":"ContainerStarted","Data":"b176cbafc2c0569a030a49b6e77c65b86211a2fb43c80054de5eb7501688cf88"} Jul 09 16:53:38.657923 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:53:38.657852 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-6bf011/sub-test-model-6bf011-kserve-6cbb7fcf95-ztkbk" event={"ID":"6641aeb0-90c9-4902-b469-9a87552c5e79","Type":"ContainerStarted","Data":"403687685ff02964beb298110cd469d6da60ef04b1b6583f45315823be5362c8"} Jul 09 16:53:38.676047 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:53:38.675997 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-6bf011/sub-test-model-6bf011-kserve-6cbb7fcf95-ztkbk" podStartSLOduration=0.675979555 podStartE2EDuration="675.979555ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:53:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:53:38.674203788 +0000 UTC m=+2303.867310202" watchObservedRunningTime="2026-07-09 16:53:38.675979555 +0000 UTC m=+2303.869085935" Jul 09 16:53:48.404285 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:53:48.404244 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-6bf011/sub-test-model-6bf011-kserve-6cbb7fcf95-ztkbk" Jul 09 16:53:48.404702 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:53:48.404330 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-6bf011/sub-test-model-6bf011-kserve-6cbb7fcf95-ztkbk" Jul 09 16:53:48.417779 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:53:48.417744 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-6bf011/sub-test-model-6bf011-kserve-6cbb7fcf95-ztkbk" Jul 09 16:53:48.727457 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:53:48.727376 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-6bf011/sub-test-model-6bf011-kserve-6cbb7fcf95-ztkbk" Jul 09 16:54:31.708449 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:54:31.708413 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-bef56b/sub-test-model-bef56b-kserve-84b8cf8cb7-tdsc6"] Jul 09 16:54:32.573508 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:54:32.573479 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-6bf011/sub-test-model-6bf011-kserve-6cbb7fcf95-ztkbk"] Jul 09 16:54:46.728987 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:54:46.728923 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-bef56b/sub-test-model-bef56b-kserve-84b8cf8cb7-tdsc6" podUID="2fe719d7-53dc-45f8-a4e6-3d95d867e0d8" containerName="main" containerID="cri-o://293c90de113dbbc8c81404489abffb787fe59b2cf21659ba80a9b5a90f9d53eb" gracePeriod=45 Jul 09 16:54:46.992118 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:54:46.992055 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-bef56b/sub-test-model-bef56b-kserve-84b8cf8cb7-tdsc6" Jul 09 16:54:47.010177 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:54:47.010117 2569 generic.go:356] "Generic (PLEG): container finished" podID="2fe719d7-53dc-45f8-a4e6-3d95d867e0d8" containerID="293c90de113dbbc8c81404489abffb787fe59b2cf21659ba80a9b5a90f9d53eb" exitCode=0 Jul 09 16:54:47.010177 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:54:47.010164 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-bef56b/sub-test-model-bef56b-kserve-84b8cf8cb7-tdsc6" event={"ID":"2fe719d7-53dc-45f8-a4e6-3d95d867e0d8","Type":"ContainerDied","Data":"293c90de113dbbc8c81404489abffb787fe59b2cf21659ba80a9b5a90f9d53eb"} Jul 09 16:54:47.010414 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:54:47.010189 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-bef56b/sub-test-model-bef56b-kserve-84b8cf8cb7-tdsc6" Jul 09 16:54:47.010414 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:54:47.010219 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-bef56b/sub-test-model-bef56b-kserve-84b8cf8cb7-tdsc6" event={"ID":"2fe719d7-53dc-45f8-a4e6-3d95d867e0d8","Type":"ContainerDied","Data":"bc989e1cdbe6797f5771d5225e25d16beecdf7e82abb0d72ea81e539b10dc3e3"} Jul 09 16:54:47.010414 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:54:47.010242 2569 scope.go:117] "RemoveContainer" containerID="293c90de113dbbc8c81404489abffb787fe59b2cf21659ba80a9b5a90f9d53eb" Jul 09 16:54:47.015214 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:54:47.015189 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zw4h6\" (UniqueName: \"kubernetes.io/projected/2fe719d7-53dc-45f8-a4e6-3d95d867e0d8-kube-api-access-zw4h6\") pod \"2fe719d7-53dc-45f8-a4e6-3d95d867e0d8\" (UID: \"2fe719d7-53dc-45f8-a4e6-3d95d867e0d8\") " Jul 09 16:54:47.015379 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:54:47.015249 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fe719d7-53dc-45f8-a4e6-3d95d867e0d8-model-cache\") pod \"2fe719d7-53dc-45f8-a4e6-3d95d867e0d8\" (UID: \"2fe719d7-53dc-45f8-a4e6-3d95d867e0d8\") " Jul 09 16:54:47.015379 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:54:47.015311 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fe719d7-53dc-45f8-a4e6-3d95d867e0d8-home\") pod \"2fe719d7-53dc-45f8-a4e6-3d95d867e0d8\" (UID: \"2fe719d7-53dc-45f8-a4e6-3d95d867e0d8\") " Jul 09 16:54:47.015379 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:54:47.015350 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fe719d7-53dc-45f8-a4e6-3d95d867e0d8-tmp-dir\") pod \"2fe719d7-53dc-45f8-a4e6-3d95d867e0d8\" (UID: \"2fe719d7-53dc-45f8-a4e6-3d95d867e0d8\") " Jul 09 16:54:47.015564 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:54:47.015421 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fe719d7-53dc-45f8-a4e6-3d95d867e0d8-tls-certs\") pod \"2fe719d7-53dc-45f8-a4e6-3d95d867e0d8\" (UID: \"2fe719d7-53dc-45f8-a4e6-3d95d867e0d8\") " Jul 09 16:54:47.015564 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:54:47.015447 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fe719d7-53dc-45f8-a4e6-3d95d867e0d8-dshm\") pod \"2fe719d7-53dc-45f8-a4e6-3d95d867e0d8\" (UID: \"2fe719d7-53dc-45f8-a4e6-3d95d867e0d8\") " Jul 09 16:54:47.015679 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:54:47.015596 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fe719d7-53dc-45f8-a4e6-3d95d867e0d8-model-cache" (OuterVolumeSpecName: "model-cache") pod "2fe719d7-53dc-45f8-a4e6-3d95d867e0d8" (UID: "2fe719d7-53dc-45f8-a4e6-3d95d867e0d8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:54:47.015679 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:54:47.015630 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fe719d7-53dc-45f8-a4e6-3d95d867e0d8-home" (OuterVolumeSpecName: "home") pod "2fe719d7-53dc-45f8-a4e6-3d95d867e0d8" (UID: "2fe719d7-53dc-45f8-a4e6-3d95d867e0d8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:54:47.015787 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:54:47.015748 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fe719d7-53dc-45f8-a4e6-3d95d867e0d8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2fe719d7-53dc-45f8-a4e6-3d95d867e0d8" (UID: "2fe719d7-53dc-45f8-a4e6-3d95d867e0d8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:54:47.015946 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:54:47.015891 2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fe719d7-53dc-45f8-a4e6-3d95d867e0d8-home\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:54:47.015946 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:54:47.015912 2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fe719d7-53dc-45f8-a4e6-3d95d867e0d8-tmp-dir\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:54:47.015946 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:54:47.015925 2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fe719d7-53dc-45f8-a4e6-3d95d867e0d8-model-cache\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:54:47.018154 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:54:47.018118 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fe719d7-53dc-45f8-a4e6-3d95d867e0d8-dshm" (OuterVolumeSpecName: "dshm") pod "2fe719d7-53dc-45f8-a4e6-3d95d867e0d8" (UID: "2fe719d7-53dc-45f8-a4e6-3d95d867e0d8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:54:47.018419 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:54:47.018391 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fe719d7-53dc-45f8-a4e6-3d95d867e0d8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2fe719d7-53dc-45f8-a4e6-3d95d867e0d8" (UID: "2fe719d7-53dc-45f8-a4e6-3d95d867e0d8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 16:54:47.018542 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:54:47.018498 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fe719d7-53dc-45f8-a4e6-3d95d867e0d8-kube-api-access-zw4h6" (OuterVolumeSpecName: "kube-api-access-zw4h6") pod "2fe719d7-53dc-45f8-a4e6-3d95d867e0d8" (UID: "2fe719d7-53dc-45f8-a4e6-3d95d867e0d8"). InnerVolumeSpecName "kube-api-access-zw4h6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 16:54:47.023777 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:54:47.023748 2569 scope.go:117] "RemoveContainer" containerID="293c90de113dbbc8c81404489abffb787fe59b2cf21659ba80a9b5a90f9d53eb" Jul 09 16:54:47.024169 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:54:47.024149 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"293c90de113dbbc8c81404489abffb787fe59b2cf21659ba80a9b5a90f9d53eb\": container with ID starting with 293c90de113dbbc8c81404489abffb787fe59b2cf21659ba80a9b5a90f9d53eb not found: ID does not exist" containerID="293c90de113dbbc8c81404489abffb787fe59b2cf21659ba80a9b5a90f9d53eb" Jul 09 16:54:47.024255 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:54:47.024183 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"293c90de113dbbc8c81404489abffb787fe59b2cf21659ba80a9b5a90f9d53eb"} err="failed to get container status \"293c90de113dbbc8c81404489abffb787fe59b2cf21659ba80a9b5a90f9d53eb\": rpc error: code = NotFound desc = could not find container \"293c90de113dbbc8c81404489abffb787fe59b2cf21659ba80a9b5a90f9d53eb\": container with ID starting with 293c90de113dbbc8c81404489abffb787fe59b2cf21659ba80a9b5a90f9d53eb not found: ID does not exist" Jul 09 16:54:47.117359 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:54:47.117321 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fe719d7-53dc-45f8-a4e6-3d95d867e0d8-tls-certs\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:54:47.117359 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:54:47.117352 2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fe719d7-53dc-45f8-a4e6-3d95d867e0d8-dshm\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:54:47.117359 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:54:47.117362 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zw4h6\" (UniqueName: \"kubernetes.io/projected/2fe719d7-53dc-45f8-a4e6-3d95d867e0d8-kube-api-access-zw4h6\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:54:47.333383 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:54:47.333345 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-bef56b/sub-test-model-bef56b-kserve-84b8cf8cb7-tdsc6"] Jul 09 16:54:47.337205 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:54:47.337177 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-bef56b/sub-test-model-bef56b-kserve-84b8cf8cb7-tdsc6"] Jul 09 16:54:47.390496 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:54:47.390464 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2fe719d7-53dc-45f8-a4e6-3d95d867e0d8" path="/var/lib/kubelet/pods/2fe719d7-53dc-45f8-a4e6-3d95d867e0d8/volumes" Jul 09 16:54:47.594037 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:54:47.594001 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-6bf011/sub-test-model-6bf011-kserve-6cbb7fcf95-ztkbk" podUID="6641aeb0-90c9-4902-b469-9a87552c5e79" containerName="main" containerID="cri-o://b176cbafc2c0569a030a49b6e77c65b86211a2fb43c80054de5eb7501688cf88" gracePeriod=45 Jul 09 16:54:47.852688 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:54:47.852620 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-6bf011/sub-test-model-6bf011-kserve-6cbb7fcf95-ztkbk" Jul 09 16:54:47.924842 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:54:47.924801 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6641aeb0-90c9-4902-b469-9a87552c5e79-tls-certs\") pod \"6641aeb0-90c9-4902-b469-9a87552c5e79\" (UID: \"6641aeb0-90c9-4902-b469-9a87552c5e79\") " Jul 09 16:54:47.924842 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:54:47.924845 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2pt4t\" (UniqueName: \"kubernetes.io/projected/6641aeb0-90c9-4902-b469-9a87552c5e79-kube-api-access-2pt4t\") pod \"6641aeb0-90c9-4902-b469-9a87552c5e79\" (UID: \"6641aeb0-90c9-4902-b469-9a87552c5e79\") " Jul 09 16:54:47.925108 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:54:47.924866 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6641aeb0-90c9-4902-b469-9a87552c5e79-tmp-dir\") pod \"6641aeb0-90c9-4902-b469-9a87552c5e79\" (UID: \"6641aeb0-90c9-4902-b469-9a87552c5e79\") " Jul 09 16:54:47.925108 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:54:47.924912 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6641aeb0-90c9-4902-b469-9a87552c5e79-dshm\") pod \"6641aeb0-90c9-4902-b469-9a87552c5e79\" (UID: \"6641aeb0-90c9-4902-b469-9a87552c5e79\") " Jul 09 16:54:47.925108 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:54:47.924945 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6641aeb0-90c9-4902-b469-9a87552c5e79-model-cache\") pod \"6641aeb0-90c9-4902-b469-9a87552c5e79\" (UID: \"6641aeb0-90c9-4902-b469-9a87552c5e79\") " Jul 09 16:54:47.925108 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:54:47.924987 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6641aeb0-90c9-4902-b469-9a87552c5e79-home\") pod \"6641aeb0-90c9-4902-b469-9a87552c5e79\" (UID: \"6641aeb0-90c9-4902-b469-9a87552c5e79\") " Jul 09 16:54:47.925281 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:54:47.925150 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6641aeb0-90c9-4902-b469-9a87552c5e79-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6641aeb0-90c9-4902-b469-9a87552c5e79" (UID: "6641aeb0-90c9-4902-b469-9a87552c5e79"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:54:47.925281 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:54:47.925248 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6641aeb0-90c9-4902-b469-9a87552c5e79-model-cache" (OuterVolumeSpecName: "model-cache") pod "6641aeb0-90c9-4902-b469-9a87552c5e79" (UID: "6641aeb0-90c9-4902-b469-9a87552c5e79"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:54:47.925281 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:54:47.925265 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6641aeb0-90c9-4902-b469-9a87552c5e79-home" (OuterVolumeSpecName: "home") pod "6641aeb0-90c9-4902-b469-9a87552c5e79" (UID: "6641aeb0-90c9-4902-b469-9a87552c5e79"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:54:47.925521 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:54:47.925498 2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6641aeb0-90c9-4902-b469-9a87552c5e79-tmp-dir\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:54:47.925578 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:54:47.925526 2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6641aeb0-90c9-4902-b469-9a87552c5e79-model-cache\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:54:47.925578 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:54:47.925538 2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6641aeb0-90c9-4902-b469-9a87552c5e79-home\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:54:47.927265 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:54:47.927242 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6641aeb0-90c9-4902-b469-9a87552c5e79-dshm" (OuterVolumeSpecName: "dshm") pod "6641aeb0-90c9-4902-b469-9a87552c5e79" (UID: "6641aeb0-90c9-4902-b469-9a87552c5e79"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:54:47.927265 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:54:47.927257 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6641aeb0-90c9-4902-b469-9a87552c5e79-kube-api-access-2pt4t" (OuterVolumeSpecName: "kube-api-access-2pt4t") pod "6641aeb0-90c9-4902-b469-9a87552c5e79" (UID: "6641aeb0-90c9-4902-b469-9a87552c5e79"). InnerVolumeSpecName "kube-api-access-2pt4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 16:54:47.927406 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:54:47.927342 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6641aeb0-90c9-4902-b469-9a87552c5e79-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6641aeb0-90c9-4902-b469-9a87552c5e79" (UID: "6641aeb0-90c9-4902-b469-9a87552c5e79"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 16:54:48.016515 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:54:48.016479 2569 generic.go:356] "Generic (PLEG): container finished" podID="6641aeb0-90c9-4902-b469-9a87552c5e79" containerID="b176cbafc2c0569a030a49b6e77c65b86211a2fb43c80054de5eb7501688cf88" exitCode=0 Jul 09 16:54:48.016700 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:54:48.016523 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-6bf011/sub-test-model-6bf011-kserve-6cbb7fcf95-ztkbk" Jul 09 16:54:48.016700 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:54:48.016565 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-6bf011/sub-test-model-6bf011-kserve-6cbb7fcf95-ztkbk" event={"ID":"6641aeb0-90c9-4902-b469-9a87552c5e79","Type":"ContainerDied","Data":"b176cbafc2c0569a030a49b6e77c65b86211a2fb43c80054de5eb7501688cf88"} Jul 09 16:54:48.016700 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:54:48.016610 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-6bf011/sub-test-model-6bf011-kserve-6cbb7fcf95-ztkbk" event={"ID":"6641aeb0-90c9-4902-b469-9a87552c5e79","Type":"ContainerDied","Data":"403687685ff02964beb298110cd469d6da60ef04b1b6583f45315823be5362c8"} Jul 09 16:54:48.016700 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:54:48.016635 2569 scope.go:117] "RemoveContainer" containerID="b176cbafc2c0569a030a49b6e77c65b86211a2fb43c80054de5eb7501688cf88" Jul 09 16:54:48.026158 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:54:48.026135 2569 scope.go:117] "RemoveContainer" containerID="b176cbafc2c0569a030a49b6e77c65b86211a2fb43c80054de5eb7501688cf88" Jul 09 16:54:48.026302 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:54:48.026284 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6641aeb0-90c9-4902-b469-9a87552c5e79-tls-certs\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:54:48.026382 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:54:48.026306 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pt4t\" (UniqueName: \"kubernetes.io/projected/6641aeb0-90c9-4902-b469-9a87552c5e79-kube-api-access-2pt4t\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:54:48.026382 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:54:48.026315 2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6641aeb0-90c9-4902-b469-9a87552c5e79-dshm\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:54:48.026503 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:54:48.026483 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b176cbafc2c0569a030a49b6e77c65b86211a2fb43c80054de5eb7501688cf88\": container with ID starting with b176cbafc2c0569a030a49b6e77c65b86211a2fb43c80054de5eb7501688cf88 not found: ID does not exist" containerID="b176cbafc2c0569a030a49b6e77c65b86211a2fb43c80054de5eb7501688cf88" Jul 09 16:54:48.026543 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:54:48.026512 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b176cbafc2c0569a030a49b6e77c65b86211a2fb43c80054de5eb7501688cf88"} err="failed to get container status \"b176cbafc2c0569a030a49b6e77c65b86211a2fb43c80054de5eb7501688cf88\": rpc error: code = NotFound desc = could not find container \"b176cbafc2c0569a030a49b6e77c65b86211a2fb43c80054de5eb7501688cf88\": container with ID starting with b176cbafc2c0569a030a49b6e77c65b86211a2fb43c80054de5eb7501688cf88 not found: ID does not exist" Jul 09 16:54:48.046179 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:54:48.046138 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-6bf011/sub-test-model-6bf011-kserve-6cbb7fcf95-ztkbk"] Jul 09 16:54:48.049296 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:54:48.049260 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-6bf011/sub-test-model-6bf011-kserve-6cbb7fcf95-ztkbk"] Jul 09 16:54:49.389330 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:54:49.389286 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6641aeb0-90c9-4902-b469-9a87552c5e79" path="/var/lib/kubelet/pods/6641aeb0-90c9-4902-b469-9a87552c5e79/volumes" Jul 09 16:55:02.066711 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:55:02.066675 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-6bf011/rate-test-model-6bf011-kserve-6d58995f79-ctdtg"] Jul 09 16:55:02.067198 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:55:02.067078 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2fe719d7-53dc-45f8-a4e6-3d95d867e0d8" containerName="main" Jul 09 16:55:02.067198 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:55:02.067089 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="2fe719d7-53dc-45f8-a4e6-3d95d867e0d8" containerName="main" Jul 09 16:55:02.067198 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:55:02.067116 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6641aeb0-90c9-4902-b469-9a87552c5e79" containerName="main" Jul 09 16:55:02.067198 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:55:02.067122 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6641aeb0-90c9-4902-b469-9a87552c5e79" containerName="main" Jul 09 16:55:02.067198 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:55:02.067175 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="6641aeb0-90c9-4902-b469-9a87552c5e79" containerName="main" Jul 09 16:55:02.067198 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:55:02.067185 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="2fe719d7-53dc-45f8-a4e6-3d95d867e0d8" containerName="main" Jul 09 16:55:02.070288 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:55:02.070272 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-6bf011/rate-test-model-6bf011-kserve-6d58995f79-ctdtg" Jul 09 16:55:02.073195 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:55:02.073175 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-6bf011\"/\"kube-root-ca.crt\"" Jul 09 16:55:02.074403 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:55:02.074384 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-6bf011\"/\"openshift-service-ca.crt\"" Jul 09 16:55:02.074506 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:55:02.074414 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-6bf011\"/\"default-dockercfg-ljp6q\"" Jul 09 16:55:02.074506 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:55:02.074424 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-6bf011\"/\"rate-test-model-6bf011-kserve-self-signed-certs\"" Jul 09 16:55:02.079007 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:55:02.078983 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-6bf011/rate-test-model-6bf011-kserve-6d58995f79-ctdtg"] Jul 09 16:55:02.156855 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:55:02.156821 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ce6e48c-7360-48be-8f09-8dc677a0a8c2-tmp-dir\") pod \"rate-test-model-6bf011-kserve-6d58995f79-ctdtg\" (UID: \"3ce6e48c-7360-48be-8f09-8dc677a0a8c2\") " pod="ai-tenant-e2e-shared-b-6bf011/rate-test-model-6bf011-kserve-6d58995f79-ctdtg" Jul 09 16:55:02.157079 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:55:02.156869 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ce6e48c-7360-48be-8f09-8dc677a0a8c2-model-cache\") pod \"rate-test-model-6bf011-kserve-6d58995f79-ctdtg\" (UID: \"3ce6e48c-7360-48be-8f09-8dc677a0a8c2\") " pod="ai-tenant-e2e-shared-b-6bf011/rate-test-model-6bf011-kserve-6d58995f79-ctdtg" Jul 09 16:55:02.157079 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:55:02.156922 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7dbv\" (UniqueName: \"kubernetes.io/projected/3ce6e48c-7360-48be-8f09-8dc677a0a8c2-kube-api-access-l7dbv\") pod \"rate-test-model-6bf011-kserve-6d58995f79-ctdtg\" (UID: \"3ce6e48c-7360-48be-8f09-8dc677a0a8c2\") " pod="ai-tenant-e2e-shared-b-6bf011/rate-test-model-6bf011-kserve-6d58995f79-ctdtg" Jul 09 16:55:02.157215 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:55:02.157078 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ce6e48c-7360-48be-8f09-8dc677a0a8c2-home\") pod \"rate-test-model-6bf011-kserve-6d58995f79-ctdtg\" (UID: \"3ce6e48c-7360-48be-8f09-8dc677a0a8c2\") " pod="ai-tenant-e2e-shared-b-6bf011/rate-test-model-6bf011-kserve-6d58995f79-ctdtg" Jul 09 16:55:02.157215 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:55:02.157112 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ce6e48c-7360-48be-8f09-8dc677a0a8c2-dshm\") pod \"rate-test-model-6bf011-kserve-6d58995f79-ctdtg\" (UID: \"3ce6e48c-7360-48be-8f09-8dc677a0a8c2\") " pod="ai-tenant-e2e-shared-b-6bf011/rate-test-model-6bf011-kserve-6d58995f79-ctdtg" Jul 09 16:55:02.157215 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:55:02.157138 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ce6e48c-7360-48be-8f09-8dc677a0a8c2-tls-certs\") pod \"rate-test-model-6bf011-kserve-6d58995f79-ctdtg\" (UID: \"3ce6e48c-7360-48be-8f09-8dc677a0a8c2\") " pod="ai-tenant-e2e-shared-b-6bf011/rate-test-model-6bf011-kserve-6d58995f79-ctdtg" Jul 09 16:55:02.258088 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:55:02.258050 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ce6e48c-7360-48be-8f09-8dc677a0a8c2-tmp-dir\") pod \"rate-test-model-6bf011-kserve-6d58995f79-ctdtg\" (UID: \"3ce6e48c-7360-48be-8f09-8dc677a0a8c2\") " pod="ai-tenant-e2e-shared-b-6bf011/rate-test-model-6bf011-kserve-6d58995f79-ctdtg" Jul 09 16:55:02.258290 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:55:02.258107 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ce6e48c-7360-48be-8f09-8dc677a0a8c2-model-cache\") pod \"rate-test-model-6bf011-kserve-6d58995f79-ctdtg\" (UID: \"3ce6e48c-7360-48be-8f09-8dc677a0a8c2\") " pod="ai-tenant-e2e-shared-b-6bf011/rate-test-model-6bf011-kserve-6d58995f79-ctdtg" Jul 09 16:55:02.258290 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:55:02.258128 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7dbv\" (UniqueName: \"kubernetes.io/projected/3ce6e48c-7360-48be-8f09-8dc677a0a8c2-kube-api-access-l7dbv\") pod \"rate-test-model-6bf011-kserve-6d58995f79-ctdtg\" (UID: \"3ce6e48c-7360-48be-8f09-8dc677a0a8c2\") " pod="ai-tenant-e2e-shared-b-6bf011/rate-test-model-6bf011-kserve-6d58995f79-ctdtg" Jul 09 16:55:02.258290 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:55:02.258208 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ce6e48c-7360-48be-8f09-8dc677a0a8c2-home\") pod \"rate-test-model-6bf011-kserve-6d58995f79-ctdtg\" (UID: \"3ce6e48c-7360-48be-8f09-8dc677a0a8c2\") " pod="ai-tenant-e2e-shared-b-6bf011/rate-test-model-6bf011-kserve-6d58995f79-ctdtg" Jul 09 16:55:02.258290 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:55:02.258226 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ce6e48c-7360-48be-8f09-8dc677a0a8c2-dshm\") pod \"rate-test-model-6bf011-kserve-6d58995f79-ctdtg\" (UID: \"3ce6e48c-7360-48be-8f09-8dc677a0a8c2\") " pod="ai-tenant-e2e-shared-b-6bf011/rate-test-model-6bf011-kserve-6d58995f79-ctdtg" Jul 09 16:55:02.258290 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:55:02.258243 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ce6e48c-7360-48be-8f09-8dc677a0a8c2-tls-certs\") pod \"rate-test-model-6bf011-kserve-6d58995f79-ctdtg\" (UID: \"3ce6e48c-7360-48be-8f09-8dc677a0a8c2\") " pod="ai-tenant-e2e-shared-b-6bf011/rate-test-model-6bf011-kserve-6d58995f79-ctdtg" Jul 09 16:55:02.258564 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:55:02.258463 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ce6e48c-7360-48be-8f09-8dc677a0a8c2-tmp-dir\") pod \"rate-test-model-6bf011-kserve-6d58995f79-ctdtg\" (UID: \"3ce6e48c-7360-48be-8f09-8dc677a0a8c2\") " pod="ai-tenant-e2e-shared-b-6bf011/rate-test-model-6bf011-kserve-6d58995f79-ctdtg" Jul 09 16:55:02.258564 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:55:02.258533 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ce6e48c-7360-48be-8f09-8dc677a0a8c2-model-cache\") pod \"rate-test-model-6bf011-kserve-6d58995f79-ctdtg\" (UID: \"3ce6e48c-7360-48be-8f09-8dc677a0a8c2\") " pod="ai-tenant-e2e-shared-b-6bf011/rate-test-model-6bf011-kserve-6d58995f79-ctdtg" Jul 09 16:55:02.258670 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:55:02.258594 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ce6e48c-7360-48be-8f09-8dc677a0a8c2-home\") pod \"rate-test-model-6bf011-kserve-6d58995f79-ctdtg\" (UID: \"3ce6e48c-7360-48be-8f09-8dc677a0a8c2\") " pod="ai-tenant-e2e-shared-b-6bf011/rate-test-model-6bf011-kserve-6d58995f79-ctdtg" Jul 09 16:55:02.260563 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:55:02.260531 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ce6e48c-7360-48be-8f09-8dc677a0a8c2-dshm\") pod \"rate-test-model-6bf011-kserve-6d58995f79-ctdtg\" (UID: \"3ce6e48c-7360-48be-8f09-8dc677a0a8c2\") " pod="ai-tenant-e2e-shared-b-6bf011/rate-test-model-6bf011-kserve-6d58995f79-ctdtg" Jul 09 16:55:02.260975 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:55:02.260955 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ce6e48c-7360-48be-8f09-8dc677a0a8c2-tls-certs\") pod \"rate-test-model-6bf011-kserve-6d58995f79-ctdtg\" (UID: \"3ce6e48c-7360-48be-8f09-8dc677a0a8c2\") " pod="ai-tenant-e2e-shared-b-6bf011/rate-test-model-6bf011-kserve-6d58995f79-ctdtg" Jul 09 16:55:02.266092 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:55:02.266075 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7dbv\" (UniqueName: \"kubernetes.io/projected/3ce6e48c-7360-48be-8f09-8dc677a0a8c2-kube-api-access-l7dbv\") pod \"rate-test-model-6bf011-kserve-6d58995f79-ctdtg\" (UID: \"3ce6e48c-7360-48be-8f09-8dc677a0a8c2\") " pod="ai-tenant-e2e-shared-b-6bf011/rate-test-model-6bf011-kserve-6d58995f79-ctdtg" Jul 09 16:55:02.381047 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:55:02.381009 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-6bf011/rate-test-model-6bf011-kserve-6d58995f79-ctdtg" Jul 09 16:55:02.520815 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:55:02.520787 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-6bf011/rate-test-model-6bf011-kserve-6d58995f79-ctdtg"] Jul 09 16:55:02.523756 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:55:02.523729 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3ce6e48c_7360_48be_8f09_8dc677a0a8c2.slice/crio-3b2cc7b4a44106cc8177dc10ea696aea078eca3ce92bead7a6eb81f5f944e332 WatchSource:0}: Error finding container 3b2cc7b4a44106cc8177dc10ea696aea078eca3ce92bead7a6eb81f5f944e332: Status 404 returned error can't find the container with id 3b2cc7b4a44106cc8177dc10ea696aea078eca3ce92bead7a6eb81f5f944e332 Jul 09 16:55:03.096457 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:55:03.096417 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-6bf011/rate-test-model-6bf011-kserve-6d58995f79-ctdtg" event={"ID":"3ce6e48c-7360-48be-8f09-8dc677a0a8c2","Type":"ContainerStarted","Data":"5b2de6116ca8672c300db94fffe285b49db948d23913b75daa12d2a3399a67fb"} Jul 09 16:55:03.096910 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:55:03.096471 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-6bf011/rate-test-model-6bf011-kserve-6d58995f79-ctdtg" event={"ID":"3ce6e48c-7360-48be-8f09-8dc677a0a8c2","Type":"ContainerStarted","Data":"3b2cc7b4a44106cc8177dc10ea696aea078eca3ce92bead7a6eb81f5f944e332"} Jul 09 16:55:12.381806 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:55:12.381743 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-6bf011/rate-test-model-6bf011-kserve-6d58995f79-ctdtg" Jul 09 16:55:12.381806 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:55:12.381819 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-6bf011/rate-test-model-6bf011-kserve-6d58995f79-ctdtg" Jul 09 16:55:12.394700 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:55:12.394664 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-6bf011/rate-test-model-6bf011-kserve-6d58995f79-ctdtg" Jul 09 16:55:12.412817 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:55:12.412763 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-6bf011/rate-test-model-6bf011-kserve-6d58995f79-ctdtg" podStartSLOduration=10.412730737 podStartE2EDuration="10.412730737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:55:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:55:03.113208733 +0000 UTC m=+2388.306315115" watchObservedRunningTime="2026-07-09 16:55:12.412730737 +0000 UTC m=+2397.605837118" Jul 09 16:55:13.164755 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:55:13.164726 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-6bf011/rate-test-model-6bf011-kserve-6d58995f79-ctdtg" Jul 09 16:55:15.447680 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:55:15.447648 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-nzrk9_a284ea64-610c-40f7-837b-fa0975c0a290/console-operator/2.log" Jul 09 16:55:15.455784 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:55:15.455757 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-49sqd_17313664-7a0f-472b-a597-39f8c5412bc3/ovn-acl-logging/0.log" Jul 09 16:55:15.457161 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:55:15.457138 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-nzrk9_a284ea64-610c-40f7-837b-fa0975c0a290/console-operator/2.log" Jul 09 16:55:15.464986 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:55:15.464968 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-49sqd_17313664-7a0f-472b-a597-39f8c5412bc3/ovn-acl-logging/0.log" Jul 09 16:55:46.777165 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:55:46.777090 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-6bf011/rate-test-model-6bf011-kserve-6d58995f79-ctdtg"] Jul 09 16:56:01.797457 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:01.797416 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-6bf011/rate-test-model-6bf011-kserve-6d58995f79-ctdtg" podUID="3ce6e48c-7360-48be-8f09-8dc677a0a8c2" containerName="main" containerID="cri-o://5b2de6116ca8672c300db94fffe285b49db948d23913b75daa12d2a3399a67fb" gracePeriod=45 Jul 09 16:56:02.061229 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:02.061161 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-6bf011/rate-test-model-6bf011-kserve-6d58995f79-ctdtg" Jul 09 16:56:02.217701 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:02.217661 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l7dbv\" (UniqueName: \"kubernetes.io/projected/3ce6e48c-7360-48be-8f09-8dc677a0a8c2-kube-api-access-l7dbv\") pod \"3ce6e48c-7360-48be-8f09-8dc677a0a8c2\" (UID: \"3ce6e48c-7360-48be-8f09-8dc677a0a8c2\") " Jul 09 16:56:02.218036 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:02.217756 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ce6e48c-7360-48be-8f09-8dc677a0a8c2-home\") pod \"3ce6e48c-7360-48be-8f09-8dc677a0a8c2\" (UID: \"3ce6e48c-7360-48be-8f09-8dc677a0a8c2\") " Jul 09 16:56:02.218036 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:02.217790 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ce6e48c-7360-48be-8f09-8dc677a0a8c2-tmp-dir\") pod \"3ce6e48c-7360-48be-8f09-8dc677a0a8c2\" (UID: \"3ce6e48c-7360-48be-8f09-8dc677a0a8c2\") " Jul 09 16:56:02.218036 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:02.217810 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ce6e48c-7360-48be-8f09-8dc677a0a8c2-model-cache\") pod \"3ce6e48c-7360-48be-8f09-8dc677a0a8c2\" (UID: \"3ce6e48c-7360-48be-8f09-8dc677a0a8c2\") " Jul 09 16:56:02.218036 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:02.217833 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ce6e48c-7360-48be-8f09-8dc677a0a8c2-dshm\") pod \"3ce6e48c-7360-48be-8f09-8dc677a0a8c2\" (UID: \"3ce6e48c-7360-48be-8f09-8dc677a0a8c2\") " Jul 09 16:56:02.218036 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:02.217929 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ce6e48c-7360-48be-8f09-8dc677a0a8c2-tls-certs\") pod \"3ce6e48c-7360-48be-8f09-8dc677a0a8c2\" (UID: \"3ce6e48c-7360-48be-8f09-8dc677a0a8c2\") " Jul 09 16:56:02.218267 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:02.218093 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ce6e48c-7360-48be-8f09-8dc677a0a8c2-model-cache" (OuterVolumeSpecName: "model-cache") pod "3ce6e48c-7360-48be-8f09-8dc677a0a8c2" (UID: "3ce6e48c-7360-48be-8f09-8dc677a0a8c2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:56:02.218267 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:02.218108 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ce6e48c-7360-48be-8f09-8dc677a0a8c2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3ce6e48c-7360-48be-8f09-8dc677a0a8c2" (UID: "3ce6e48c-7360-48be-8f09-8dc677a0a8c2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:56:02.218267 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:02.218121 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ce6e48c-7360-48be-8f09-8dc677a0a8c2-home" (OuterVolumeSpecName: "home") pod "3ce6e48c-7360-48be-8f09-8dc677a0a8c2" (UID: "3ce6e48c-7360-48be-8f09-8dc677a0a8c2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:56:02.218387 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:02.218305 2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ce6e48c-7360-48be-8f09-8dc677a0a8c2-home\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:56:02.218387 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:02.218324 2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ce6e48c-7360-48be-8f09-8dc677a0a8c2-tmp-dir\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:56:02.218387 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:02.218337 2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ce6e48c-7360-48be-8f09-8dc677a0a8c2-model-cache\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:56:02.220075 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:02.220039 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ce6e48c-7360-48be-8f09-8dc677a0a8c2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3ce6e48c-7360-48be-8f09-8dc677a0a8c2" (UID: "3ce6e48c-7360-48be-8f09-8dc677a0a8c2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 16:56:02.220209 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:02.220099 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ce6e48c-7360-48be-8f09-8dc677a0a8c2-kube-api-access-l7dbv" (OuterVolumeSpecName: "kube-api-access-l7dbv") pod "3ce6e48c-7360-48be-8f09-8dc677a0a8c2" (UID: "3ce6e48c-7360-48be-8f09-8dc677a0a8c2"). InnerVolumeSpecName "kube-api-access-l7dbv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 16:56:02.220209 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:02.220126 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ce6e48c-7360-48be-8f09-8dc677a0a8c2-dshm" (OuterVolumeSpecName: "dshm") pod "3ce6e48c-7360-48be-8f09-8dc677a0a8c2" (UID: "3ce6e48c-7360-48be-8f09-8dc677a0a8c2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 16:56:02.322913 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:02.322785 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7dbv\" (UniqueName: \"kubernetes.io/projected/3ce6e48c-7360-48be-8f09-8dc677a0a8c2-kube-api-access-l7dbv\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:56:02.322913 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:02.322813 2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ce6e48c-7360-48be-8f09-8dc677a0a8c2-dshm\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:56:02.322913 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:02.322822 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ce6e48c-7360-48be-8f09-8dc677a0a8c2-tls-certs\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:56:02.399381 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:02.399339 2569 generic.go:356] "Generic (PLEG): container finished" podID="3ce6e48c-7360-48be-8f09-8dc677a0a8c2" containerID="5b2de6116ca8672c300db94fffe285b49db948d23913b75daa12d2a3399a67fb" exitCode=0 Jul 09 16:56:02.399552 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:02.399408 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-6bf011/rate-test-model-6bf011-kserve-6d58995f79-ctdtg" Jul 09 16:56:02.399552 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:02.399429 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-6bf011/rate-test-model-6bf011-kserve-6d58995f79-ctdtg" event={"ID":"3ce6e48c-7360-48be-8f09-8dc677a0a8c2","Type":"ContainerDied","Data":"5b2de6116ca8672c300db94fffe285b49db948d23913b75daa12d2a3399a67fb"} Jul 09 16:56:02.399552 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:02.399481 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-6bf011/rate-test-model-6bf011-kserve-6d58995f79-ctdtg" event={"ID":"3ce6e48c-7360-48be-8f09-8dc677a0a8c2","Type":"ContainerDied","Data":"3b2cc7b4a44106cc8177dc10ea696aea078eca3ce92bead7a6eb81f5f944e332"} Jul 09 16:56:02.399552 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:02.399500 2569 scope.go:117] "RemoveContainer" containerID="5b2de6116ca8672c300db94fffe285b49db948d23913b75daa12d2a3399a67fb" Jul 09 16:56:02.409310 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:02.409291 2569 scope.go:117] "RemoveContainer" containerID="5b2de6116ca8672c300db94fffe285b49db948d23913b75daa12d2a3399a67fb" Jul 09 16:56:02.409652 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:56:02.409620 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b2de6116ca8672c300db94fffe285b49db948d23913b75daa12d2a3399a67fb\": container with ID starting with 5b2de6116ca8672c300db94fffe285b49db948d23913b75daa12d2a3399a67fb not found: ID does not exist" containerID="5b2de6116ca8672c300db94fffe285b49db948d23913b75daa12d2a3399a67fb" Jul 09 16:56:02.409723 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:02.409664 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b2de6116ca8672c300db94fffe285b49db948d23913b75daa12d2a3399a67fb"} err="failed to get container status \"5b2de6116ca8672c300db94fffe285b49db948d23913b75daa12d2a3399a67fb\": rpc error: code = NotFound desc = could not find container \"5b2de6116ca8672c300db94fffe285b49db948d23913b75daa12d2a3399a67fb\": container with ID starting with 5b2de6116ca8672c300db94fffe285b49db948d23913b75daa12d2a3399a67fb not found: ID does not exist" Jul 09 16:56:02.421110 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:02.421078 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-6bf011/rate-test-model-6bf011-kserve-6d58995f79-ctdtg"] Jul 09 16:56:02.424305 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:02.424279 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-6bf011/rate-test-model-6bf011-kserve-6d58995f79-ctdtg"] Jul 09 16:56:03.389647 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:03.389604 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ce6e48c-7360-48be-8f09-8dc677a0a8c2" path="/var/lib/kubelet/pods/3ce6e48c-7360-48be-8f09-8dc677a0a8c2/volumes" Jul 09 16:56:06.871352 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:06.871320 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-bef56b-57c67b66fb-wqdxl"] Jul 09 16:56:06.871930 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:06.871547 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-bef56b-57c67b66fb-wqdxl" podUID="92b20001-ee0f-43ec-a9ab-8ddb1a6e0cd1" containerName="maas-api" containerID="cri-o://f945ae412f6bb8a4544961b2916d8173a6f75fb56d3d1cfcdf5f96cf459705b9" gracePeriod=30 Jul 09 16:56:07.123566 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:07.123493 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-bef56b-57c67b66fb-wqdxl" Jul 09 16:56:07.266907 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:07.266840 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92b20001-ee0f-43ec-a9ab-8ddb1a6e0cd1-maas-api-tls\") pod \"92b20001-ee0f-43ec-a9ab-8ddb1a6e0cd1\" (UID: \"92b20001-ee0f-43ec-a9ab-8ddb1a6e0cd1\") " Jul 09 16:56:07.267119 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:07.266953 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92b20001-ee0f-43ec-a9ab-8ddb1a6e0cd1-openshift-service-ca\") pod \"92b20001-ee0f-43ec-a9ab-8ddb1a6e0cd1\" (UID: \"92b20001-ee0f-43ec-a9ab-8ddb1a6e0cd1\") " Jul 09 16:56:07.267119 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:07.266992 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xq7xd\" (UniqueName: \"kubernetes.io/projected/92b20001-ee0f-43ec-a9ab-8ddb1a6e0cd1-kube-api-access-xq7xd\") pod \"92b20001-ee0f-43ec-a9ab-8ddb1a6e0cd1\" (UID: \"92b20001-ee0f-43ec-a9ab-8ddb1a6e0cd1\") " Jul 09 16:56:07.267693 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:07.267586 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92b20001-ee0f-43ec-a9ab-8ddb1a6e0cd1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "92b20001-ee0f-43ec-a9ab-8ddb1a6e0cd1" (UID: "92b20001-ee0f-43ec-a9ab-8ddb1a6e0cd1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 16:56:07.268295 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:07.267773 2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92b20001-ee0f-43ec-a9ab-8ddb1a6e0cd1-openshift-service-ca\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:56:07.270201 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:07.270172 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92b20001-ee0f-43ec-a9ab-8ddb1a6e0cd1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "92b20001-ee0f-43ec-a9ab-8ddb1a6e0cd1" (UID: "92b20001-ee0f-43ec-a9ab-8ddb1a6e0cd1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 16:56:07.270327 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:07.270275 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92b20001-ee0f-43ec-a9ab-8ddb1a6e0cd1-kube-api-access-xq7xd" (OuterVolumeSpecName: "kube-api-access-xq7xd") pod "92b20001-ee0f-43ec-a9ab-8ddb1a6e0cd1" (UID: "92b20001-ee0f-43ec-a9ab-8ddb1a6e0cd1"). InnerVolumeSpecName "kube-api-access-xq7xd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 16:56:07.369447 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:07.369388 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xq7xd\" (UniqueName: \"kubernetes.io/projected/92b20001-ee0f-43ec-a9ab-8ddb1a6e0cd1-kube-api-access-xq7xd\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:56:07.369447 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:07.369439 2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92b20001-ee0f-43ec-a9ab-8ddb1a6e0cd1-maas-api-tls\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:56:07.426008 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:07.425973 2569 generic.go:356] "Generic (PLEG): container finished" podID="92b20001-ee0f-43ec-a9ab-8ddb1a6e0cd1" containerID="f945ae412f6bb8a4544961b2916d8173a6f75fb56d3d1cfcdf5f96cf459705b9" exitCode=0 Jul 09 16:56:07.426182 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:07.426030 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-bef56b-57c67b66fb-wqdxl" Jul 09 16:56:07.426182 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:07.426042 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-bef56b-57c67b66fb-wqdxl" event={"ID":"92b20001-ee0f-43ec-a9ab-8ddb1a6e0cd1","Type":"ContainerDied","Data":"f945ae412f6bb8a4544961b2916d8173a6f75fb56d3d1cfcdf5f96cf459705b9"} Jul 09 16:56:07.426182 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:07.426085 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-bef56b-57c67b66fb-wqdxl" event={"ID":"92b20001-ee0f-43ec-a9ab-8ddb1a6e0cd1","Type":"ContainerDied","Data":"7f73df0bf1ab9168e82350702891f1375b764c64cc36c9994fcf92f04a95bb79"} Jul 09 16:56:07.426182 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:07.426105 2569 scope.go:117] "RemoveContainer" containerID="f945ae412f6bb8a4544961b2916d8173a6f75fb56d3d1cfcdf5f96cf459705b9" Jul 09 16:56:07.435244 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:07.435224 2569 scope.go:117] "RemoveContainer" containerID="f945ae412f6bb8a4544961b2916d8173a6f75fb56d3d1cfcdf5f96cf459705b9" Jul 09 16:56:07.435572 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:56:07.435552 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f945ae412f6bb8a4544961b2916d8173a6f75fb56d3d1cfcdf5f96cf459705b9\": container with ID starting with f945ae412f6bb8a4544961b2916d8173a6f75fb56d3d1cfcdf5f96cf459705b9 not found: ID does not exist" containerID="f945ae412f6bb8a4544961b2916d8173a6f75fb56d3d1cfcdf5f96cf459705b9" Jul 09 16:56:07.435643 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:07.435586 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f945ae412f6bb8a4544961b2916d8173a6f75fb56d3d1cfcdf5f96cf459705b9"} err="failed to get container status \"f945ae412f6bb8a4544961b2916d8173a6f75fb56d3d1cfcdf5f96cf459705b9\": rpc error: code = NotFound desc = could not find container \"f945ae412f6bb8a4544961b2916d8173a6f75fb56d3d1cfcdf5f96cf459705b9\": container with ID starting with f945ae412f6bb8a4544961b2916d8173a6f75fb56d3d1cfcdf5f96cf459705b9 not found: ID does not exist" Jul 09 16:56:07.443572 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:07.443546 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-bef56b-57c67b66fb-wqdxl"] Jul 09 16:56:07.446905 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:07.446866 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-bef56b-57c67b66fb-wqdxl"] Jul 09 16:56:09.390149 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:09.390117 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92b20001-ee0f-43ec-a9ab-8ddb1a6e0cd1" path="/var/lib/kubelet/pods/92b20001-ee0f-43ec-a9ab-8ddb1a6e0cd1/volumes" Jul 09 16:56:14.018274 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:14.018225 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-6bf011-89fc49d7c-cbp7x"] Jul 09 16:56:14.018747 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:14.018514 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6bf011-89fc49d7c-cbp7x" podUID="1425d20c-1dde-4d40-b44e-490af34d6f0c" containerName="maas-api" containerID="cri-o://ebb7bf06497de4907fb130168be50d3f2eded0370052aa202e3f88cbd1cd5a77" gracePeriod=30 Jul 09 16:56:14.284800 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:14.284773 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6bf011-89fc49d7c-cbp7x" Jul 09 16:56:14.321213 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:14.321179 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1425d20c-1dde-4d40-b44e-490af34d6f0c-maas-api-tls\") pod \"1425d20c-1dde-4d40-b44e-490af34d6f0c\" (UID: \"1425d20c-1dde-4d40-b44e-490af34d6f0c\") " Jul 09 16:56:14.321361 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:14.321224 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7qg5l\" (UniqueName: \"kubernetes.io/projected/1425d20c-1dde-4d40-b44e-490af34d6f0c-kube-api-access-7qg5l\") pod \"1425d20c-1dde-4d40-b44e-490af34d6f0c\" (UID: \"1425d20c-1dde-4d40-b44e-490af34d6f0c\") " Jul 09 16:56:14.321361 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:14.321258 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1425d20c-1dde-4d40-b44e-490af34d6f0c-openshift-service-ca\") pod \"1425d20c-1dde-4d40-b44e-490af34d6f0c\" (UID: \"1425d20c-1dde-4d40-b44e-490af34d6f0c\") " Jul 09 16:56:14.321704 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:14.321680 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1425d20c-1dde-4d40-b44e-490af34d6f0c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1425d20c-1dde-4d40-b44e-490af34d6f0c" (UID: "1425d20c-1dde-4d40-b44e-490af34d6f0c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 16:56:14.323460 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:14.323432 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1425d20c-1dde-4d40-b44e-490af34d6f0c-kube-api-access-7qg5l" (OuterVolumeSpecName: "kube-api-access-7qg5l") pod "1425d20c-1dde-4d40-b44e-490af34d6f0c" (UID: "1425d20c-1dde-4d40-b44e-490af34d6f0c"). InnerVolumeSpecName "kube-api-access-7qg5l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 16:56:14.323595 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:14.323499 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1425d20c-1dde-4d40-b44e-490af34d6f0c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1425d20c-1dde-4d40-b44e-490af34d6f0c" (UID: "1425d20c-1dde-4d40-b44e-490af34d6f0c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 16:56:14.422284 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:14.422238 2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1425d20c-1dde-4d40-b44e-490af34d6f0c-openshift-service-ca\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:56:14.422284 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:14.422272 2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1425d20c-1dde-4d40-b44e-490af34d6f0c-maas-api-tls\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:56:14.422284 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:14.422281 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qg5l\" (UniqueName: \"kubernetes.io/projected/1425d20c-1dde-4d40-b44e-490af34d6f0c-kube-api-access-7qg5l\") on node \"ip-10-0-135-114.ec2.internal\" DevicePath \"\"" Jul 09 16:56:14.467934 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:14.467861 2569 generic.go:356] "Generic (PLEG): container finished" podID="1425d20c-1dde-4d40-b44e-490af34d6f0c" containerID="ebb7bf06497de4907fb130168be50d3f2eded0370052aa202e3f88cbd1cd5a77" exitCode=0 Jul 09 16:56:14.467934 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:14.467935 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6bf011-89fc49d7c-cbp7x" Jul 09 16:56:14.468195 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:14.467945 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6bf011-89fc49d7c-cbp7x" event={"ID":"1425d20c-1dde-4d40-b44e-490af34d6f0c","Type":"ContainerDied","Data":"ebb7bf06497de4907fb130168be50d3f2eded0370052aa202e3f88cbd1cd5a77"} Jul 09 16:56:14.468195 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:14.468009 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6bf011-89fc49d7c-cbp7x" event={"ID":"1425d20c-1dde-4d40-b44e-490af34d6f0c","Type":"ContainerDied","Data":"6b31cc4f3c016b5d89d43489929136cbf3e741a440d4c5be433062fe41662760"} Jul 09 16:56:14.468195 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:14.468039 2569 scope.go:117] "RemoveContainer" containerID="ebb7bf06497de4907fb130168be50d3f2eded0370052aa202e3f88cbd1cd5a77" Jul 09 16:56:14.479204 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:14.479180 2569 scope.go:117] "RemoveContainer" containerID="ebb7bf06497de4907fb130168be50d3f2eded0370052aa202e3f88cbd1cd5a77" Jul 09 16:56:14.479500 ip-10-0-135-114 kubenswrapper[2569]: E0709 16:56:14.479481 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebb7bf06497de4907fb130168be50d3f2eded0370052aa202e3f88cbd1cd5a77\": container with ID starting with ebb7bf06497de4907fb130168be50d3f2eded0370052aa202e3f88cbd1cd5a77 not found: ID does not exist" containerID="ebb7bf06497de4907fb130168be50d3f2eded0370052aa202e3f88cbd1cd5a77" Jul 09 16:56:14.479571 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:14.479514 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebb7bf06497de4907fb130168be50d3f2eded0370052aa202e3f88cbd1cd5a77"} err="failed to get container status \"ebb7bf06497de4907fb130168be50d3f2eded0370052aa202e3f88cbd1cd5a77\": rpc error: code = NotFound desc = could not find container \"ebb7bf06497de4907fb130168be50d3f2eded0370052aa202e3f88cbd1cd5a77\": container with ID starting with ebb7bf06497de4907fb130168be50d3f2eded0370052aa202e3f88cbd1cd5a77 not found: ID does not exist" Jul 09 16:56:14.490644 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:14.490618 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-6bf011-89fc49d7c-cbp7x"] Jul 09 16:56:14.494678 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:14.494656 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-6bf011-89fc49d7c-cbp7x"] Jul 09 16:56:15.389690 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:15.389657 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1425d20c-1dde-4d40-b44e-490af34d6f0c" path="/var/lib/kubelet/pods/1425d20c-1dde-4d40-b44e-490af34d6f0c/volumes" Jul 09 16:56:25.780539 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:25.780509 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-64fcf5b9b9-79wkk_08a340b4-8031-4dd5-b4e9-dd683adeaa99/manager/0.log" Jul 09 16:56:26.136367 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:26.136339 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7df8c55954-fxg9l_880f381d-9f12-4935-93d6-46862471d4bc/copy-manifests/0.log" Jul 09 16:56:26.149539 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:26.149510 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7df8c55954-fxg9l_880f381d-9f12-4935-93d6-46862471d4bc/manager/0.log" Jul 09 16:56:26.731593 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:26.731559 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-54556c5fb4-6kntc_68ff35f9-7681-411d-ad27-8f49d787c5a0/maas-api/0.log" Jul 09 16:56:26.849931 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:26.849902 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-7jfm6_aea78521-1cd6-45a6-b7e4-df4d1d5cdd44/postgres/0.log" Jul 09 16:56:28.397638 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:28.397588 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-vdkcb_e7fce909-26bb-4713-bac8-a2a046e4c5b0/manager/0.log" Jul 09 16:56:28.632356 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:28.632319 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-xkzkb_0847f205-b153-4da8-8434-ec8556afa22c/registry-server/0.log" Jul 09 16:56:28.985821 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:28.985792 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-q2k74_97412e3e-1a86-48b2-b77e-c8620809630d/manager/0.log" Jul 09 16:56:29.955155 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:29.955124 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-t2tml_9972f323-cbe6-4d1b-bf97-f7f1261f3d74/payload-pre-processing/0.log" Jul 09 16:56:30.076121 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:30.076084 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-9tvlf_58f7fc44-58a9-45a4-95d3-12d4983bb487/payload-processing/0.log" Jul 09 16:56:30.192558 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:30.192525 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-84d7f6c55d-59vjr_fe900560-5002-4708-98e4-237e41014fbf/router/0.log" Jul 09 16:56:30.665514 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:30.665483 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-t8wx2_43458597-f606-4740-b30d-c3b6a3f82245/main/0.log" Jul 09 16:56:30.899021 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:30.898996 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnmpk9_9b052225-8cfe-4adb-b6e2-e1e0195efa56/main/0.log" Jul 09 16:56:31.130096 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:31.130067 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-cp5nl_3fa5d4e6-ec46-432f-a0f8-e37e910a2725/main/0.log" Jul 09 16:56:37.994326 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:37.994298 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-f6wzq_3bd017f3-bffb-4016-bdac-48c6fe9fa352/global-pull-secret-syncer/0.log" Jul 09 16:56:38.162329 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:38.162300 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-zj5qc_e6d9b732-f9d0-48e1-b3a2-b9641eda1c0b/konnectivity-agent/0.log" Jul 09 16:56:38.186820 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:38.186773 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-114.ec2.internal_ef080aa212d8ff70e6e7c4e03505fc8a/haproxy/0.log" Jul 09 16:56:43.243160 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:43.243130 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-vdkcb_e7fce909-26bb-4713-bac8-a2a046e4c5b0/manager/0.log" Jul 09 16:56:43.318196 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:43.318164 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-xkzkb_0847f205-b153-4da8-8434-ec8556afa22c/registry-server/0.log" Jul 09 16:56:43.514832 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:43.514743 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-q2k74_97412e3e-1a86-48b2-b77e-c8620809630d/manager/0.log" Jul 09 16:56:45.271520 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:45.271483 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-vdbkl_022557ee-2bed-48ef-a717-3bf8a3c53dc5/cluster-monitoring-operator/0.log" Jul 09 16:56:45.621918 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:45.621889 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-c9pj8_792741f3-785c-420e-98c5-01167e3649ad/node-exporter/0.log" Jul 09 16:56:45.648466 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:45.648437 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-c9pj8_792741f3-785c-420e-98c5-01167e3649ad/kube-rbac-proxy/0.log" Jul 09 16:56:45.674889 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:45.674852 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-c9pj8_792741f3-785c-420e-98c5-01167e3649ad/init-textfile/0.log" Jul 09 16:56:47.659767 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:47.659731 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-b7vxd/perf-node-gather-daemonset-52wc4"] Jul 09 16:56:47.660182 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:47.660166 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ce6e48c-7360-48be-8f09-8dc677a0a8c2" containerName="main" Jul 09 16:56:47.660182 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:47.660180 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ce6e48c-7360-48be-8f09-8dc677a0a8c2" containerName="main" Jul 09 16:56:47.660261 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:47.660197 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92b20001-ee0f-43ec-a9ab-8ddb1a6e0cd1" containerName="maas-api" Jul 09 16:56:47.660261 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:47.660203 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="92b20001-ee0f-43ec-a9ab-8ddb1a6e0cd1" containerName="maas-api" Jul 09 16:56:47.660261 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:47.660219 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1425d20c-1dde-4d40-b44e-490af34d6f0c" containerName="maas-api" Jul 09 16:56:47.660261 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:47.660224 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="1425d20c-1dde-4d40-b44e-490af34d6f0c" containerName="maas-api" Jul 09 16:56:47.660386 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:47.660296 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="1425d20c-1dde-4d40-b44e-490af34d6f0c" containerName="maas-api" Jul 09 16:56:47.660386 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:47.660306 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ce6e48c-7360-48be-8f09-8dc677a0a8c2" containerName="main" Jul 09 16:56:47.660386 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:47.660313 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="92b20001-ee0f-43ec-a9ab-8ddb1a6e0cd1" containerName="maas-api" Jul 09 16:56:47.663652 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:47.663634 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-b7vxd/perf-node-gather-daemonset-52wc4" Jul 09 16:56:47.666232 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:47.666202 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-b7vxd\"/\"default-dockercfg-w4l5d\"" Jul 09 16:56:47.667350 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:47.667319 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-b7vxd\"/\"openshift-service-ca.crt\"" Jul 09 16:56:47.667505 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:47.667362 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-b7vxd\"/\"kube-root-ca.crt\"" Jul 09 16:56:47.673150 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:47.673126 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-b7vxd/perf-node-gather-daemonset-52wc4"] Jul 09 16:56:47.707867 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:47.707837 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6cf856758c-2hrtq_f8eb4809-6734-4b9d-b89f-396679014b4a/networking-console-plugin/0.log" Jul 09 16:56:47.816840 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:47.816801 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/544d8568-b1c6-4c70-8d85-eddadfd991c4-podres\") pod \"perf-node-gather-daemonset-52wc4\" (UID: \"544d8568-b1c6-4c70-8d85-eddadfd991c4\") " pod="openshift-must-gather-b7vxd/perf-node-gather-daemonset-52wc4" Jul 09 16:56:47.816840 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:47.816840 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xt7rz\" (UniqueName: \"kubernetes.io/projected/544d8568-b1c6-4c70-8d85-eddadfd991c4-kube-api-access-xt7rz\") pod \"perf-node-gather-daemonset-52wc4\" (UID: \"544d8568-b1c6-4c70-8d85-eddadfd991c4\") " pod="openshift-must-gather-b7vxd/perf-node-gather-daemonset-52wc4" Jul 09 16:56:47.817082 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:47.816975 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/544d8568-b1c6-4c70-8d85-eddadfd991c4-proc\") pod \"perf-node-gather-daemonset-52wc4\" (UID: \"544d8568-b1c6-4c70-8d85-eddadfd991c4\") " pod="openshift-must-gather-b7vxd/perf-node-gather-daemonset-52wc4" Jul 09 16:56:47.817082 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:47.817014 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/544d8568-b1c6-4c70-8d85-eddadfd991c4-sys\") pod \"perf-node-gather-daemonset-52wc4\" (UID: \"544d8568-b1c6-4c70-8d85-eddadfd991c4\") " pod="openshift-must-gather-b7vxd/perf-node-gather-daemonset-52wc4" Jul 09 16:56:47.817082 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:47.817051 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/544d8568-b1c6-4c70-8d85-eddadfd991c4-lib-modules\") pod \"perf-node-gather-daemonset-52wc4\" (UID: \"544d8568-b1c6-4c70-8d85-eddadfd991c4\") " pod="openshift-must-gather-b7vxd/perf-node-gather-daemonset-52wc4" Jul 09 16:56:47.917629 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:47.917513 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/544d8568-b1c6-4c70-8d85-eddadfd991c4-proc\") pod \"perf-node-gather-daemonset-52wc4\" (UID: \"544d8568-b1c6-4c70-8d85-eddadfd991c4\") " pod="openshift-must-gather-b7vxd/perf-node-gather-daemonset-52wc4" Jul 09 16:56:47.917629 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:47.917564 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/544d8568-b1c6-4c70-8d85-eddadfd991c4-sys\") pod \"perf-node-gather-daemonset-52wc4\" (UID: \"544d8568-b1c6-4c70-8d85-eddadfd991c4\") " pod="openshift-must-gather-b7vxd/perf-node-gather-daemonset-52wc4" Jul 09 16:56:47.917629 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:47.917591 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/544d8568-b1c6-4c70-8d85-eddadfd991c4-lib-modules\") pod \"perf-node-gather-daemonset-52wc4\" (UID: \"544d8568-b1c6-4c70-8d85-eddadfd991c4\") " pod="openshift-must-gather-b7vxd/perf-node-gather-daemonset-52wc4" Jul 09 16:56:47.917960 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:47.917644 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/544d8568-b1c6-4c70-8d85-eddadfd991c4-podres\") pod \"perf-node-gather-daemonset-52wc4\" (UID: \"544d8568-b1c6-4c70-8d85-eddadfd991c4\") " pod="openshift-must-gather-b7vxd/perf-node-gather-daemonset-52wc4" Jul 09 16:56:47.917960 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:47.917664 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xt7rz\" (UniqueName: \"kubernetes.io/projected/544d8568-b1c6-4c70-8d85-eddadfd991c4-kube-api-access-xt7rz\") pod \"perf-node-gather-daemonset-52wc4\" (UID: \"544d8568-b1c6-4c70-8d85-eddadfd991c4\") " pod="openshift-must-gather-b7vxd/perf-node-gather-daemonset-52wc4" Jul 09 16:56:47.917960 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:47.917674 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/544d8568-b1c6-4c70-8d85-eddadfd991c4-sys\") pod \"perf-node-gather-daemonset-52wc4\" (UID: \"544d8568-b1c6-4c70-8d85-eddadfd991c4\") " pod="openshift-must-gather-b7vxd/perf-node-gather-daemonset-52wc4" Jul 09 16:56:47.917960 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:47.917678 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/544d8568-b1c6-4c70-8d85-eddadfd991c4-proc\") pod \"perf-node-gather-daemonset-52wc4\" (UID: \"544d8568-b1c6-4c70-8d85-eddadfd991c4\") " pod="openshift-must-gather-b7vxd/perf-node-gather-daemonset-52wc4" Jul 09 16:56:47.917960 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:47.917757 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/544d8568-b1c6-4c70-8d85-eddadfd991c4-podres\") pod \"perf-node-gather-daemonset-52wc4\" (UID: \"544d8568-b1c6-4c70-8d85-eddadfd991c4\") " pod="openshift-must-gather-b7vxd/perf-node-gather-daemonset-52wc4" Jul 09 16:56:47.917960 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:47.917798 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/544d8568-b1c6-4c70-8d85-eddadfd991c4-lib-modules\") pod \"perf-node-gather-daemonset-52wc4\" (UID: \"544d8568-b1c6-4c70-8d85-eddadfd991c4\") " pod="openshift-must-gather-b7vxd/perf-node-gather-daemonset-52wc4" Jul 09 16:56:47.925969 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:47.925937 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xt7rz\" (UniqueName: \"kubernetes.io/projected/544d8568-b1c6-4c70-8d85-eddadfd991c4-kube-api-access-xt7rz\") pod \"perf-node-gather-daemonset-52wc4\" (UID: \"544d8568-b1c6-4c70-8d85-eddadfd991c4\") " pod="openshift-must-gather-b7vxd/perf-node-gather-daemonset-52wc4" Jul 09 16:56:47.974142 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:47.974111 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-b7vxd/perf-node-gather-daemonset-52wc4" Jul 09 16:56:48.113495 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:48.113468 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-b7vxd/perf-node-gather-daemonset-52wc4"] Jul 09 16:56:48.116176 ip-10-0-135-114 kubenswrapper[2569]: W0709 16:56:48.116149 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod544d8568_b1c6_4c70_8d85_eddadfd991c4.slice/crio-fe82fc1edafa160de1ca9d1448d8e2c87703f1ce8ad347dc38277fbb2d285378 WatchSource:0}: Error finding container fe82fc1edafa160de1ca9d1448d8e2c87703f1ce8ad347dc38277fbb2d285378: Status 404 returned error can't find the container with id fe82fc1edafa160de1ca9d1448d8e2c87703f1ce8ad347dc38277fbb2d285378 Jul 09 16:56:48.374965 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:48.374935 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-nzrk9_a284ea64-610c-40f7-837b-fa0975c0a290/console-operator/2.log" Jul 09 16:56:48.379403 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:48.379378 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-nzrk9_a284ea64-610c-40f7-837b-fa0975c0a290/console-operator/3.log" Jul 09 16:56:48.632940 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:48.632902 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b7vxd/perf-node-gather-daemonset-52wc4" event={"ID":"544d8568-b1c6-4c70-8d85-eddadfd991c4","Type":"ContainerStarted","Data":"c426b1da25f132cab8ae30c2ec296fcc98acf3660d4093e2f6ec39b6d064683b"} Jul 09 16:56:48.633132 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:48.632961 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b7vxd/perf-node-gather-daemonset-52wc4" event={"ID":"544d8568-b1c6-4c70-8d85-eddadfd991c4","Type":"ContainerStarted","Data":"fe82fc1edafa160de1ca9d1448d8e2c87703f1ce8ad347dc38277fbb2d285378"} Jul 09 16:56:48.633132 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:48.633033 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-b7vxd/perf-node-gather-daemonset-52wc4" Jul 09 16:56:48.653909 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:48.653810 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-b7vxd/perf-node-gather-daemonset-52wc4" podStartSLOduration=1.653787828 podStartE2EDuration="1.653787828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 16:56:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:56:48.650196305 +0000 UTC m=+2493.843302697" watchObservedRunningTime="2026-07-09 16:56:48.653787828 +0000 UTC m=+2493.846894216" Jul 09 16:56:49.533376 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:49.533340 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78495597f6-vzds7_4b943693-db9f-4291-90a6-1cf6cc00a714/volume-data-source-validator/0.log" Jul 09 16:56:50.669477 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:50.669439 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-q8zll_e1cf555d-4290-4caa-a5f6-eaca63d05a1d/dns/0.log" Jul 09 16:56:50.690859 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:50.690826 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-q8zll_e1cf555d-4290-4caa-a5f6-eaca63d05a1d/kube-rbac-proxy/0.log" Jul 09 16:56:50.773002 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:50.772973 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xx9mg_88be5834-3fd7-45da-83d5-4cb1d0f84c75/dns-node-resolver/0.log" Jul 09 16:56:51.376420 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:51.376391 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jj6qs_c17a000a-a79a-4dd8-af71-1fc47077bea7/node-ca/0.log" Jul 09 16:56:52.588396 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:52.588366 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-t2tml_9972f323-cbe6-4d1b-bf97-f7f1261f3d74/payload-pre-processing/0.log" Jul 09 16:56:52.622082 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:52.622048 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-9tvlf_58f7fc44-58a9-45a4-95d3-12d4983bb487/payload-processing/0.log" Jul 09 16:56:52.645803 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:52.645775 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-84d7f6c55d-59vjr_fe900560-5002-4708-98e4-237e41014fbf/router/0.log" Jul 09 16:56:53.333519 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:53.333493 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-4rch5_cc6ebce3-3ee7-4b8d-a11d-1cb084a6b780/serve-healthcheck-canary/0.log" Jul 09 16:56:53.867868 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:53.867839 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-6jksf_bb610783-4a86-4291-a9f0-42deb83b322d/insights-operator/0.log" Jul 09 16:56:53.868789 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:53.868771 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-6jksf_bb610783-4a86-4291-a9f0-42deb83b322d/insights-operator/1.log" Jul 09 16:56:53.966722 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:53.966696 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rkqp6_031a3833-1870-44a3-abfb-9a840df32ac3/kube-rbac-proxy/0.log" Jul 09 16:56:53.988011 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:53.987982 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rkqp6_031a3833-1870-44a3-abfb-9a840df32ac3/exporter/0.log" Jul 09 16:56:54.010863 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:54.010837 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rkqp6_031a3833-1870-44a3-abfb-9a840df32ac3/extractor/0.log" Jul 09 16:56:54.647762 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:54.647731 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-b7vxd/perf-node-gather-daemonset-52wc4" Jul 09 16:56:56.395834 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:56.395806 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-64fcf5b9b9-79wkk_08a340b4-8031-4dd5-b4e9-dd683adeaa99/manager/0.log" Jul 09 16:56:56.606561 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:56.606534 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7df8c55954-fxg9l_880f381d-9f12-4935-93d6-46862471d4bc/manager/0.log" Jul 09 16:56:56.626797 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:56.626770 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7df8c55954-fxg9l_880f381d-9f12-4935-93d6-46862471d4bc/copy-manifests/0.log" Jul 09 16:56:58.022149 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:58.022118 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5b747dc9cb-w7zdv_92c8265d-4487-47f0-a924-e3990dcfbd87/manager/0.log" Jul 09 16:56:58.902655 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:58.902554 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-t2tml_9972f323-cbe6-4d1b-bf97-f7f1261f3d74/payload-pre-processing/0.log" Jul 09 16:56:58.923778 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:58.923739 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-9tvlf_58f7fc44-58a9-45a4-95d3-12d4983bb487/payload-processing/0.log" Jul 09 16:56:58.943596 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:56:58.943564 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-84d7f6c55d-59vjr_fe900560-5002-4708-98e4-237e41014fbf/router/0.log" Jul 09 16:57:00.317726 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:57:00.317612 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-t2tml_9972f323-cbe6-4d1b-bf97-f7f1261f3d74/payload-pre-processing/0.log" Jul 09 16:57:00.343848 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:57:00.343816 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-9tvlf_58f7fc44-58a9-45a4-95d3-12d4983bb487/payload-processing/0.log" Jul 09 16:57:00.373885 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:57:00.373851 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-84d7f6c55d-59vjr_fe900560-5002-4708-98e4-237e41014fbf/router/0.log" Jul 09 16:57:03.439570 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:57:03.439538 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-fr9br_c2cc6efd-36ec-42da-9025-c01334ab3bc0/migrator/0.log" Jul 09 16:57:03.462074 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:57:03.462044 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-fr9br_c2cc6efd-36ec-42da-9025-c01334ab3bc0/graceful-termination/0.log" Jul 09 16:57:03.823920 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:57:03.823785 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-xjcmf_f54c8925-c09f-46d8-84fb-9f3af042dc41/kube-storage-version-migrator-operator/1.log" Jul 09 16:57:03.824849 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:57:03.824826 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-xjcmf_f54c8925-c09f-46d8-84fb-9f3af042dc41/kube-storage-version-migrator-operator/0.log" Jul 09 16:57:04.949524 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:57:04.949499 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6q72v_1a7bbb58-c6a9-4763-a12b-c2ae62a5beaf/kube-multus-additional-cni-plugins/0.log" Jul 09 16:57:04.971112 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:57:04.971078 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6q72v_1a7bbb58-c6a9-4763-a12b-c2ae62a5beaf/egress-router-binary-copy/0.log" Jul 09 16:57:04.995861 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:57:04.995832 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6q72v_1a7bbb58-c6a9-4763-a12b-c2ae62a5beaf/cni-plugins/0.log" Jul 09 16:57:05.017608 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:57:05.017578 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6q72v_1a7bbb58-c6a9-4763-a12b-c2ae62a5beaf/bond-cni-plugin/0.log" Jul 09 16:57:05.039828 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:57:05.039805 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6q72v_1a7bbb58-c6a9-4763-a12b-c2ae62a5beaf/routeoverride-cni/0.log" Jul 09 16:57:05.060596 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:57:05.060567 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6q72v_1a7bbb58-c6a9-4763-a12b-c2ae62a5beaf/whereabouts-cni-bincopy/0.log" Jul 09 16:57:05.084087 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:57:05.084063 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6q72v_1a7bbb58-c6a9-4763-a12b-c2ae62a5beaf/whereabouts-cni/0.log" Jul 09 16:57:05.350446 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:57:05.350416 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-xrbgl_bed43223-f252-48e2-b014-b917007e39dd/kube-multus/0.log" Jul 09 16:57:05.375618 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:57:05.375595 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-b7sxk_6f45ad4f-0e3f-432e-8328-715ec5f97a69/network-metrics-daemon/0.log" Jul 09 16:57:05.397346 ip-10-0-135-114 kubenswrapper[2569]: I0709 16:57:05.397317 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-b7sxk_6f45ad4f-0e3f-432e-8328-715ec5f97a69/kube-rbac-proxy/0.log"