Jul 22 21:03:50.821537 ip-10-0-135-37 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 22 21:03:50.821550 ip-10-0-135-37 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 22 21:03:50.821560 ip-10-0-135-37 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 22 21:03:50.821898 ip-10-0-135-37 systemd[1]: Failed to start Kubernetes Kubelet. Jul 22 21:04:00.841862 ip-10-0-135-37 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 22 21:04:00.841880 ip-10-0-135-37 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot e015b86fd512404aba03f612d44b34fc -- Jul 22 21:05:44.775444 ip-10-0-135-37 systemd[1]: Starting Kubernetes Kubelet... Jul 22 21:05:45.151328 ip-10-0-135-37 kubenswrapper[2572]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 22 21:05:45.151328 ip-10-0-135-37 kubenswrapper[2572]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 22 21:05:45.151328 ip-10-0-135-37 kubenswrapper[2572]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 22 21:05:45.151328 ip-10-0-135-37 kubenswrapper[2572]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 22 21:05:45.151328 ip-10-0-135-37 kubenswrapper[2572]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 22 21:05:45.152292 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.152110 2572 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 22 21:05:45.157119 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157103 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 21:05:45.157119 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157120 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 21:05:45.157208 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157124 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 21:05:45.157208 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157127 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 21:05:45.157208 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157131 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 21:05:45.157208 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157134 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 21:05:45.157208 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157137 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 21:05:45.157208 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157140 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 21:05:45.157208 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157143 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 21:05:45.157208 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157146 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 21:05:45.157208 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157149 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 21:05:45.157208 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157152 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 21:05:45.157208 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157154 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 21:05:45.157208 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157178 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 21:05:45.157208 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157183 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 21:05:45.157208 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157187 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 21:05:45.157208 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157191 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 21:05:45.157208 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157195 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 21:05:45.157208 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157197 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 21:05:45.157208 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157200 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 21:05:45.157208 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157203 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 21:05:45.157652 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157207 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 21:05:45.157652 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157211 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 21:05:45.157652 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157215 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 21:05:45.157652 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157218 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 21:05:45.157652 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157221 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 21:05:45.157652 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157224 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 21:05:45.157652 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157226 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 21:05:45.157652 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157229 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 21:05:45.157652 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157231 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 21:05:45.157652 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157234 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 21:05:45.157652 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157236 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 21:05:45.157652 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157239 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 21:05:45.157652 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157243 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 21:05:45.157652 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157247 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 21:05:45.157652 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157249 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 21:05:45.157652 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157252 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 21:05:45.157652 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157254 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 21:05:45.157652 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157257 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 21:05:45.157652 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157259 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 21:05:45.158122 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157262 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 21:05:45.158122 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157264 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 21:05:45.158122 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157267 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 21:05:45.158122 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157269 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 21:05:45.158122 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157272 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 21:05:45.158122 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157274 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 21:05:45.158122 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157277 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 21:05:45.158122 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157279 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 21:05:45.158122 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157282 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 21:05:45.158122 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157284 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 21:05:45.158122 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157287 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 21:05:45.158122 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157290 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 21:05:45.158122 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157292 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 21:05:45.158122 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157295 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 21:05:45.158122 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157297 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 21:05:45.158122 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157300 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 21:05:45.158122 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157302 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 21:05:45.158122 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157305 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 21:05:45.158122 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157307 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 21:05:45.158122 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157309 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 21:05:45.158633 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157312 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 21:05:45.158633 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157314 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 21:05:45.158633 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157317 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 21:05:45.158633 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157319 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 21:05:45.158633 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157322 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 21:05:45.158633 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157324 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 21:05:45.158633 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157326 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 21:05:45.158633 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157329 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 21:05:45.158633 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157332 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 21:05:45.158633 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157334 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 21:05:45.158633 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157337 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 21:05:45.158633 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157343 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 21:05:45.158633 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157346 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 21:05:45.158633 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157348 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 21:05:45.158633 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157351 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 21:05:45.158633 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157353 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 21:05:45.158633 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157356 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 21:05:45.158633 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157358 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 21:05:45.158633 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157361 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 21:05:45.158633 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157364 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 21:05:45.159098 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157366 2572 feature_gate.go:328] unrecognized feature gate: Example Jul 22 21:05:45.159098 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157368 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 21:05:45.159098 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157371 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 21:05:45.159098 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157374 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 21:05:45.159098 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157376 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 21:05:45.159098 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157379 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 21:05:45.159098 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157381 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 21:05:45.159098 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157384 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 21:05:45.159098 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157386 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 21:05:45.159098 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157389 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 21:05:45.159098 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157391 2572 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 21:05:45.159098 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157394 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 21:05:45.159098 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157396 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 21:05:45.159098 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157399 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 21:05:45.159098 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157401 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 21:05:45.159098 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157404 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 21:05:45.159098 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157406 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 21:05:45.159098 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157408 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 21:05:45.159098 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.157411 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 21:05:45.159098 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158669 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 21:05:45.159583 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158676 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 21:05:45.159583 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158681 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 21:05:45.159583 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158684 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 21:05:45.159583 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158687 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 21:05:45.159583 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158690 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 21:05:45.159583 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158693 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 21:05:45.159583 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158696 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 21:05:45.159583 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158699 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 21:05:45.159583 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158702 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 21:05:45.159583 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158704 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 21:05:45.159583 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158707 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 21:05:45.159583 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158709 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 21:05:45.159583 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158713 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 21:05:45.159583 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158717 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 21:05:45.159583 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158722 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 21:05:45.159583 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158725 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 21:05:45.159583 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158727 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 21:05:45.159583 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158730 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 21:05:45.159583 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158734 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 21:05:45.160055 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158737 2572 feature_gate.go:328] unrecognized feature gate: Example Jul 22 21:05:45.160055 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158739 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 21:05:45.160055 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158742 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 21:05:45.160055 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158745 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 21:05:45.160055 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158748 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 21:05:45.160055 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158750 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 21:05:45.160055 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158752 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 21:05:45.160055 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158755 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 21:05:45.160055 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158757 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 21:05:45.160055 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158759 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 21:05:45.160055 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158762 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 21:05:45.160055 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158765 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 21:05:45.160055 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158767 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 21:05:45.160055 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158770 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 21:05:45.160055 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158772 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 21:05:45.160055 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158775 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 21:05:45.160055 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158777 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 21:05:45.160055 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158779 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 21:05:45.160055 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158782 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 21:05:45.160055 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158785 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 21:05:45.160556 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158789 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 21:05:45.160556 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158792 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 21:05:45.160556 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158794 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 21:05:45.160556 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158797 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 21:05:45.160556 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158799 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 21:05:45.160556 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158802 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 21:05:45.160556 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158804 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 21:05:45.160556 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158807 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 21:05:45.160556 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158810 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 21:05:45.160556 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158812 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 21:05:45.160556 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158814 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 21:05:45.160556 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158817 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 21:05:45.160556 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158819 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 21:05:45.160556 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158822 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 21:05:45.160556 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158824 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 21:05:45.160556 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158826 2572 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 21:05:45.160556 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158829 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 21:05:45.160556 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158831 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 21:05:45.160556 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158833 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 21:05:45.160556 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158836 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 21:05:45.161062 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158838 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 21:05:45.161062 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158841 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 21:05:45.161062 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158844 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 21:05:45.161062 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158847 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 21:05:45.161062 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158849 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 21:05:45.161062 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158851 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 21:05:45.161062 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158854 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 21:05:45.161062 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158856 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 21:05:45.161062 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158858 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 21:05:45.161062 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158861 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 21:05:45.161062 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158863 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 21:05:45.161062 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158866 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 21:05:45.161062 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158868 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 21:05:45.161062 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158870 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 21:05:45.161062 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158873 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 21:05:45.161062 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158875 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 21:05:45.161062 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158877 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 21:05:45.161062 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158880 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 21:05:45.161062 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158882 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 21:05:45.161062 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158885 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 21:05:45.161566 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158887 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 21:05:45.161566 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158890 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 21:05:45.161566 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158893 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 21:05:45.161566 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158895 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 21:05:45.161566 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158898 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 21:05:45.161566 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158900 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 21:05:45.161566 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158903 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 21:05:45.161566 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158905 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 21:05:45.161566 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158908 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 21:05:45.161566 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158910 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 21:05:45.161566 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158914 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 21:05:45.161566 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158917 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 21:05:45.161566 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158919 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 21:05:45.161566 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158922 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 21:05:45.161566 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158925 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 21:05:45.161566 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158927 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 21:05:45.161566 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158930 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 21:05:45.161566 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158932 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 21:05:45.161566 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.158934 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 21:05:45.161566 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159304 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 21:05:45.162035 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159310 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 21:05:45.162035 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159313 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 21:05:45.162035 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159316 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 21:05:45.162035 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159320 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 21:05:45.162035 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159322 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 21:05:45.162035 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159325 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 21:05:45.162035 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159328 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 21:05:45.162035 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159331 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 21:05:45.162035 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159333 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 21:05:45.162035 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159336 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 21:05:45.162035 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159339 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 21:05:45.162035 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159342 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 21:05:45.162035 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159344 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 21:05:45.162035 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159347 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 21:05:45.162035 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159349 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 21:05:45.162035 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159352 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 21:05:45.162035 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159354 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 21:05:45.162035 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159356 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 21:05:45.162035 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159359 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 21:05:45.162035 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159362 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 21:05:45.162534 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159364 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 21:05:45.162534 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159367 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 21:05:45.162534 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159369 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 21:05:45.162534 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159371 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 21:05:45.162534 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159375 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 21:05:45.162534 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159379 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 21:05:45.162534 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159382 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 21:05:45.162534 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159384 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 21:05:45.162534 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159387 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 21:05:45.162534 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159389 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 21:05:45.162534 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159392 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 21:05:45.162534 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159394 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 21:05:45.162534 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159397 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 21:05:45.162534 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159399 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 21:05:45.162534 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159402 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 21:05:45.162534 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159404 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 21:05:45.162534 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159408 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 21:05:45.162534 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159410 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 21:05:45.162534 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159413 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 21:05:45.162534 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159415 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 21:05:45.163023 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159417 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 21:05:45.163023 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159420 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 21:05:45.163023 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159423 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 21:05:45.163023 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159425 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 21:05:45.163023 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159428 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 21:05:45.163023 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159430 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 21:05:45.163023 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159432 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 21:05:45.163023 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159435 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 21:05:45.163023 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159437 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 21:05:45.163023 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159439 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 21:05:45.163023 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159442 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 21:05:45.163023 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159444 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 21:05:45.163023 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159446 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 21:05:45.163023 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159449 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 21:05:45.163023 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159451 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 21:05:45.163023 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159453 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 21:05:45.163023 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159456 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 21:05:45.163023 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159458 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 21:05:45.163023 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159461 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 21:05:45.163494 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159463 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 21:05:45.163494 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159470 2572 feature_gate.go:328] unrecognized feature gate: Example Jul 22 21:05:45.163494 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159472 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 21:05:45.163494 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159475 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 21:05:45.163494 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159477 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 21:05:45.163494 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159480 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 21:05:45.163494 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159482 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 21:05:45.163494 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159485 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 21:05:45.163494 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159488 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 21:05:45.163494 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159490 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 21:05:45.163494 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159493 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 21:05:45.163494 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159496 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 21:05:45.163494 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159500 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 21:05:45.163494 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159504 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 21:05:45.163494 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159507 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 21:05:45.163494 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159510 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 21:05:45.163494 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159512 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 21:05:45.163494 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159516 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 21:05:45.163494 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159519 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 21:05:45.164069 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159522 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 21:05:45.164069 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159525 2572 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 21:05:45.164069 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159528 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 21:05:45.164069 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159530 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 21:05:45.164069 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159533 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 21:05:45.164069 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159536 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 21:05:45.164069 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159538 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 21:05:45.164069 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159541 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 21:05:45.164069 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159543 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 21:05:45.164069 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159547 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 21:05:45.164069 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159549 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 21:05:45.164069 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159551 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 21:05:45.164069 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159554 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 21:05:45.164069 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159556 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 21:05:45.164069 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159560 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 21:05:45.164069 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159563 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 21:05:45.164069 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159566 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 21:05:45.164069 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159568 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 21:05:45.164069 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159571 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 21:05:45.164069 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.159573 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 21:05:45.164069 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159639 2572 flags.go:64] FLAG: --address="0.0.0.0" Jul 22 21:05:45.164602 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159646 2572 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 22 21:05:45.164602 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159652 2572 flags.go:64] FLAG: --anonymous-auth="true" Jul 22 21:05:45.164602 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159656 2572 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 22 21:05:45.164602 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159660 2572 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 22 21:05:45.164602 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159664 2572 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 22 21:05:45.164602 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159667 2572 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 22 21:05:45.164602 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159672 2572 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 22 21:05:45.164602 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159675 2572 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 22 21:05:45.164602 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159678 2572 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 22 21:05:45.164602 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159681 2572 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 22 21:05:45.164602 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159684 2572 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 22 21:05:45.164602 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159687 2572 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 22 21:05:45.164602 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159690 2572 flags.go:64] FLAG: --cgroup-root="" Jul 22 21:05:45.164602 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159692 2572 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 22 21:05:45.164602 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159695 2572 flags.go:64] FLAG: --client-ca-file="" Jul 22 21:05:45.164602 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159698 2572 flags.go:64] FLAG: --cloud-provider="external" Jul 22 21:05:45.164602 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159701 2572 flags.go:64] FLAG: --cluster-dns="[]" Jul 22 21:05:45.164602 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159705 2572 flags.go:64] FLAG: --cluster-domain="" Jul 22 21:05:45.164602 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159708 2572 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 22 21:05:45.164602 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159711 2572 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 22 21:05:45.164602 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159717 2572 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 22 21:05:45.164602 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159720 2572 flags.go:64] FLAG: --container-log-max-files="5" Jul 22 21:05:45.164602 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159723 2572 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 22 21:05:45.164602 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159726 2572 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 22 21:05:45.165188 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159729 2572 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 22 21:05:45.165188 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159733 2572 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 22 21:05:45.165188 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159736 2572 flags.go:64] FLAG: --contention-profiling="false" Jul 22 21:05:45.165188 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159739 2572 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 22 21:05:45.165188 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159742 2572 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 22 21:05:45.165188 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159745 2572 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 22 21:05:45.165188 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159748 2572 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 22 21:05:45.165188 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159756 2572 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 22 21:05:45.165188 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159759 2572 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 22 21:05:45.165188 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159761 2572 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 22 21:05:45.165188 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159764 2572 flags.go:64] FLAG: --enable-load-reader="false" Jul 22 21:05:45.165188 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159767 2572 flags.go:64] FLAG: --enable-server="true" Jul 22 21:05:45.165188 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159770 2572 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 22 21:05:45.165188 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159775 2572 flags.go:64] FLAG: --event-burst="100" Jul 22 21:05:45.165188 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159778 2572 flags.go:64] FLAG: --event-qps="50" Jul 22 21:05:45.165188 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159780 2572 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 22 21:05:45.165188 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159783 2572 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 22 21:05:45.165188 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159786 2572 flags.go:64] FLAG: --eviction-hard="" Jul 22 21:05:45.165188 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159789 2572 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 22 21:05:45.165188 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159792 2572 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 22 21:05:45.165188 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159794 2572 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 22 21:05:45.165188 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159797 2572 flags.go:64] FLAG: --eviction-soft="" Jul 22 21:05:45.165188 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159800 2572 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 22 21:05:45.165188 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159803 2572 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 22 21:05:45.165188 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159806 2572 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 22 21:05:45.165844 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159808 2572 flags.go:64] FLAG: --experimental-mounter-path="" Jul 22 21:05:45.165844 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159811 2572 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 22 21:05:45.165844 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159814 2572 flags.go:64] FLAG: --fail-swap-on="true" Jul 22 21:05:45.165844 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159818 2572 flags.go:64] FLAG: --feature-gates="" Jul 22 21:05:45.165844 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159822 2572 flags.go:64] FLAG: --file-check-frequency="20s" Jul 22 21:05:45.165844 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159825 2572 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 22 21:05:45.165844 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159828 2572 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 22 21:05:45.165844 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159831 2572 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 22 21:05:45.165844 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159836 2572 flags.go:64] FLAG: --healthz-port="10248" Jul 22 21:05:45.165844 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159838 2572 flags.go:64] FLAG: --help="false" Jul 22 21:05:45.165844 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159841 2572 flags.go:64] FLAG: --hostname-override="ip-10-0-135-37.ec2.internal" Jul 22 21:05:45.165844 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159844 2572 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 22 21:05:45.165844 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159847 2572 flags.go:64] FLAG: --http-check-frequency="20s" Jul 22 21:05:45.165844 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159850 2572 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 22 21:05:45.165844 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159853 2572 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 22 21:05:45.165844 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159857 2572 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 22 21:05:45.165844 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159859 2572 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 22 21:05:45.165844 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159862 2572 flags.go:64] FLAG: --image-service-endpoint="" Jul 22 21:05:45.165844 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159865 2572 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 22 21:05:45.165844 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159868 2572 flags.go:64] FLAG: --kube-api-burst="100" Jul 22 21:05:45.165844 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159871 2572 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 22 21:05:45.165844 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159874 2572 flags.go:64] FLAG: --kube-api-qps="50" Jul 22 21:05:45.165844 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159877 2572 flags.go:64] FLAG: --kube-reserved="" Jul 22 21:05:45.165844 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159879 2572 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 22 21:05:45.166521 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159882 2572 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 22 21:05:45.166521 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159885 2572 flags.go:64] FLAG: --kubelet-cgroups="" Jul 22 21:05:45.166521 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159888 2572 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 22 21:05:45.166521 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159890 2572 flags.go:64] FLAG: --lock-file="" Jul 22 21:05:45.166521 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159893 2572 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 22 21:05:45.166521 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159896 2572 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 22 21:05:45.166521 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159898 2572 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 22 21:05:45.166521 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159903 2572 flags.go:64] FLAG: --log-json-split-stream="false" Jul 22 21:05:45.166521 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159906 2572 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 22 21:05:45.166521 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159909 2572 flags.go:64] FLAG: --log-text-split-stream="false" Jul 22 21:05:45.166521 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159911 2572 flags.go:64] FLAG: --logging-format="text" Jul 22 21:05:45.166521 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159916 2572 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 22 21:05:45.166521 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159919 2572 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 22 21:05:45.166521 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159922 2572 flags.go:64] FLAG: --manifest-url="" Jul 22 21:05:45.166521 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159925 2572 flags.go:64] FLAG: --manifest-url-header="" Jul 22 21:05:45.166521 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159929 2572 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 22 21:05:45.166521 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159933 2572 flags.go:64] FLAG: --max-open-files="1000000" Jul 22 21:05:45.166521 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159936 2572 flags.go:64] FLAG: --max-pods="110" Jul 22 21:05:45.166521 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159939 2572 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 22 21:05:45.166521 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159942 2572 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 22 21:05:45.166521 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159945 2572 flags.go:64] FLAG: --memory-manager-policy="None" Jul 22 21:05:45.166521 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159948 2572 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 22 21:05:45.166521 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159951 2572 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 22 21:05:45.166521 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159953 2572 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 22 21:05:45.166521 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159956 2572 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 22 21:05:45.167121 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159963 2572 flags.go:64] FLAG: --node-status-max-images="50" Jul 22 21:05:45.167121 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159966 2572 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 22 21:05:45.167121 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159968 2572 flags.go:64] FLAG: --oom-score-adj="-999" Jul 22 21:05:45.167121 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159971 2572 flags.go:64] FLAG: --pod-cidr="" Jul 22 21:05:45.167121 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159975 2572 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 22 21:05:45.167121 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159979 2572 flags.go:64] FLAG: --pod-manifest-path="" Jul 22 21:05:45.167121 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159982 2572 flags.go:64] FLAG: --pod-max-pids="-1" Jul 22 21:05:45.167121 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159985 2572 flags.go:64] FLAG: --pods-per-core="0" Jul 22 21:05:45.167121 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159988 2572 flags.go:64] FLAG: --port="10250" Jul 22 21:05:45.167121 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159991 2572 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 22 21:05:45.167121 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159993 2572 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-07548a367cfdde0bd" Jul 22 21:05:45.167121 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159996 2572 flags.go:64] FLAG: --qos-reserved="" Jul 22 21:05:45.167121 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.159999 2572 flags.go:64] FLAG: --read-only-port="10255" Jul 22 21:05:45.167121 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.160002 2572 flags.go:64] FLAG: --register-node="true" Jul 22 21:05:45.167121 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.160004 2572 flags.go:64] FLAG: --register-with-taints="" Jul 22 21:05:45.167121 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.160008 2572 flags.go:64] FLAG: --registry-burst="10" Jul 22 21:05:45.167121 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.160010 2572 flags.go:64] FLAG: --registry-qps="5" Jul 22 21:05:45.167121 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.160013 2572 flags.go:64] FLAG: --reserved-cpus="" Jul 22 21:05:45.167121 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.160017 2572 flags.go:64] FLAG: --reserved-memory="" Jul 22 21:05:45.167121 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.160021 2572 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 22 21:05:45.167121 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.160024 2572 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 22 21:05:45.167121 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.160027 2572 flags.go:64] FLAG: --rotate-certificates="false" Jul 22 21:05:45.167121 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.160029 2572 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 22 21:05:45.167121 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.160032 2572 flags.go:64] FLAG: --runonce="false" Jul 22 21:05:45.167121 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.160036 2572 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 22 21:05:45.167730 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.160039 2572 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 22 21:05:45.167730 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.160041 2572 flags.go:64] FLAG: --seccomp-default="false" Jul 22 21:05:45.167730 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.160044 2572 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 22 21:05:45.167730 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.160047 2572 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 22 21:05:45.167730 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.160049 2572 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 22 21:05:45.167730 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.160052 2572 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 22 21:05:45.167730 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.160055 2572 flags.go:64] FLAG: --storage-driver-password="root" Jul 22 21:05:45.167730 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.160058 2572 flags.go:64] FLAG: --storage-driver-secure="false" Jul 22 21:05:45.167730 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.160061 2572 flags.go:64] FLAG: --storage-driver-table="stats" Jul 22 21:05:45.167730 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.160064 2572 flags.go:64] FLAG: --storage-driver-user="root" Jul 22 21:05:45.167730 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.160067 2572 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 22 21:05:45.167730 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.160070 2572 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 22 21:05:45.167730 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.160073 2572 flags.go:64] FLAG: --system-cgroups="" Jul 22 21:05:45.167730 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.160076 2572 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 22 21:05:45.167730 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.160081 2572 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 22 21:05:45.167730 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.160084 2572 flags.go:64] FLAG: --tls-cert-file="" Jul 22 21:05:45.167730 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.160086 2572 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 22 21:05:45.167730 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.160091 2572 flags.go:64] FLAG: --tls-min-version="" Jul 22 21:05:45.167730 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.160093 2572 flags.go:64] FLAG: --tls-private-key-file="" Jul 22 21:05:45.167730 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.160096 2572 flags.go:64] FLAG: --topology-manager-policy="none" Jul 22 21:05:45.167730 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.160098 2572 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 22 21:05:45.167730 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.160101 2572 flags.go:64] FLAG: --topology-manager-scope="container" Jul 22 21:05:45.167730 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.160104 2572 flags.go:64] FLAG: --v="2" Jul 22 21:05:45.167730 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.160107 2572 flags.go:64] FLAG: --version="false" Jul 22 21:05:45.167730 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.160111 2572 flags.go:64] FLAG: --vmodule="" Jul 22 21:05:45.168332 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.160116 2572 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 22 21:05:45.168332 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.160119 2572 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 22 21:05:45.168332 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160245 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 21:05:45.168332 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160250 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 21:05:45.168332 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160253 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 21:05:45.168332 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160256 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 21:05:45.168332 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160260 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 21:05:45.168332 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160264 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 21:05:45.168332 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160266 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 21:05:45.168332 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160269 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 21:05:45.168332 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160271 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 21:05:45.168332 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160274 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 21:05:45.168332 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160276 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 21:05:45.168332 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160279 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 21:05:45.168332 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160281 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 21:05:45.168332 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160284 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 21:05:45.168332 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160286 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 21:05:45.168332 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160288 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 21:05:45.168332 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160291 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 21:05:45.168332 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160293 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 21:05:45.169002 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160296 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 21:05:45.169002 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160298 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 21:05:45.169002 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160301 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 21:05:45.169002 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160303 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 21:05:45.169002 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160305 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 21:05:45.169002 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160308 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 21:05:45.169002 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160311 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 21:05:45.169002 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160314 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 21:05:45.169002 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160316 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 21:05:45.169002 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160318 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 21:05:45.169002 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160321 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 21:05:45.169002 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160323 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 21:05:45.169002 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160327 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 21:05:45.169002 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160329 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 21:05:45.169002 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160332 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 21:05:45.169002 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160334 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 21:05:45.169002 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160337 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 21:05:45.169002 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160339 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 21:05:45.169002 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160343 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 21:05:45.169002 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160345 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 21:05:45.169657 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160348 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 21:05:45.169657 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160350 2572 feature_gate.go:328] unrecognized feature gate: Example Jul 22 21:05:45.169657 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160352 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 21:05:45.169657 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160356 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 21:05:45.169657 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160360 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 21:05:45.169657 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160363 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 21:05:45.169657 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160365 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 21:05:45.169657 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160368 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 21:05:45.169657 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160371 2572 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 21:05:45.169657 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160373 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 21:05:45.169657 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160376 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 21:05:45.169657 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160379 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 21:05:45.169657 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160382 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 21:05:45.169657 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160385 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 21:05:45.169657 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160387 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 21:05:45.169657 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160390 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 21:05:45.169657 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160392 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 21:05:45.169657 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160394 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 21:05:45.169657 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160397 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 21:05:45.170131 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160399 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 21:05:45.170131 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160402 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 21:05:45.170131 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160404 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 21:05:45.170131 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160406 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 21:05:45.170131 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160409 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 21:05:45.170131 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160413 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 21:05:45.170131 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160415 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 21:05:45.170131 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160418 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 21:05:45.170131 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160420 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 21:05:45.170131 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160422 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 21:05:45.170131 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160425 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 21:05:45.170131 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160429 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 21:05:45.170131 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160431 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 21:05:45.170131 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160434 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 21:05:45.170131 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160436 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 21:05:45.170131 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160439 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 21:05:45.170131 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160441 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 21:05:45.170131 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160444 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 21:05:45.170131 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160447 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 21:05:45.170131 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160451 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 21:05:45.170699 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160454 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 21:05:45.170699 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160456 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 21:05:45.170699 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160459 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 21:05:45.170699 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160461 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 21:05:45.170699 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160464 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 21:05:45.170699 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160466 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 21:05:45.170699 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160469 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 21:05:45.170699 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160472 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 21:05:45.170699 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160474 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 21:05:45.170699 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160476 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 21:05:45.170699 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160479 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 21:05:45.170699 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160481 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 21:05:45.170699 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160484 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 21:05:45.170699 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160486 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 21:05:45.170699 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160488 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 21:05:45.170699 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160490 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 21:05:45.170699 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160493 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 21:05:45.170699 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160496 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 21:05:45.170699 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160499 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 21:05:45.171154 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160503 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 21:05:45.171154 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160506 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 21:05:45.171154 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.160509 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 21:05:45.171154 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.160515 2572 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 22 21:05:45.171154 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.167573 2572 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 22 21:05:45.171154 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.167586 2572 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 22 21:05:45.171154 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.167605 2572 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 22 21:05:45.171154 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.167610 2572 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 22 21:05:45.171154 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167663 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 21:05:45.171154 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167668 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 21:05:45.171154 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167671 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 21:05:45.171154 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167674 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 21:05:45.171154 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167678 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 21:05:45.171154 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167682 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 21:05:45.171534 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167685 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 21:05:45.171534 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167688 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 21:05:45.171534 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167691 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 21:05:45.171534 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167694 2572 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 21:05:45.171534 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167697 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 21:05:45.171534 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167699 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 21:05:45.171534 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167702 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 21:05:45.171534 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167705 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 21:05:45.171534 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167708 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 21:05:45.171534 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167711 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 21:05:45.171534 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167713 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 21:05:45.171534 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167715 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 21:05:45.171534 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167718 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 21:05:45.171534 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167720 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 21:05:45.171534 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167723 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 21:05:45.171534 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167725 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 21:05:45.171534 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167728 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 21:05:45.171534 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167730 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 21:05:45.171534 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167733 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 21:05:45.171534 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167735 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 21:05:45.172012 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167738 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 21:05:45.172012 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167740 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 21:05:45.172012 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167743 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 21:05:45.172012 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167747 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 21:05:45.172012 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167750 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 21:05:45.172012 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167753 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 21:05:45.172012 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167755 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 21:05:45.172012 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167758 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 21:05:45.172012 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167761 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 21:05:45.172012 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167763 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 21:05:45.172012 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167766 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 21:05:45.172012 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167768 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 21:05:45.172012 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167771 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 21:05:45.172012 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167773 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 21:05:45.172012 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167776 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 21:05:45.172012 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167778 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 21:05:45.172012 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167781 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 21:05:45.172012 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167784 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 21:05:45.172012 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167786 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 21:05:45.172012 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167788 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 21:05:45.172523 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167791 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 21:05:45.172523 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167793 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 21:05:45.172523 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167796 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 21:05:45.172523 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167798 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 21:05:45.172523 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167801 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 21:05:45.172523 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167803 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 21:05:45.172523 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167805 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 21:05:45.172523 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167808 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 21:05:45.172523 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167810 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 21:05:45.172523 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167812 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 21:05:45.172523 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167815 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 21:05:45.172523 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167817 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 21:05:45.172523 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167820 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 21:05:45.172523 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167822 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 21:05:45.172523 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167825 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 21:05:45.172523 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167828 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 21:05:45.172523 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167831 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 21:05:45.172523 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167835 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 21:05:45.172523 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167838 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 21:05:45.172982 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167841 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 21:05:45.172982 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167844 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 21:05:45.172982 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167847 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 21:05:45.172982 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167849 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 21:05:45.172982 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167852 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 21:05:45.172982 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167854 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 21:05:45.172982 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167857 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 21:05:45.172982 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167859 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 21:05:45.172982 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167861 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 21:05:45.172982 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167864 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 21:05:45.172982 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167866 2572 feature_gate.go:328] unrecognized feature gate: Example Jul 22 21:05:45.172982 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167868 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 21:05:45.172982 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167871 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 21:05:45.172982 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167874 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 21:05:45.172982 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167877 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 21:05:45.172982 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167881 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 21:05:45.172982 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167884 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 21:05:45.172982 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167886 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 21:05:45.172982 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167889 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 21:05:45.173464 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167891 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 21:05:45.173464 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167894 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 21:05:45.173464 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167896 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 21:05:45.173464 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167898 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 21:05:45.173464 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167901 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 21:05:45.173464 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167904 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 21:05:45.173464 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167906 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 21:05:45.173464 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167909 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 21:05:45.173464 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167911 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 21:05:45.173464 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167914 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 21:05:45.173464 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167916 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 21:05:45.173464 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167919 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 21:05:45.173464 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167922 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 21:05:45.173464 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167924 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 21:05:45.173464 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.167927 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 21:05:45.173861 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.167932 2572 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 22 21:05:45.173861 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168037 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 21:05:45.173861 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168041 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 21:05:45.173861 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168044 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 21:05:45.173861 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168048 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 21:05:45.173861 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168050 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 21:05:45.173861 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168053 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 21:05:45.173861 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168056 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 21:05:45.173861 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168058 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 21:05:45.173861 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168060 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 21:05:45.173861 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168063 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 21:05:45.173861 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168065 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 21:05:45.173861 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168067 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 21:05:45.173861 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168070 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 21:05:45.173861 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168072 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 21:05:45.174245 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168075 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 21:05:45.174245 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168077 2572 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 21:05:45.174245 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168079 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 21:05:45.174245 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168082 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 21:05:45.174245 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168084 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 21:05:45.174245 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168087 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 21:05:45.174245 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168089 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 21:05:45.174245 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168092 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 21:05:45.174245 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168095 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 21:05:45.174245 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168097 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 21:05:45.174245 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168099 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 21:05:45.174245 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168102 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 21:05:45.174245 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168104 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 21:05:45.174245 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168106 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 21:05:45.174245 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168109 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 21:05:45.174245 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168111 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 21:05:45.174245 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168114 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 21:05:45.174245 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168116 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 21:05:45.174245 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168118 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 21:05:45.174245 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168121 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 21:05:45.174723 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168123 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 21:05:45.174723 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168127 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 21:05:45.174723 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168130 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 21:05:45.174723 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168133 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 21:05:45.174723 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168136 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 21:05:45.174723 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168138 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 21:05:45.174723 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168140 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 21:05:45.174723 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168143 2572 feature_gate.go:328] unrecognized feature gate: Example Jul 22 21:05:45.174723 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168145 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 21:05:45.174723 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168147 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 21:05:45.174723 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168150 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 21:05:45.174723 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168152 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 21:05:45.174723 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168154 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 21:05:45.174723 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168157 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 21:05:45.174723 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168178 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 21:05:45.174723 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168181 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 21:05:45.174723 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168184 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 21:05:45.174723 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168186 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 21:05:45.174723 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168190 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 21:05:45.175265 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168193 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 21:05:45.175265 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168196 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 21:05:45.175265 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168199 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 21:05:45.175265 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168202 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 21:05:45.175265 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168204 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 21:05:45.175265 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168207 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 21:05:45.175265 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168210 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 21:05:45.175265 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168213 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 21:05:45.175265 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168217 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 21:05:45.175265 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168219 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 21:05:45.175265 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168222 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 21:05:45.175265 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168224 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 21:05:45.175265 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168228 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 21:05:45.175265 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168230 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 21:05:45.175265 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168233 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 21:05:45.175265 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168235 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 21:05:45.175265 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168237 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 21:05:45.175265 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168240 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 21:05:45.175265 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168242 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 21:05:45.175706 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168244 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 21:05:45.175706 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168247 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 21:05:45.175706 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168249 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 21:05:45.175706 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168251 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 21:05:45.175706 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168254 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 21:05:45.175706 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168256 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 21:05:45.175706 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168258 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 21:05:45.175706 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168261 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 21:05:45.175706 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168263 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 21:05:45.175706 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168266 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 21:05:45.175706 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168268 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 21:05:45.175706 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168270 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 21:05:45.175706 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168273 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 21:05:45.175706 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168275 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 21:05:45.175706 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168277 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 21:05:45.175706 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168280 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 21:05:45.175706 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168282 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 21:05:45.175706 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168285 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 21:05:45.175706 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168287 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 21:05:45.175706 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168290 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 21:05:45.176187 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168292 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 21:05:45.176187 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168294 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 21:05:45.176187 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168298 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 21:05:45.176187 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168300 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 21:05:45.176187 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168303 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 21:05:45.176187 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168305 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 21:05:45.176187 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:45.168308 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 21:05:45.176187 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.168313 2572 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 22 21:05:45.176187 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.168643 2572 server.go:962] "Client rotation is on, will bootstrap in background" Jul 22 21:05:45.176187 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.171441 2572 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 22 21:05:45.176187 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.173009 2572 server.go:1019] "Starting client certificate rotation" Jul 22 21:05:45.176187 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.173108 2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 22 21:05:45.176187 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.173148 2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 22 21:05:45.176511 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.175576 2572 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 22 21:05:45.179778 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.179759 2572 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 22 21:05:45.189583 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.189569 2572 log.go:25] "Validated CRI v1 runtime API" Jul 22 21:05:45.192669 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.192062 2572 log.go:25] "Validated CRI v1 image API" Jul 22 21:05:45.193278 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.193262 2572 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 22 21:05:45.195608 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.195586 2572 fs.go:135] Filesystem UUIDs: map[7687b274-6be1-49eb-88bf-17909c948524:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 eae67623-2758-4143-8771-064f4c0ef3e1:/dev/nvme0n1p4] Jul 22 21:05:45.195670 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.195608 2572 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 22 21:05:45.199589 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.199486 2572 manager.go:216] Machine: {Timestamp:2026-07-22 21:05:45.199003619 +0000 UTC m=+0.313395973 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3107643 MemoryCapacity:32812171264 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec29c68043a6203bdc163875c4b453c5 SystemUUID:ec29c680-43a6-203b-dc16-3875c4b453c5 BootID:e015b86f-d512-404a-ba03-f612d44b34fc Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:04:ff:ce:a4:2b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:04:ff:ce:a4:2b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:be:ec:ab:50:6b:00 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812171264 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 22 21:05:45.199589 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.199586 2572 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 22 21:05:45.199705 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.199652 2572 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 22 21:05:45.201263 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.201245 2572 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 22 21:05:45.202553 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.202531 2572 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 22 21:05:45.202680 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.202560 2572 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-37.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 22 21:05:45.203143 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.203134 2572 topology_manager.go:138] "Creating topology manager with none policy" Jul 22 21:05:45.203191 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.203146 2572 container_manager_linux.go:309] "Creating device plugin manager" Jul 22 21:05:45.203191 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.203153 2572 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 22 21:05:45.203191 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.203183 2572 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 22 21:05:45.204798 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.204788 2572 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 22 21:05:45.208006 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.207994 2572 state_mem.go:36] "Initialized new in-memory state store" Jul 22 21:05:45.208108 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.208099 2572 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 22 21:05:45.208143 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.208127 2572 kubelet.go:486] "Attempting to sync node with API server" Jul 22 21:05:45.208143 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.208136 2572 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 22 21:05:45.208221 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.208148 2572 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 22 21:05:45.208221 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.208157 2572 kubelet.go:398] "Adding apiserver pod source" Jul 22 21:05:45.208221 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.208178 2572 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 22 21:05:45.208450 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.208440 2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 22 21:05:45.208486 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.208458 2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 22 21:05:45.210362 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.210340 2572 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 22 21:05:45.214094 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.214076 2572 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 22 21:05:45.214193 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.214116 2572 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 22 21:05:45.215517 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.215502 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 22 21:05:45.215592 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.215522 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 22 21:05:45.215592 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.215531 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 22 21:05:45.215592 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.215538 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 22 21:05:45.215592 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.215551 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 22 21:05:45.215592 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.215559 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 22 21:05:45.215592 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.215570 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 22 21:05:45.215592 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.215578 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 22 21:05:45.215592 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.215588 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 22 21:05:45.215592 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.215596 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 22 21:05:45.215852 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.215608 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 22 21:05:45.215852 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.215616 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 22 21:05:45.216362 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.216349 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 22 21:05:45.216416 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.216364 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 22 21:05:45.218821 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.218800 2572 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-37.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 22 21:05:45.219458 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:05:45.219433 2572 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 22 21:05:45.219458 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:05:45.219435 2572 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-37.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 22 21:05:45.219657 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.219644 2572 server.go:1268] "Started kubelet" Jul 22 21:05:45.219732 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.219709 2572 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 22 21:05:45.219853 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.219781 2572 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 22 21:05:45.219904 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.219879 2572 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 22 21:05:45.220364 ip-10-0-135-37 systemd[1]: Started Kubernetes Kubelet. Jul 22 21:05:45.221839 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.221823 2572 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 22 21:05:45.222538 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.222523 2572 server.go:310] "Adding debug handlers to kubelet server" Jul 22 21:05:45.224497 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.224477 2572 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tvl6x" Jul 22 21:05:45.225638 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:05:45.225620 2572 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 22 21:05:45.230036 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.230023 2572 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 22 21:05:45.230100 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.230049 2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 22 21:05:45.230154 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:05:45.230138 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-37.ec2.internal\" not found" Jul 22 21:05:45.230732 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.230711 2572 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 22 21:05:45.230732 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.230726 2572 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 22 21:05:45.230870 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.230741 2572 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 22 21:05:45.230870 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.230852 2572 reconstruct.go:97] "Volume reconstruction finished" Jul 22 21:05:45.230870 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.230861 2572 reconciler.go:29] "Reconciler: start to sync state" Jul 22 21:05:45.230984 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.230903 2572 factory.go:55] Registering systemd factory Jul 22 21:05:45.230984 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.230915 2572 factory.go:223] Registration of the systemd container factory successfully Jul 22 21:05:45.231136 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.231105 2572 factory.go:153] Registering CRI-O factory Jul 22 21:05:45.231136 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.231133 2572 factory.go:223] Registration of the crio container factory successfully Jul 22 21:05:45.231260 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.231221 2572 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 22 21:05:45.231260 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.231244 2572 factory.go:103] Registering Raw factory Jul 22 21:05:45.231260 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.231260 2572 manager.go:1196] Started watching for new ooms in manager Jul 22 21:05:45.231673 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.231653 2572 manager.go:349] Starting recovery of all containers Jul 22 21:05:45.232767 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.232742 2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tvl6x" Jul 22 21:05:45.238138 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:05:45.238110 2572 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 22 21:05:45.239194 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:05:45.238195 2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-37.ec2.internal.18c4b89ef62677cc default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-37.ec2.internal,UID:ip-10-0-135-37.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-37.ec2.internal,},FirstTimestamp:2026-07-22 21:05:45.219618764 +0000 UTC m=+0.334011101,LastTimestamp:2026-07-22 21:05:45.219618764 +0000 UTC m=+0.334011101,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-37.ec2.internal,}" Jul 22 21:05:45.239299 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:05:45.239281 2572 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-37.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 22 21:05:45.239858 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.239835 2572 manager.go:354] Recovery completed Jul 22 21:05:45.244569 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:05:45.244547 2572 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Jul 22 21:05:45.247422 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.247410 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 21:05:45.249329 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.249243 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-37.ec2.internal" event="NodeHasSufficientMemory" Jul 22 21:05:45.249329 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.249271 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-37.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 21:05:45.249329 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.249283 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-37.ec2.internal" event="NodeHasSufficientPID" Jul 22 21:05:45.249792 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.249778 2572 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 22 21:05:45.249792 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.249789 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 21:05:45.249905 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.249792 2572 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 22 21:05:45.249905 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.249848 2572 state_mem.go:36] "Initialized new in-memory state store" Jul 22 21:05:45.250429 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.250415 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-37.ec2.internal" event="NodeHasSufficientMemory" Jul 22 21:05:45.250536 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.250445 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-37.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 21:05:45.250536 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.250460 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-37.ec2.internal" event="NodeHasSufficientPID" Jul 22 21:05:45.250978 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.250966 2572 policy_none.go:49] "None policy: Start" Jul 22 21:05:45.251014 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.250982 2572 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 22 21:05:45.251014 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.250991 2572 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 22 21:05:45.251922 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.251912 2572 policy_none.go:47] "Start" Jul 22 21:05:45.303493 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.288886 2572 manager.go:337] "Starting Device Plugin manager" Jul 22 21:05:45.303493 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:05:45.289038 2572 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 22 21:05:45.303493 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.289050 2572 server.go:87] "Starting device plugin registration server" Jul 22 21:05:45.303493 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.289268 2572 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 22 21:05:45.303493 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.289281 2572 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 22 21:05:45.303493 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.289382 2572 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 22 21:05:45.303493 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.289456 2572 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 22 21:05:45.303493 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.289466 2572 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 22 21:05:45.303493 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:05:45.289866 2572 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 22 21:05:45.303493 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:05:45.289908 2572 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-37.ec2.internal\" not found" Jul 22 21:05:45.375031 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.375002 2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 22 21:05:45.376312 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.376298 2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 22 21:05:45.376402 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.376340 2572 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 22 21:05:45.376402 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.376365 2572 kubelet.go:2448] "Starting kubelet main sync loop" Jul 22 21:05:45.376502 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:05:45.376403 2572 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 22 21:05:45.378517 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.378498 2572 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 21:05:45.389380 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.389357 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 21:05:45.390092 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.390076 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-37.ec2.internal" event="NodeHasSufficientMemory" Jul 22 21:05:45.390187 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.390109 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-37.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 21:05:45.390187 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.390123 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-37.ec2.internal" event="NodeHasSufficientPID" Jul 22 21:05:45.390187 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.390151 2572 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-37.ec2.internal" Jul 22 21:05:45.399399 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.399379 2572 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-37.ec2.internal" Jul 22 21:05:45.399484 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:05:45.399402 2572 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-37.ec2.internal\": node \"ip-10-0-135-37.ec2.internal\" not found" Jul 22 21:05:45.412126 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:05:45.412081 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-37.ec2.internal\" not found" Jul 22 21:05:45.477050 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.477026 2572 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-135-37.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-37.ec2.internal"] Jul 22 21:05:45.477131 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.477080 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 21:05:45.477943 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.477928 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-37.ec2.internal" event="NodeHasSufficientMemory" Jul 22 21:05:45.478013 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.477955 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-37.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 21:05:45.478013 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.477966 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-37.ec2.internal" event="NodeHasSufficientPID" Jul 22 21:05:45.479309 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.479297 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 21:05:45.479415 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.479401 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-37.ec2.internal" Jul 22 21:05:45.479461 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.479445 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 21:05:45.479954 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.479938 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-37.ec2.internal" event="NodeHasSufficientMemory" Jul 22 21:05:45.480026 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.479954 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-37.ec2.internal" event="NodeHasSufficientMemory" Jul 22 21:05:45.480026 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.479977 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-37.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 21:05:45.480026 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.479987 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-37.ec2.internal" event="NodeHasSufficientPID" Jul 22 21:05:45.480140 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.480037 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-37.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 21:05:45.480140 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.480068 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-37.ec2.internal" event="NodeHasSufficientPID" Jul 22 21:05:45.481414 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.481398 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-37.ec2.internal" Jul 22 21:05:45.481495 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.481425 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 21:05:45.482085 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.482069 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-37.ec2.internal" event="NodeHasSufficientMemory" Jul 22 21:05:45.482146 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.482101 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-37.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 21:05:45.482146 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.482116 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-37.ec2.internal" event="NodeHasSufficientPID" Jul 22 21:05:45.506786 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:05:45.506759 2572 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-37.ec2.internal\" not found" node="ip-10-0-135-37.ec2.internal" Jul 22 21:05:45.511179 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:05:45.511149 2572 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-37.ec2.internal\" not found" node="ip-10-0-135-37.ec2.internal" Jul 22 21:05:45.512142 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:05:45.512126 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-37.ec2.internal\" not found" Jul 22 21:05:45.532822 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.532806 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/091f2ee13ccf617382e7e126cc5d9e12-config\") pod \"kube-apiserver-proxy-ip-10-0-135-37.ec2.internal\" (UID: \"091f2ee13ccf617382e7e126cc5d9e12\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-37.ec2.internal" Jul 22 21:05:45.532911 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.532828 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e462d762995f2dc89c7e851abf11f67f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-37.ec2.internal\" (UID: \"e462d762995f2dc89c7e851abf11f67f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-37.ec2.internal" Jul 22 21:05:45.532911 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.532848 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e462d762995f2dc89c7e851abf11f67f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-37.ec2.internal\" (UID: \"e462d762995f2dc89c7e851abf11f67f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-37.ec2.internal" Jul 22 21:05:45.612486 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:05:45.612468 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-37.ec2.internal\" not found" Jul 22 21:05:45.633896 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.633878 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/091f2ee13ccf617382e7e126cc5d9e12-config\") pod \"kube-apiserver-proxy-ip-10-0-135-37.ec2.internal\" (UID: \"091f2ee13ccf617382e7e126cc5d9e12\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-37.ec2.internal" Jul 22 21:05:45.633980 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.633900 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e462d762995f2dc89c7e851abf11f67f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-37.ec2.internal\" (UID: \"e462d762995f2dc89c7e851abf11f67f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-37.ec2.internal" Jul 22 21:05:45.633980 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.633916 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e462d762995f2dc89c7e851abf11f67f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-37.ec2.internal\" (UID: \"e462d762995f2dc89c7e851abf11f67f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-37.ec2.internal" Jul 22 21:05:45.633980 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.633956 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e462d762995f2dc89c7e851abf11f67f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-37.ec2.internal\" (UID: \"e462d762995f2dc89c7e851abf11f67f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-37.ec2.internal" Jul 22 21:05:45.633980 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.633970 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/091f2ee13ccf617382e7e126cc5d9e12-config\") pod \"kube-apiserver-proxy-ip-10-0-135-37.ec2.internal\" (UID: \"091f2ee13ccf617382e7e126cc5d9e12\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-37.ec2.internal" Jul 22 21:05:45.634107 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.633984 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e462d762995f2dc89c7e851abf11f67f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-37.ec2.internal\" (UID: \"e462d762995f2dc89c7e851abf11f67f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-37.ec2.internal" Jul 22 21:05:45.713346 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:05:45.713276 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-37.ec2.internal\" not found" Jul 22 21:05:45.810861 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.810837 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-37.ec2.internal" Jul 22 21:05:45.813663 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:05:45.813646 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-37.ec2.internal\" not found" Jul 22 21:05:45.815375 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:45.815154 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-37.ec2.internal" Jul 22 21:05:45.913993 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:05:45.913957 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-37.ec2.internal\" not found" Jul 22 21:05:46.014469 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:05:46.014403 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-37.ec2.internal\" not found" Jul 22 21:05:46.087271 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:46.087246 2572 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 21:05:46.115222 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:05:46.115193 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-37.ec2.internal\" not found" Jul 22 21:05:46.172731 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:46.172707 2572 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 22 21:05:46.173120 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:46.172811 2572 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 22 21:05:46.173120 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:46.172852 2572 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 22 21:05:46.216186 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:05:46.216153 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-37.ec2.internal\" not found" Jul 22 21:05:46.238784 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:46.238752 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-21 21:00:45 +0000 UTC" deadline="2028-03-11 06:34:35.06490407 +0000 UTC" Jul 22 21:05:46.238784 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:46.238776 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14337h28m48.826132021s" Jul 22 21:05:46.316310 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:05:46.316275 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-37.ec2.internal\" not found" Jul 22 21:05:46.395369 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:46.395334 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode462d762995f2dc89c7e851abf11f67f.slice/crio-602e6359baeeb08c973d7a69f6b48a0c624a59ef52f7a4c4683fb2b3c22e6936 WatchSource:0}: Error finding container 602e6359baeeb08c973d7a69f6b48a0c624a59ef52f7a4c4683fb2b3c22e6936: Status 404 returned error can't find the container with id 602e6359baeeb08c973d7a69f6b48a0c624a59ef52f7a4c4683fb2b3c22e6936 Jul 22 21:05:46.400145 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:46.400129 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 21:05:46.417341 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:05:46.417322 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-37.ec2.internal\" not found" Jul 22 21:05:46.517557 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:05:46.517504 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-37.ec2.internal\" not found" Jul 22 21:05:46.520653 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:46.520634 2572 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 21:05:46.617913 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:05:46.617865 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-37.ec2.internal\" not found" Jul 22 21:05:46.718635 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:05:46.718609 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-37.ec2.internal\" not found" Jul 22 21:05:46.771562 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:46.771538 2572 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 21:05:46.830374 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:46.830353 2572 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-37.ec2.internal" Jul 22 21:05:46.840826 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:46.840809 2572 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 22 21:05:46.841755 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:46.841743 2572 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-37.ec2.internal" Jul 22 21:05:46.853112 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:46.853093 2572 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 22 21:05:47.209939 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.209906 2572 apiserver.go:52] "Watching apiserver" Jul 22 21:05:47.218295 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.218267 2572 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 22 21:05:47.221192 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.221130 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-zrg2r","openshift-dns/node-resolver-z2g8w","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-37.ec2.internal","openshift-multus/multus-kpw4k","openshift-network-diagnostics/network-check-target-tn98z","openshift-ovn-kubernetes/ovnkube-node-7w87p","kube-system/konnectivity-agent-m7pxh","kube-system/kube-apiserver-proxy-ip-10-0-135-37.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-87wqc","openshift-cluster-node-tuning-operator/tuned-dxs8z","openshift-image-registry/node-ca-4s9cs","openshift-multus/multus-additional-cni-plugins-42g4d","openshift-multus/network-metrics-daemon-lj48g","openshift-network-operator/iptables-alerter-rdlwm"] Jul 22 21:05:47.224535 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.224506 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-m7pxh" Jul 22 21:05:47.227433 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.227378 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 22 21:05:47.227584 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.227463 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-dgk9s\"" Jul 22 21:05:47.227889 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.227863 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 22 21:05:47.227998 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.227923 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lj48g" Jul 22 21:05:47.228068 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:05:47.227993 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lj48g" podUID="8a91c48d-374d-4fc3-9114-d5c88ea18f63" Jul 22 21:05:47.230435 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.230415 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rdlwm" Jul 22 21:05:47.231028 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.231010 2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 22 21:05:47.233128 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.232861 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-kpw4k" Jul 22 21:05:47.233128 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.232965 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-n7hh8\"" Jul 22 21:05:47.233330 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.233210 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 22 21:05:47.233330 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.233300 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 22 21:05:47.233439 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.233353 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 22 21:05:47.235139 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.235111 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tn98z" Jul 22 21:05:47.235295 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:05:47.235192 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tn98z" podUID="07cbf59d-dabc-4ba4-92b8-ec63306521cf" Jul 22 21:05:47.235443 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.235421 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 22 21:05:47.235534 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.235451 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-45g5p\"" Jul 22 21:05:47.235742 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.235715 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 22 21:05:47.235742 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.235726 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 22 21:05:47.235887 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.235820 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 22 21:05:47.237894 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.237844 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7w87p" Jul 22 21:05:47.240114 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.240091 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-87wqc" Jul 22 21:05:47.240948 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.240927 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 22 21:05:47.241026 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.240996 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 22 21:05:47.241216 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.241187 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 22 21:05:47.241363 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.241345 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 22 21:05:47.241775 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.241755 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-kmw94\"" Jul 22 21:05:47.241976 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.241960 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 22 21:05:47.242191 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.242174 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 22 21:05:47.242685 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.242662 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/37b20c60-c50a-42df-acd7-2164653d94ea-agent-certs\") pod \"konnectivity-agent-m7pxh\" (UID: \"37b20c60-c50a-42df-acd7-2164653d94ea\") " pod="kube-system/konnectivity-agent-m7pxh" Jul 22 21:05:47.242803 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.242702 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a91c48d-374d-4fc3-9114-d5c88ea18f63-metrics-certs\") pod \"network-metrics-daemon-lj48g\" (UID: \"8a91c48d-374d-4fc3-9114-d5c88ea18f63\") " pod="openshift-multus/network-metrics-daemon-lj48g" Jul 22 21:05:47.242803 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.242727 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zrg2r" Jul 22 21:05:47.242803 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.242731 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25hmb\" (UniqueName: \"kubernetes.io/projected/3e6d7a69-6a07-40bf-9b57-c7a4828cb08a-kube-api-access-25hmb\") pod \"iptables-alerter-rdlwm\" (UID: \"3e6d7a69-6a07-40bf-9b57-c7a4828cb08a\") " pod="openshift-network-operator/iptables-alerter-rdlwm" Jul 22 21:05:47.242803 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.242757 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2d3ff930-d226-4a49-9cb5-aa7d02eb37af-cnibin\") pod \"multus-kpw4k\" (UID: \"2d3ff930-d226-4a49-9cb5-aa7d02eb37af\") " pod="openshift-multus/multus-kpw4k" Jul 22 21:05:47.242803 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.242782 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2d3ff930-d226-4a49-9cb5-aa7d02eb37af-os-release\") pod \"multus-kpw4k\" (UID: \"2d3ff930-d226-4a49-9cb5-aa7d02eb37af\") " pod="openshift-multus/multus-kpw4k" Jul 22 21:05:47.242803 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:05:47.242786 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zrg2r" podUID="4f3c7912-ed80-4a24-ada3-b2a3033bb210" Jul 22 21:05:47.242803 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.242804 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2d3ff930-d226-4a49-9cb5-aa7d02eb37af-host-run-k8s-cni-cncf-io\") pod \"multus-kpw4k\" (UID: \"2d3ff930-d226-4a49-9cb5-aa7d02eb37af\") " pod="openshift-multus/multus-kpw4k" Jul 22 21:05:47.243162 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.242828 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2d3ff930-d226-4a49-9cb5-aa7d02eb37af-host-var-lib-cni-bin\") pod \"multus-kpw4k\" (UID: \"2d3ff930-d226-4a49-9cb5-aa7d02eb37af\") " pod="openshift-multus/multus-kpw4k" Jul 22 21:05:47.243162 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.242852 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2d3ff930-d226-4a49-9cb5-aa7d02eb37af-system-cni-dir\") pod \"multus-kpw4k\" (UID: \"2d3ff930-d226-4a49-9cb5-aa7d02eb37af\") " pod="openshift-multus/multus-kpw4k" Jul 22 21:05:47.243162 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.242877 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2d3ff930-d226-4a49-9cb5-aa7d02eb37af-cni-binary-copy\") pod \"multus-kpw4k\" (UID: \"2d3ff930-d226-4a49-9cb5-aa7d02eb37af\") " pod="openshift-multus/multus-kpw4k" Jul 22 21:05:47.243162 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.242900 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2d3ff930-d226-4a49-9cb5-aa7d02eb37af-multus-socket-dir-parent\") pod \"multus-kpw4k\" (UID: \"2d3ff930-d226-4a49-9cb5-aa7d02eb37af\") " pod="openshift-multus/multus-kpw4k" Jul 22 21:05:47.243162 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.242927 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2d3ff930-d226-4a49-9cb5-aa7d02eb37af-host-run-netns\") pod \"multus-kpw4k\" (UID: \"2d3ff930-d226-4a49-9cb5-aa7d02eb37af\") " pod="openshift-multus/multus-kpw4k" Jul 22 21:05:47.243162 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.242950 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2d3ff930-d226-4a49-9cb5-aa7d02eb37af-host-var-lib-kubelet\") pod \"multus-kpw4k\" (UID: \"2d3ff930-d226-4a49-9cb5-aa7d02eb37af\") " pod="openshift-multus/multus-kpw4k" Jul 22 21:05:47.243162 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.242998 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2d3ff930-d226-4a49-9cb5-aa7d02eb37af-multus-conf-dir\") pod \"multus-kpw4k\" (UID: \"2d3ff930-d226-4a49-9cb5-aa7d02eb37af\") " pod="openshift-multus/multus-kpw4k" Jul 22 21:05:47.243162 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.243021 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2d3ff930-d226-4a49-9cb5-aa7d02eb37af-host-run-multus-certs\") pod \"multus-kpw4k\" (UID: \"2d3ff930-d226-4a49-9cb5-aa7d02eb37af\") " pod="openshift-multus/multus-kpw4k" Jul 22 21:05:47.243162 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.243044 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 22 21:05:47.243162 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.243049 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d3ff930-d226-4a49-9cb5-aa7d02eb37af-etc-kubernetes\") pod \"multus-kpw4k\" (UID: \"2d3ff930-d226-4a49-9cb5-aa7d02eb37af\") " pod="openshift-multus/multus-kpw4k" Jul 22 21:05:47.243162 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.243105 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3e6d7a69-6a07-40bf-9b57-c7a4828cb08a-iptables-alerter-script\") pod \"iptables-alerter-rdlwm\" (UID: \"3e6d7a69-6a07-40bf-9b57-c7a4828cb08a\") " pod="openshift-network-operator/iptables-alerter-rdlwm" Jul 22 21:05:47.243162 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.243145 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wz58x\" (UniqueName: \"kubernetes.io/projected/2d3ff930-d226-4a49-9cb5-aa7d02eb37af-kube-api-access-wz58x\") pod \"multus-kpw4k\" (UID: \"2d3ff930-d226-4a49-9cb5-aa7d02eb37af\") " pod="openshift-multus/multus-kpw4k" Jul 22 21:05:47.243162 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.243166 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/37b20c60-c50a-42df-acd7-2164653d94ea-konnectivity-ca\") pod \"konnectivity-agent-m7pxh\" (UID: \"37b20c60-c50a-42df-acd7-2164653d94ea\") " pod="kube-system/konnectivity-agent-m7pxh" Jul 22 21:05:47.243721 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.243187 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slbtr\" (UniqueName: \"kubernetes.io/projected/8a91c48d-374d-4fc3-9114-d5c88ea18f63-kube-api-access-slbtr\") pod \"network-metrics-daemon-lj48g\" (UID: \"8a91c48d-374d-4fc3-9114-d5c88ea18f63\") " pod="openshift-multus/network-metrics-daemon-lj48g" Jul 22 21:05:47.243721 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.243214 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3e6d7a69-6a07-40bf-9b57-c7a4828cb08a-host-slash\") pod \"iptables-alerter-rdlwm\" (UID: \"3e6d7a69-6a07-40bf-9b57-c7a4828cb08a\") " pod="openshift-network-operator/iptables-alerter-rdlwm" Jul 22 21:05:47.243721 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.243259 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2d3ff930-d226-4a49-9cb5-aa7d02eb37af-multus-cni-dir\") pod \"multus-kpw4k\" (UID: \"2d3ff930-d226-4a49-9cb5-aa7d02eb37af\") " pod="openshift-multus/multus-kpw4k" Jul 22 21:05:47.243721 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.243283 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2d3ff930-d226-4a49-9cb5-aa7d02eb37af-host-var-lib-cni-multus\") pod \"multus-kpw4k\" (UID: \"2d3ff930-d226-4a49-9cb5-aa7d02eb37af\") " pod="openshift-multus/multus-kpw4k" Jul 22 21:05:47.243721 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.243316 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2d3ff930-d226-4a49-9cb5-aa7d02eb37af-hostroot\") pod \"multus-kpw4k\" (UID: \"2d3ff930-d226-4a49-9cb5-aa7d02eb37af\") " pod="openshift-multus/multus-kpw4k" Jul 22 21:05:47.243721 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.243358 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2d3ff930-d226-4a49-9cb5-aa7d02eb37af-multus-daemon-config\") pod \"multus-kpw4k\" (UID: \"2d3ff930-d226-4a49-9cb5-aa7d02eb37af\") " pod="openshift-multus/multus-kpw4k" Jul 22 21:05:47.244451 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.244429 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 22 21:05:47.244538 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.244503 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-mvvrt\"" Jul 22 21:05:47.244538 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.244524 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 22 21:05:47.245266 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.245248 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-z2g8w" Jul 22 21:05:47.246794 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.246773 2572 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 22 21:05:47.247490 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.247470 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-pgrh7\"" Jul 22 21:05:47.247719 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.247688 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dxs8z" Jul 22 21:05:47.247796 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.247752 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 22 21:05:47.248141 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.248124 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 22 21:05:47.250107 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.250087 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-8sblh\"" Jul 22 21:05:47.250201 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.250174 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 22 21:05:47.250368 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.250352 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 22 21:05:47.250709 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.250690 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4s9cs" Jul 22 21:05:47.252981 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.252962 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 22 21:05:47.253483 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.253453 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-42g4d" Jul 22 21:05:47.253584 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.253489 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-srggh\"" Jul 22 21:05:47.253584 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.253505 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 22 21:05:47.253584 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.253490 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 22 21:05:47.258741 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.258721 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 22 21:05:47.258831 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.258811 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-l8mk4\"" Jul 22 21:05:47.259052 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.259031 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 22 21:05:47.267094 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.267066 2572 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-v2crw" Jul 22 21:05:47.273785 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.273768 2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-v2crw" Jul 22 21:05:47.332614 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.332587 2572 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 22 21:05:47.343962 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.343929 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hwlt\" (UniqueName: \"kubernetes.io/projected/40f77f02-2bc6-4339-a948-7631a9603ce1-kube-api-access-8hwlt\") pod \"node-resolver-z2g8w\" (UID: \"40f77f02-2bc6-4339-a948-7631a9603ce1\") " pod="openshift-dns/node-resolver-z2g8w" Jul 22 21:05:47.344061 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.343967 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/45b1939d-7d91-4899-b489-7e2609a5878d-serviceca\") pod \"node-ca-4s9cs\" (UID: \"45b1939d-7d91-4899-b489-7e2609a5878d\") " pod="openshift-image-registry/node-ca-4s9cs" Jul 22 21:05:47.344061 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.343993 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3e6d7a69-6a07-40bf-9b57-c7a4828cb08a-iptables-alerter-script\") pod \"iptables-alerter-rdlwm\" (UID: \"3e6d7a69-6a07-40bf-9b57-c7a4828cb08a\") " pod="openshift-network-operator/iptables-alerter-rdlwm" Jul 22 21:05:47.344061 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.344019 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c29841d1-b93c-4f1e-9455-477c2208081f-host-kubelet\") pod \"ovnkube-node-7w87p\" (UID: \"c29841d1-b93c-4f1e-9455-477c2208081f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w87p" Jul 22 21:05:47.344061 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.344042 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c29841d1-b93c-4f1e-9455-477c2208081f-node-log\") pod \"ovnkube-node-7w87p\" (UID: \"c29841d1-b93c-4f1e-9455-477c2208081f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w87p" Jul 22 21:05:47.344268 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.344064 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/40f77f02-2bc6-4339-a948-7631a9603ce1-hosts-file\") pod \"node-resolver-z2g8w\" (UID: \"40f77f02-2bc6-4339-a948-7631a9603ce1\") " pod="openshift-dns/node-resolver-z2g8w" Jul 22 21:05:47.344268 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.344091 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4fe33109-ef55-4c16-935f-cc6280e9d58e-etc-sysctl-conf\") pod \"tuned-dxs8z\" (UID: \"4fe33109-ef55-4c16-935f-cc6280e9d58e\") " pod="openshift-cluster-node-tuning-operator/tuned-dxs8z" Jul 22 21:05:47.344268 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.344133 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/45b1939d-7d91-4899-b489-7e2609a5878d-host\") pod \"node-ca-4s9cs\" (UID: \"45b1939d-7d91-4899-b489-7e2609a5878d\") " pod="openshift-image-registry/node-ca-4s9cs" Jul 22 21:05:47.344268 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.344193 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slbtr\" (UniqueName: \"kubernetes.io/projected/8a91c48d-374d-4fc3-9114-d5c88ea18f63-kube-api-access-slbtr\") pod \"network-metrics-daemon-lj48g\" (UID: \"8a91c48d-374d-4fc3-9114-d5c88ea18f63\") " pod="openshift-multus/network-metrics-daemon-lj48g" Jul 22 21:05:47.344268 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.344251 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2d3ff930-d226-4a49-9cb5-aa7d02eb37af-multus-cni-dir\") pod \"multus-kpw4k\" (UID: \"2d3ff930-d226-4a49-9cb5-aa7d02eb37af\") " pod="openshift-multus/multus-kpw4k" Jul 22 21:05:47.344466 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.344393 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2d3ff930-d226-4a49-9cb5-aa7d02eb37af-host-var-lib-cni-multus\") pod \"multus-kpw4k\" (UID: \"2d3ff930-d226-4a49-9cb5-aa7d02eb37af\") " pod="openshift-multus/multus-kpw4k" Jul 22 21:05:47.344466 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.344427 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcr84\" (UniqueName: \"kubernetes.io/projected/c29841d1-b93c-4f1e-9455-477c2208081f-kube-api-access-tcr84\") pod \"ovnkube-node-7w87p\" (UID: \"c29841d1-b93c-4f1e-9455-477c2208081f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w87p" Jul 22 21:05:47.344466 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.344459 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cf2ac247-7e8e-46b0-991e-e82a384991c1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-42g4d\" (UID: \"cf2ac247-7e8e-46b0-991e-e82a384991c1\") " pod="openshift-multus/multus-additional-cni-plugins-42g4d" Jul 22 21:05:47.344574 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.344486 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4f3c7912-ed80-4a24-ada3-b2a3033bb210-original-pull-secret\") pod \"global-pull-secret-syncer-zrg2r\" (UID: \"4f3c7912-ed80-4a24-ada3-b2a3033bb210\") " pod="kube-system/global-pull-secret-syncer-zrg2r" Jul 22 21:05:47.344574 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.344511 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2d3ff930-d226-4a49-9cb5-aa7d02eb37af-multus-cni-dir\") pod \"multus-kpw4k\" (UID: \"2d3ff930-d226-4a49-9cb5-aa7d02eb37af\") " pod="openshift-multus/multus-kpw4k" Jul 22 21:05:47.344574 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.344527 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cf2ac247-7e8e-46b0-991e-e82a384991c1-os-release\") pod \"multus-additional-cni-plugins-42g4d\" (UID: \"cf2ac247-7e8e-46b0-991e-e82a384991c1\") " pod="openshift-multus/multus-additional-cni-plugins-42g4d" Jul 22 21:05:47.344574 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.344548 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2d3ff930-d226-4a49-9cb5-aa7d02eb37af-host-var-lib-cni-multus\") pod \"multus-kpw4k\" (UID: \"2d3ff930-d226-4a49-9cb5-aa7d02eb37af\") " pod="openshift-multus/multus-kpw4k" Jul 22 21:05:47.344574 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.344567 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cf2ac247-7e8e-46b0-991e-e82a384991c1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-42g4d\" (UID: \"cf2ac247-7e8e-46b0-991e-e82a384991c1\") " pod="openshift-multus/multus-additional-cni-plugins-42g4d" Jul 22 21:05:47.344761 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.344581 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3e6d7a69-6a07-40bf-9b57-c7a4828cb08a-iptables-alerter-script\") pod \"iptables-alerter-rdlwm\" (UID: \"3e6d7a69-6a07-40bf-9b57-c7a4828cb08a\") " pod="openshift-network-operator/iptables-alerter-rdlwm" Jul 22 21:05:47.344761 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.344596 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4fe33109-ef55-4c16-935f-cc6280e9d58e-etc-modprobe-d\") pod \"tuned-dxs8z\" (UID: \"4fe33109-ef55-4c16-935f-cc6280e9d58e\") " pod="openshift-cluster-node-tuning-operator/tuned-dxs8z" Jul 22 21:05:47.344761 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.344628 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2d3ff930-d226-4a49-9cb5-aa7d02eb37af-multus-socket-dir-parent\") pod \"multus-kpw4k\" (UID: \"2d3ff930-d226-4a49-9cb5-aa7d02eb37af\") " pod="openshift-multus/multus-kpw4k" Jul 22 21:05:47.344761 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.344654 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2d3ff930-d226-4a49-9cb5-aa7d02eb37af-host-var-lib-kubelet\") pod \"multus-kpw4k\" (UID: \"2d3ff930-d226-4a49-9cb5-aa7d02eb37af\") " pod="openshift-multus/multus-kpw4k" Jul 22 21:05:47.344761 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.344682 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c29841d1-b93c-4f1e-9455-477c2208081f-host-run-netns\") pod \"ovnkube-node-7w87p\" (UID: \"c29841d1-b93c-4f1e-9455-477c2208081f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w87p" Jul 22 21:05:47.344761 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.344705 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2d3ff930-d226-4a49-9cb5-aa7d02eb37af-multus-socket-dir-parent\") pod \"multus-kpw4k\" (UID: \"2d3ff930-d226-4a49-9cb5-aa7d02eb37af\") " pod="openshift-multus/multus-kpw4k" Jul 22 21:05:47.344761 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.344708 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c29841d1-b93c-4f1e-9455-477c2208081f-host-cni-netd\") pod \"ovnkube-node-7w87p\" (UID: \"c29841d1-b93c-4f1e-9455-477c2208081f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w87p" Jul 22 21:05:47.344761 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.344758 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2d3ff930-d226-4a49-9cb5-aa7d02eb37af-host-var-lib-kubelet\") pod \"multus-kpw4k\" (UID: \"2d3ff930-d226-4a49-9cb5-aa7d02eb37af\") " pod="openshift-multus/multus-kpw4k" Jul 22 21:05:47.345076 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.344796 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a449924b-6d2d-40f9-97a1-584a066510a3-device-dir\") pod \"aws-ebs-csi-driver-node-87wqc\" (UID: \"a449924b-6d2d-40f9-97a1-584a066510a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-87wqc" Jul 22 21:05:47.345076 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.344831 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c29841d1-b93c-4f1e-9455-477c2208081f-systemd-units\") pod \"ovnkube-node-7w87p\" (UID: \"c29841d1-b93c-4f1e-9455-477c2208081f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w87p" Jul 22 21:05:47.345076 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.344861 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a449924b-6d2d-40f9-97a1-584a066510a3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-87wqc\" (UID: \"a449924b-6d2d-40f9-97a1-584a066510a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-87wqc" Jul 22 21:05:47.345076 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.344892 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a449924b-6d2d-40f9-97a1-584a066510a3-etc-selinux\") pod \"aws-ebs-csi-driver-node-87wqc\" (UID: \"a449924b-6d2d-40f9-97a1-584a066510a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-87wqc" Jul 22 21:05:47.345076 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.344915 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4fe33109-ef55-4c16-935f-cc6280e9d58e-sys\") pod \"tuned-dxs8z\" (UID: \"4fe33109-ef55-4c16-935f-cc6280e9d58e\") " pod="openshift-cluster-node-tuning-operator/tuned-dxs8z" Jul 22 21:05:47.345076 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.344941 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cf2ac247-7e8e-46b0-991e-e82a384991c1-system-cni-dir\") pod \"multus-additional-cni-plugins-42g4d\" (UID: \"cf2ac247-7e8e-46b0-991e-e82a384991c1\") " pod="openshift-multus/multus-additional-cni-plugins-42g4d" Jul 22 21:05:47.345076 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.344968 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4f3c7912-ed80-4a24-ada3-b2a3033bb210-dbus\") pod \"global-pull-secret-syncer-zrg2r\" (UID: \"4f3c7912-ed80-4a24-ada3-b2a3033bb210\") " pod="kube-system/global-pull-secret-syncer-zrg2r" Jul 22 21:05:47.345076 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.344991 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4fe33109-ef55-4c16-935f-cc6280e9d58e-lib-modules\") pod \"tuned-dxs8z\" (UID: \"4fe33109-ef55-4c16-935f-cc6280e9d58e\") " pod="openshift-cluster-node-tuning-operator/tuned-dxs8z" Jul 22 21:05:47.345076 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.345016 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wskpm\" (UniqueName: \"kubernetes.io/projected/45b1939d-7d91-4899-b489-7e2609a5878d-kube-api-access-wskpm\") pod \"node-ca-4s9cs\" (UID: \"45b1939d-7d91-4899-b489-7e2609a5878d\") " pod="openshift-image-registry/node-ca-4s9cs" Jul 22 21:05:47.345076 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.345071 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2d3ff930-d226-4a49-9cb5-aa7d02eb37af-cnibin\") pod \"multus-kpw4k\" (UID: \"2d3ff930-d226-4a49-9cb5-aa7d02eb37af\") " pod="openshift-multus/multus-kpw4k" Jul 22 21:05:47.345643 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.345109 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c29841d1-b93c-4f1e-9455-477c2208081f-var-lib-openvswitch\") pod \"ovnkube-node-7w87p\" (UID: \"c29841d1-b93c-4f1e-9455-477c2208081f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w87p" Jul 22 21:05:47.345643 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.345139 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c29841d1-b93c-4f1e-9455-477c2208081f-run-ovn\") pod \"ovnkube-node-7w87p\" (UID: \"c29841d1-b93c-4f1e-9455-477c2208081f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w87p" Jul 22 21:05:47.345643 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.345154 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2d3ff930-d226-4a49-9cb5-aa7d02eb37af-cnibin\") pod \"multus-kpw4k\" (UID: \"2d3ff930-d226-4a49-9cb5-aa7d02eb37af\") " pod="openshift-multus/multus-kpw4k" Jul 22 21:05:47.345643 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.345163 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cf2ac247-7e8e-46b0-991e-e82a384991c1-cnibin\") pod \"multus-additional-cni-plugins-42g4d\" (UID: \"cf2ac247-7e8e-46b0-991e-e82a384991c1\") " pod="openshift-multus/multus-additional-cni-plugins-42g4d" Jul 22 21:05:47.345643 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.345211 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4fe33109-ef55-4c16-935f-cc6280e9d58e-tmp\") pod \"tuned-dxs8z\" (UID: \"4fe33109-ef55-4c16-935f-cc6280e9d58e\") " pod="openshift-cluster-node-tuning-operator/tuned-dxs8z" Jul 22 21:05:47.345643 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.345237 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2d3ff930-d226-4a49-9cb5-aa7d02eb37af-system-cni-dir\") pod \"multus-kpw4k\" (UID: \"2d3ff930-d226-4a49-9cb5-aa7d02eb37af\") " pod="openshift-multus/multus-kpw4k" Jul 22 21:05:47.345643 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.345262 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2d3ff930-d226-4a49-9cb5-aa7d02eb37af-host-run-multus-certs\") pod \"multus-kpw4k\" (UID: \"2d3ff930-d226-4a49-9cb5-aa7d02eb37af\") " pod="openshift-multus/multus-kpw4k" Jul 22 21:05:47.345643 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.345287 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c29841d1-b93c-4f1e-9455-477c2208081f-ovnkube-config\") pod \"ovnkube-node-7w87p\" (UID: \"c29841d1-b93c-4f1e-9455-477c2208081f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w87p" Jul 22 21:05:47.345643 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.345308 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2d3ff930-d226-4a49-9cb5-aa7d02eb37af-host-run-multus-certs\") pod \"multus-kpw4k\" (UID: \"2d3ff930-d226-4a49-9cb5-aa7d02eb37af\") " pod="openshift-multus/multus-kpw4k" Jul 22 21:05:47.345643 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.345312 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cf2ac247-7e8e-46b0-991e-e82a384991c1-cni-binary-copy\") pod \"multus-additional-cni-plugins-42g4d\" (UID: \"cf2ac247-7e8e-46b0-991e-e82a384991c1\") " pod="openshift-multus/multus-additional-cni-plugins-42g4d" Jul 22 21:05:47.345643 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.345350 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a449924b-6d2d-40f9-97a1-584a066510a3-sys-fs\") pod \"aws-ebs-csi-driver-node-87wqc\" (UID: \"a449924b-6d2d-40f9-97a1-584a066510a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-87wqc" Jul 22 21:05:47.345643 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.345313 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2d3ff930-d226-4a49-9cb5-aa7d02eb37af-system-cni-dir\") pod \"multus-kpw4k\" (UID: \"2d3ff930-d226-4a49-9cb5-aa7d02eb37af\") " pod="openshift-multus/multus-kpw4k" Jul 22 21:05:47.345643 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.345375 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbcq7\" (UniqueName: \"kubernetes.io/projected/a449924b-6d2d-40f9-97a1-584a066510a3-kube-api-access-sbcq7\") pod \"aws-ebs-csi-driver-node-87wqc\" (UID: \"a449924b-6d2d-40f9-97a1-584a066510a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-87wqc" Jul 22 21:05:47.345643 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.345403 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4f3c7912-ed80-4a24-ada3-b2a3033bb210-kubelet-config\") pod \"global-pull-secret-syncer-zrg2r\" (UID: \"4f3c7912-ed80-4a24-ada3-b2a3033bb210\") " pod="kube-system/global-pull-secret-syncer-zrg2r" Jul 22 21:05:47.345643 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.345430 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wz58x\" (UniqueName: \"kubernetes.io/projected/2d3ff930-d226-4a49-9cb5-aa7d02eb37af-kube-api-access-wz58x\") pod \"multus-kpw4k\" (UID: \"2d3ff930-d226-4a49-9cb5-aa7d02eb37af\") " pod="openshift-multus/multus-kpw4k" Jul 22 21:05:47.345643 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.345467 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/37b20c60-c50a-42df-acd7-2164653d94ea-konnectivity-ca\") pod \"konnectivity-agent-m7pxh\" (UID: \"37b20c60-c50a-42df-acd7-2164653d94ea\") " pod="kube-system/konnectivity-agent-m7pxh" Jul 22 21:05:47.345643 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.345493 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c29841d1-b93c-4f1e-9455-477c2208081f-run-systemd\") pod \"ovnkube-node-7w87p\" (UID: \"c29841d1-b93c-4f1e-9455-477c2208081f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w87p" Jul 22 21:05:47.346223 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.345531 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c29841d1-b93c-4f1e-9455-477c2208081f-etc-openvswitch\") pod \"ovnkube-node-7w87p\" (UID: \"c29841d1-b93c-4f1e-9455-477c2208081f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w87p" Jul 22 21:05:47.346223 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.345556 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4fe33109-ef55-4c16-935f-cc6280e9d58e-etc-kubernetes\") pod \"tuned-dxs8z\" (UID: \"4fe33109-ef55-4c16-935f-cc6280e9d58e\") " pod="openshift-cluster-node-tuning-operator/tuned-dxs8z" Jul 22 21:05:47.346223 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.345579 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4fe33109-ef55-4c16-935f-cc6280e9d58e-run\") pod \"tuned-dxs8z\" (UID: \"4fe33109-ef55-4c16-935f-cc6280e9d58e\") " pod="openshift-cluster-node-tuning-operator/tuned-dxs8z" Jul 22 21:05:47.346223 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.345720 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6q5s\" (UniqueName: \"kubernetes.io/projected/4fe33109-ef55-4c16-935f-cc6280e9d58e-kube-api-access-z6q5s\") pod \"tuned-dxs8z\" (UID: \"4fe33109-ef55-4c16-935f-cc6280e9d58e\") " pod="openshift-cluster-node-tuning-operator/tuned-dxs8z" Jul 22 21:05:47.346223 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.345757 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2d3ff930-d226-4a49-9cb5-aa7d02eb37af-hostroot\") pod \"multus-kpw4k\" (UID: \"2d3ff930-d226-4a49-9cb5-aa7d02eb37af\") " pod="openshift-multus/multus-kpw4k" Jul 22 21:05:47.346223 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.345783 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c29841d1-b93c-4f1e-9455-477c2208081f-host-cni-bin\") pod \"ovnkube-node-7w87p\" (UID: \"c29841d1-b93c-4f1e-9455-477c2208081f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w87p" Jul 22 21:05:47.346223 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.345810 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cf2ac247-7e8e-46b0-991e-e82a384991c1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-42g4d\" (UID: \"cf2ac247-7e8e-46b0-991e-e82a384991c1\") " pod="openshift-multus/multus-additional-cni-plugins-42g4d" Jul 22 21:05:47.346223 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.345837 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4fe33109-ef55-4c16-935f-cc6280e9d58e-etc-tuned\") pod \"tuned-dxs8z\" (UID: \"4fe33109-ef55-4c16-935f-cc6280e9d58e\") " pod="openshift-cluster-node-tuning-operator/tuned-dxs8z" Jul 22 21:05:47.346223 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.345874 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2d3ff930-d226-4a49-9cb5-aa7d02eb37af-host-run-k8s-cni-cncf-io\") pod \"multus-kpw4k\" (UID: \"2d3ff930-d226-4a49-9cb5-aa7d02eb37af\") " pod="openshift-multus/multus-kpw4k" Jul 22 21:05:47.346223 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.345901 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2d3ff930-d226-4a49-9cb5-aa7d02eb37af-host-var-lib-cni-bin\") pod \"multus-kpw4k\" (UID: \"2d3ff930-d226-4a49-9cb5-aa7d02eb37af\") " pod="openshift-multus/multus-kpw4k" Jul 22 21:05:47.346223 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.345926 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2d3ff930-d226-4a49-9cb5-aa7d02eb37af-multus-conf-dir\") pod \"multus-kpw4k\" (UID: \"2d3ff930-d226-4a49-9cb5-aa7d02eb37af\") " pod="openshift-multus/multus-kpw4k" Jul 22 21:05:47.346223 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.345953 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d3ff930-d226-4a49-9cb5-aa7d02eb37af-etc-kubernetes\") pod \"multus-kpw4k\" (UID: \"2d3ff930-d226-4a49-9cb5-aa7d02eb37af\") " pod="openshift-multus/multus-kpw4k" Jul 22 21:05:47.346223 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.345978 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/37b20c60-c50a-42df-acd7-2164653d94ea-agent-certs\") pod \"konnectivity-agent-m7pxh\" (UID: \"37b20c60-c50a-42df-acd7-2164653d94ea\") " pod="kube-system/konnectivity-agent-m7pxh" Jul 22 21:05:47.346223 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.346004 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c29841d1-b93c-4f1e-9455-477c2208081f-host-run-ovn-kubernetes\") pod \"ovnkube-node-7w87p\" (UID: \"c29841d1-b93c-4f1e-9455-477c2208081f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w87p" Jul 22 21:05:47.346223 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.346031 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c29841d1-b93c-4f1e-9455-477c2208081f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7w87p\" (UID: \"c29841d1-b93c-4f1e-9455-477c2208081f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w87p" Jul 22 21:05:47.346223 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.346060 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c29841d1-b93c-4f1e-9455-477c2208081f-ovnkube-script-lib\") pod \"ovnkube-node-7w87p\" (UID: \"c29841d1-b93c-4f1e-9455-477c2208081f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w87p" Jul 22 21:05:47.346223 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.346086 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2d3ff930-d226-4a49-9cb5-aa7d02eb37af-cni-binary-copy\") pod \"multus-kpw4k\" (UID: \"2d3ff930-d226-4a49-9cb5-aa7d02eb37af\") " pod="openshift-multus/multus-kpw4k" Jul 22 21:05:47.346887 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.346110 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/37b20c60-c50a-42df-acd7-2164653d94ea-konnectivity-ca\") pod \"konnectivity-agent-m7pxh\" (UID: \"37b20c60-c50a-42df-acd7-2164653d94ea\") " pod="kube-system/konnectivity-agent-m7pxh" Jul 22 21:05:47.346887 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.346112 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2d3ff930-d226-4a49-9cb5-aa7d02eb37af-host-run-netns\") pod \"multus-kpw4k\" (UID: \"2d3ff930-d226-4a49-9cb5-aa7d02eb37af\") " pod="openshift-multus/multus-kpw4k" Jul 22 21:05:47.346887 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.346154 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c29841d1-b93c-4f1e-9455-477c2208081f-ovn-node-metrics-cert\") pod \"ovnkube-node-7w87p\" (UID: \"c29841d1-b93c-4f1e-9455-477c2208081f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w87p" Jul 22 21:05:47.346887 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.346174 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2d3ff930-d226-4a49-9cb5-aa7d02eb37af-hostroot\") pod \"multus-kpw4k\" (UID: \"2d3ff930-d226-4a49-9cb5-aa7d02eb37af\") " pod="openshift-multus/multus-kpw4k" Jul 22 21:05:47.346887 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.346185 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65c4s\" (UniqueName: \"kubernetes.io/projected/cf2ac247-7e8e-46b0-991e-e82a384991c1-kube-api-access-65c4s\") pod \"multus-additional-cni-plugins-42g4d\" (UID: \"cf2ac247-7e8e-46b0-991e-e82a384991c1\") " pod="openshift-multus/multus-additional-cni-plugins-42g4d" Jul 22 21:05:47.346887 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.346239 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3e6d7a69-6a07-40bf-9b57-c7a4828cb08a-host-slash\") pod \"iptables-alerter-rdlwm\" (UID: \"3e6d7a69-6a07-40bf-9b57-c7a4828cb08a\") " pod="openshift-network-operator/iptables-alerter-rdlwm" Jul 22 21:05:47.346887 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.346260 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d3ff930-d226-4a49-9cb5-aa7d02eb37af-etc-kubernetes\") pod \"multus-kpw4k\" (UID: \"2d3ff930-d226-4a49-9cb5-aa7d02eb37af\") " pod="openshift-multus/multus-kpw4k" Jul 22 21:05:47.346887 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.346266 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2d3ff930-d226-4a49-9cb5-aa7d02eb37af-multus-daemon-config\") pod \"multus-kpw4k\" (UID: \"2d3ff930-d226-4a49-9cb5-aa7d02eb37af\") " pod="openshift-multus/multus-kpw4k" Jul 22 21:05:47.346887 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.346295 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c29841d1-b93c-4f1e-9455-477c2208081f-host-slash\") pod \"ovnkube-node-7w87p\" (UID: \"c29841d1-b93c-4f1e-9455-477c2208081f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w87p" Jul 22 21:05:47.346887 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.346372 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c29841d1-b93c-4f1e-9455-477c2208081f-log-socket\") pod \"ovnkube-node-7w87p\" (UID: \"c29841d1-b93c-4f1e-9455-477c2208081f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w87p" Jul 22 21:05:47.346887 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.346398 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a449924b-6d2d-40f9-97a1-584a066510a3-registration-dir\") pod \"aws-ebs-csi-driver-node-87wqc\" (UID: \"a449924b-6d2d-40f9-97a1-584a066510a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-87wqc" Jul 22 21:05:47.346887 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.346424 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40f77f02-2bc6-4339-a948-7631a9603ce1-tmp-dir\") pod \"node-resolver-z2g8w\" (UID: \"40f77f02-2bc6-4339-a948-7631a9603ce1\") " pod="openshift-dns/node-resolver-z2g8w" Jul 22 21:05:47.346887 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.346448 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4fe33109-ef55-4c16-935f-cc6280e9d58e-etc-sysconfig\") pod \"tuned-dxs8z\" (UID: \"4fe33109-ef55-4c16-935f-cc6280e9d58e\") " pod="openshift-cluster-node-tuning-operator/tuned-dxs8z" Jul 22 21:05:47.346887 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.346472 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4fe33109-ef55-4c16-935f-cc6280e9d58e-etc-sysctl-d\") pod \"tuned-dxs8z\" (UID: \"4fe33109-ef55-4c16-935f-cc6280e9d58e\") " pod="openshift-cluster-node-tuning-operator/tuned-dxs8z" Jul 22 21:05:47.346887 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.346473 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3e6d7a69-6a07-40bf-9b57-c7a4828cb08a-host-slash\") pod \"iptables-alerter-rdlwm\" (UID: \"3e6d7a69-6a07-40bf-9b57-c7a4828cb08a\") " pod="openshift-network-operator/iptables-alerter-rdlwm" Jul 22 21:05:47.346887 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.346578 2572 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 22 21:05:47.347537 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.347023 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2d3ff930-d226-4a49-9cb5-aa7d02eb37af-multus-daemon-config\") pod \"multus-kpw4k\" (UID: \"2d3ff930-d226-4a49-9cb5-aa7d02eb37af\") " pod="openshift-multus/multus-kpw4k" Jul 22 21:05:47.347537 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.347066 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a91c48d-374d-4fc3-9114-d5c88ea18f63-metrics-certs\") pod \"network-metrics-daemon-lj48g\" (UID: \"8a91c48d-374d-4fc3-9114-d5c88ea18f63\") " pod="openshift-multus/network-metrics-daemon-lj48g" Jul 22 21:05:47.347537 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.347091 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25hmb\" (UniqueName: \"kubernetes.io/projected/3e6d7a69-6a07-40bf-9b57-c7a4828cb08a-kube-api-access-25hmb\") pod \"iptables-alerter-rdlwm\" (UID: \"3e6d7a69-6a07-40bf-9b57-c7a4828cb08a\") " pod="openshift-network-operator/iptables-alerter-rdlwm" Jul 22 21:05:47.347537 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.347112 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2d3ff930-d226-4a49-9cb5-aa7d02eb37af-os-release\") pod \"multus-kpw4k\" (UID: \"2d3ff930-d226-4a49-9cb5-aa7d02eb37af\") " pod="openshift-multus/multus-kpw4k" Jul 22 21:05:47.347537 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.347136 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c29841d1-b93c-4f1e-9455-477c2208081f-env-overrides\") pod \"ovnkube-node-7w87p\" (UID: \"c29841d1-b93c-4f1e-9455-477c2208081f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w87p" Jul 22 21:05:47.347537 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.347156 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4fe33109-ef55-4c16-935f-cc6280e9d58e-etc-systemd\") pod \"tuned-dxs8z\" (UID: \"4fe33109-ef55-4c16-935f-cc6280e9d58e\") " pod="openshift-cluster-node-tuning-operator/tuned-dxs8z" Jul 22 21:05:47.347537 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.347178 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4fe33109-ef55-4c16-935f-cc6280e9d58e-var-lib-kubelet\") pod \"tuned-dxs8z\" (UID: \"4fe33109-ef55-4c16-935f-cc6280e9d58e\") " pod="openshift-cluster-node-tuning-operator/tuned-dxs8z" Jul 22 21:05:47.347537 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.347199 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4fe33109-ef55-4c16-935f-cc6280e9d58e-host\") pod \"tuned-dxs8z\" (UID: \"4fe33109-ef55-4c16-935f-cc6280e9d58e\") " pod="openshift-cluster-node-tuning-operator/tuned-dxs8z" Jul 22 21:05:47.347537 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.347218 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c29841d1-b93c-4f1e-9455-477c2208081f-run-openvswitch\") pod \"ovnkube-node-7w87p\" (UID: \"c29841d1-b93c-4f1e-9455-477c2208081f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w87p" Jul 22 21:05:47.347537 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.347263 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqf67\" (UniqueName: \"kubernetes.io/projected/07cbf59d-dabc-4ba4-92b8-ec63306521cf-kube-api-access-hqf67\") pod \"network-check-target-tn98z\" (UID: \"07cbf59d-dabc-4ba4-92b8-ec63306521cf\") " pod="openshift-network-diagnostics/network-check-target-tn98z" Jul 22 21:05:47.347537 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.347288 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a449924b-6d2d-40f9-97a1-584a066510a3-socket-dir\") pod \"aws-ebs-csi-driver-node-87wqc\" (UID: \"a449924b-6d2d-40f9-97a1-584a066510a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-87wqc" Jul 22 21:05:47.347537 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.346519 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2d3ff930-d226-4a49-9cb5-aa7d02eb37af-host-run-k8s-cni-cncf-io\") pod \"multus-kpw4k\" (UID: \"2d3ff930-d226-4a49-9cb5-aa7d02eb37af\") " pod="openshift-multus/multus-kpw4k" Jul 22 21:05:47.347537 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.347379 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2d3ff930-d226-4a49-9cb5-aa7d02eb37af-host-var-lib-cni-bin\") pod \"multus-kpw4k\" (UID: \"2d3ff930-d226-4a49-9cb5-aa7d02eb37af\") " pod="openshift-multus/multus-kpw4k" Jul 22 21:05:47.347537 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:05:47.347512 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 21:05:47.348076 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:05:47.347758 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a91c48d-374d-4fc3-9114-d5c88ea18f63-metrics-certs podName:8a91c48d-374d-4fc3-9114-d5c88ea18f63 nodeName:}" failed. No retries permitted until 2026-07-22 21:05:47.847633772 +0000 UTC m=+2.962026097 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a91c48d-374d-4fc3-9114-d5c88ea18f63-metrics-certs") pod "network-metrics-daemon-lj48g" (UID: "8a91c48d-374d-4fc3-9114-d5c88ea18f63") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 21:05:47.348076 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.347880 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2d3ff930-d226-4a49-9cb5-aa7d02eb37af-os-release\") pod \"multus-kpw4k\" (UID: \"2d3ff930-d226-4a49-9cb5-aa7d02eb37af\") " pod="openshift-multus/multus-kpw4k" Jul 22 21:05:47.348076 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.347939 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2d3ff930-d226-4a49-9cb5-aa7d02eb37af-multus-conf-dir\") pod \"multus-kpw4k\" (UID: \"2d3ff930-d226-4a49-9cb5-aa7d02eb37af\") " pod="openshift-multus/multus-kpw4k" Jul 22 21:05:47.348076 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.347983 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2d3ff930-d226-4a49-9cb5-aa7d02eb37af-host-run-netns\") pod \"multus-kpw4k\" (UID: \"2d3ff930-d226-4a49-9cb5-aa7d02eb37af\") " pod="openshift-multus/multus-kpw4k" Jul 22 21:05:47.348265 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.348232 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2d3ff930-d226-4a49-9cb5-aa7d02eb37af-cni-binary-copy\") pod \"multus-kpw4k\" (UID: \"2d3ff930-d226-4a49-9cb5-aa7d02eb37af\") " pod="openshift-multus/multus-kpw4k" Jul 22 21:05:47.350462 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.350438 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/37b20c60-c50a-42df-acd7-2164653d94ea-agent-certs\") pod \"konnectivity-agent-m7pxh\" (UID: \"37b20c60-c50a-42df-acd7-2164653d94ea\") " pod="kube-system/konnectivity-agent-m7pxh" Jul 22 21:05:47.353540 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.353520 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slbtr\" (UniqueName: \"kubernetes.io/projected/8a91c48d-374d-4fc3-9114-d5c88ea18f63-kube-api-access-slbtr\") pod \"network-metrics-daemon-lj48g\" (UID: \"8a91c48d-374d-4fc3-9114-d5c88ea18f63\") " pod="openshift-multus/network-metrics-daemon-lj48g" Jul 22 21:05:47.353844 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.353815 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wz58x\" (UniqueName: \"kubernetes.io/projected/2d3ff930-d226-4a49-9cb5-aa7d02eb37af-kube-api-access-wz58x\") pod \"multus-kpw4k\" (UID: \"2d3ff930-d226-4a49-9cb5-aa7d02eb37af\") " pod="openshift-multus/multus-kpw4k" Jul 22 21:05:47.358232 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.358210 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25hmb\" (UniqueName: \"kubernetes.io/projected/3e6d7a69-6a07-40bf-9b57-c7a4828cb08a-kube-api-access-25hmb\") pod \"iptables-alerter-rdlwm\" (UID: \"3e6d7a69-6a07-40bf-9b57-c7a4828cb08a\") " pod="openshift-network-operator/iptables-alerter-rdlwm" Jul 22 21:05:47.381099 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.381058 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-37.ec2.internal" event={"ID":"e462d762995f2dc89c7e851abf11f67f","Type":"ContainerStarted","Data":"602e6359baeeb08c973d7a69f6b48a0c624a59ef52f7a4c4683fb2b3c22e6936"} Jul 22 21:05:47.424882 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.424855 2572 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 21:05:47.448532 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.448505 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cf2ac247-7e8e-46b0-991e-e82a384991c1-cni-binary-copy\") pod \"multus-additional-cni-plugins-42g4d\" (UID: \"cf2ac247-7e8e-46b0-991e-e82a384991c1\") " pod="openshift-multus/multus-additional-cni-plugins-42g4d" Jul 22 21:05:47.448666 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.448537 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a449924b-6d2d-40f9-97a1-584a066510a3-sys-fs\") pod \"aws-ebs-csi-driver-node-87wqc\" (UID: \"a449924b-6d2d-40f9-97a1-584a066510a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-87wqc" Jul 22 21:05:47.448666 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.448555 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbcq7\" (UniqueName: \"kubernetes.io/projected/a449924b-6d2d-40f9-97a1-584a066510a3-kube-api-access-sbcq7\") pod \"aws-ebs-csi-driver-node-87wqc\" (UID: \"a449924b-6d2d-40f9-97a1-584a066510a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-87wqc" Jul 22 21:05:47.448666 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.448582 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4f3c7912-ed80-4a24-ada3-b2a3033bb210-kubelet-config\") pod \"global-pull-secret-syncer-zrg2r\" (UID: \"4f3c7912-ed80-4a24-ada3-b2a3033bb210\") " pod="kube-system/global-pull-secret-syncer-zrg2r" Jul 22 21:05:47.448666 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.448662 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4f3c7912-ed80-4a24-ada3-b2a3033bb210-kubelet-config\") pod \"global-pull-secret-syncer-zrg2r\" (UID: \"4f3c7912-ed80-4a24-ada3-b2a3033bb210\") " pod="kube-system/global-pull-secret-syncer-zrg2r" Jul 22 21:05:47.448898 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.448670 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a449924b-6d2d-40f9-97a1-584a066510a3-sys-fs\") pod \"aws-ebs-csi-driver-node-87wqc\" (UID: \"a449924b-6d2d-40f9-97a1-584a066510a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-87wqc" Jul 22 21:05:47.448898 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.448699 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c29841d1-b93c-4f1e-9455-477c2208081f-run-systemd\") pod \"ovnkube-node-7w87p\" (UID: \"c29841d1-b93c-4f1e-9455-477c2208081f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w87p" Jul 22 21:05:47.448898 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.448726 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c29841d1-b93c-4f1e-9455-477c2208081f-etc-openvswitch\") pod \"ovnkube-node-7w87p\" (UID: \"c29841d1-b93c-4f1e-9455-477c2208081f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w87p" Jul 22 21:05:47.448898 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.448751 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4fe33109-ef55-4c16-935f-cc6280e9d58e-etc-kubernetes\") pod \"tuned-dxs8z\" (UID: \"4fe33109-ef55-4c16-935f-cc6280e9d58e\") " pod="openshift-cluster-node-tuning-operator/tuned-dxs8z" Jul 22 21:05:47.448898 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.448777 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4fe33109-ef55-4c16-935f-cc6280e9d58e-run\") pod \"tuned-dxs8z\" (UID: \"4fe33109-ef55-4c16-935f-cc6280e9d58e\") " pod="openshift-cluster-node-tuning-operator/tuned-dxs8z" Jul 22 21:05:47.448898 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.448804 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6q5s\" (UniqueName: \"kubernetes.io/projected/4fe33109-ef55-4c16-935f-cc6280e9d58e-kube-api-access-z6q5s\") pod \"tuned-dxs8z\" (UID: \"4fe33109-ef55-4c16-935f-cc6280e9d58e\") " pod="openshift-cluster-node-tuning-operator/tuned-dxs8z" Jul 22 21:05:47.448898 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.448824 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c29841d1-b93c-4f1e-9455-477c2208081f-etc-openvswitch\") pod \"ovnkube-node-7w87p\" (UID: \"c29841d1-b93c-4f1e-9455-477c2208081f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w87p" Jul 22 21:05:47.448898 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.448855 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c29841d1-b93c-4f1e-9455-477c2208081f-run-systemd\") pod \"ovnkube-node-7w87p\" (UID: \"c29841d1-b93c-4f1e-9455-477c2208081f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w87p" Jul 22 21:05:47.448898 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.448830 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c29841d1-b93c-4f1e-9455-477c2208081f-host-cni-bin\") pod \"ovnkube-node-7w87p\" (UID: \"c29841d1-b93c-4f1e-9455-477c2208081f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w87p" Jul 22 21:05:47.448898 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.448875 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c29841d1-b93c-4f1e-9455-477c2208081f-host-cni-bin\") pod \"ovnkube-node-7w87p\" (UID: \"c29841d1-b93c-4f1e-9455-477c2208081f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w87p" Jul 22 21:05:47.448898 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.448906 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cf2ac247-7e8e-46b0-991e-e82a384991c1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-42g4d\" (UID: \"cf2ac247-7e8e-46b0-991e-e82a384991c1\") " pod="openshift-multus/multus-additional-cni-plugins-42g4d" Jul 22 21:05:47.449562 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.448930 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4fe33109-ef55-4c16-935f-cc6280e9d58e-run\") pod \"tuned-dxs8z\" (UID: \"4fe33109-ef55-4c16-935f-cc6280e9d58e\") " pod="openshift-cluster-node-tuning-operator/tuned-dxs8z" Jul 22 21:05:47.449562 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.449276 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cf2ac247-7e8e-46b0-991e-e82a384991c1-cni-binary-copy\") pod \"multus-additional-cni-plugins-42g4d\" (UID: \"cf2ac247-7e8e-46b0-991e-e82a384991c1\") " pod="openshift-multus/multus-additional-cni-plugins-42g4d" Jul 22 21:05:47.449562 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.449373 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4fe33109-ef55-4c16-935f-cc6280e9d58e-etc-tuned\") pod \"tuned-dxs8z\" (UID: \"4fe33109-ef55-4c16-935f-cc6280e9d58e\") " pod="openshift-cluster-node-tuning-operator/tuned-dxs8z" Jul 22 21:05:47.449562 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.449418 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c29841d1-b93c-4f1e-9455-477c2208081f-host-run-ovn-kubernetes\") pod \"ovnkube-node-7w87p\" (UID: \"c29841d1-b93c-4f1e-9455-477c2208081f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w87p" Jul 22 21:05:47.449562 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.449437 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c29841d1-b93c-4f1e-9455-477c2208081f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7w87p\" (UID: \"c29841d1-b93c-4f1e-9455-477c2208081f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w87p" Jul 22 21:05:47.449562 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.449454 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c29841d1-b93c-4f1e-9455-477c2208081f-ovnkube-script-lib\") pod \"ovnkube-node-7w87p\" (UID: \"c29841d1-b93c-4f1e-9455-477c2208081f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w87p" Jul 22 21:05:47.449562 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.449471 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c29841d1-b93c-4f1e-9455-477c2208081f-ovn-node-metrics-cert\") pod \"ovnkube-node-7w87p\" (UID: \"c29841d1-b93c-4f1e-9455-477c2208081f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w87p" Jul 22 21:05:47.449562 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.449523 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4fe33109-ef55-4c16-935f-cc6280e9d58e-etc-kubernetes\") pod \"tuned-dxs8z\" (UID: \"4fe33109-ef55-4c16-935f-cc6280e9d58e\") " pod="openshift-cluster-node-tuning-operator/tuned-dxs8z" Jul 22 21:05:47.449562 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.449565 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65c4s\" (UniqueName: \"kubernetes.io/projected/cf2ac247-7e8e-46b0-991e-e82a384991c1-kube-api-access-65c4s\") pod \"multus-additional-cni-plugins-42g4d\" (UID: \"cf2ac247-7e8e-46b0-991e-e82a384991c1\") " pod="openshift-multus/multus-additional-cni-plugins-42g4d" Jul 22 21:05:47.450092 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.449606 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c29841d1-b93c-4f1e-9455-477c2208081f-host-slash\") pod \"ovnkube-node-7w87p\" (UID: \"c29841d1-b93c-4f1e-9455-477c2208081f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w87p" Jul 22 21:05:47.450092 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.449634 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c29841d1-b93c-4f1e-9455-477c2208081f-log-socket\") pod \"ovnkube-node-7w87p\" (UID: \"c29841d1-b93c-4f1e-9455-477c2208081f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w87p" Jul 22 21:05:47.450092 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.449681 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c29841d1-b93c-4f1e-9455-477c2208081f-log-socket\") pod \"ovnkube-node-7w87p\" (UID: \"c29841d1-b93c-4f1e-9455-477c2208081f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w87p" Jul 22 21:05:47.450092 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.449727 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c29841d1-b93c-4f1e-9455-477c2208081f-host-slash\") pod \"ovnkube-node-7w87p\" (UID: \"c29841d1-b93c-4f1e-9455-477c2208081f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w87p" Jul 22 21:05:47.450092 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.449734 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cf2ac247-7e8e-46b0-991e-e82a384991c1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-42g4d\" (UID: \"cf2ac247-7e8e-46b0-991e-e82a384991c1\") " pod="openshift-multus/multus-additional-cni-plugins-42g4d" Jul 22 21:05:47.450092 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.449769 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c29841d1-b93c-4f1e-9455-477c2208081f-host-run-ovn-kubernetes\") pod \"ovnkube-node-7w87p\" (UID: \"c29841d1-b93c-4f1e-9455-477c2208081f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w87p" Jul 22 21:05:47.450092 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.449786 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a449924b-6d2d-40f9-97a1-584a066510a3-registration-dir\") pod \"aws-ebs-csi-driver-node-87wqc\" (UID: \"a449924b-6d2d-40f9-97a1-584a066510a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-87wqc" Jul 22 21:05:47.450092 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.449798 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c29841d1-b93c-4f1e-9455-477c2208081f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7w87p\" (UID: \"c29841d1-b93c-4f1e-9455-477c2208081f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w87p" Jul 22 21:05:47.450092 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.449847 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40f77f02-2bc6-4339-a948-7631a9603ce1-tmp-dir\") pod \"node-resolver-z2g8w\" (UID: \"40f77f02-2bc6-4339-a948-7631a9603ce1\") " pod="openshift-dns/node-resolver-z2g8w" Jul 22 21:05:47.450092 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.449866 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a449924b-6d2d-40f9-97a1-584a066510a3-registration-dir\") pod \"aws-ebs-csi-driver-node-87wqc\" (UID: \"a449924b-6d2d-40f9-97a1-584a066510a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-87wqc" Jul 22 21:05:47.450092 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.449903 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4fe33109-ef55-4c16-935f-cc6280e9d58e-etc-sysconfig\") pod \"tuned-dxs8z\" (UID: \"4fe33109-ef55-4c16-935f-cc6280e9d58e\") " pod="openshift-cluster-node-tuning-operator/tuned-dxs8z" Jul 22 21:05:47.450092 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.449929 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4fe33109-ef55-4c16-935f-cc6280e9d58e-etc-sysctl-d\") pod \"tuned-dxs8z\" (UID: \"4fe33109-ef55-4c16-935f-cc6280e9d58e\") " pod="openshift-cluster-node-tuning-operator/tuned-dxs8z" Jul 22 21:05:47.450092 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.450013 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4fe33109-ef55-4c16-935f-cc6280e9d58e-etc-sysconfig\") pod \"tuned-dxs8z\" (UID: \"4fe33109-ef55-4c16-935f-cc6280e9d58e\") " pod="openshift-cluster-node-tuning-operator/tuned-dxs8z" Jul 22 21:05:47.450092 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.450051 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c29841d1-b93c-4f1e-9455-477c2208081f-env-overrides\") pod \"ovnkube-node-7w87p\" (UID: \"c29841d1-b93c-4f1e-9455-477c2208081f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w87p" Jul 22 21:05:47.450092 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.450079 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4fe33109-ef55-4c16-935f-cc6280e9d58e-etc-systemd\") pod \"tuned-dxs8z\" (UID: \"4fe33109-ef55-4c16-935f-cc6280e9d58e\") " pod="openshift-cluster-node-tuning-operator/tuned-dxs8z" Jul 22 21:05:47.450690 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.450106 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4fe33109-ef55-4c16-935f-cc6280e9d58e-var-lib-kubelet\") pod \"tuned-dxs8z\" (UID: \"4fe33109-ef55-4c16-935f-cc6280e9d58e\") " pod="openshift-cluster-node-tuning-operator/tuned-dxs8z" Jul 22 21:05:47.450690 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.450130 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4fe33109-ef55-4c16-935f-cc6280e9d58e-host\") pod \"tuned-dxs8z\" (UID: \"4fe33109-ef55-4c16-935f-cc6280e9d58e\") " pod="openshift-cluster-node-tuning-operator/tuned-dxs8z" Jul 22 21:05:47.450690 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.450148 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40f77f02-2bc6-4339-a948-7631a9603ce1-tmp-dir\") pod \"node-resolver-z2g8w\" (UID: \"40f77f02-2bc6-4339-a948-7631a9603ce1\") " pod="openshift-dns/node-resolver-z2g8w" Jul 22 21:05:47.450690 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.450153 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c29841d1-b93c-4f1e-9455-477c2208081f-run-openvswitch\") pod \"ovnkube-node-7w87p\" (UID: \"c29841d1-b93c-4f1e-9455-477c2208081f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w87p" Jul 22 21:05:47.450690 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.450203 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4fe33109-ef55-4c16-935f-cc6280e9d58e-etc-systemd\") pod \"tuned-dxs8z\" (UID: \"4fe33109-ef55-4c16-935f-cc6280e9d58e\") " pod="openshift-cluster-node-tuning-operator/tuned-dxs8z" Jul 22 21:05:47.450690 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.450206 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c29841d1-b93c-4f1e-9455-477c2208081f-run-openvswitch\") pod \"ovnkube-node-7w87p\" (UID: \"c29841d1-b93c-4f1e-9455-477c2208081f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w87p" Jul 22 21:05:47.450690 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.450217 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqf67\" (UniqueName: \"kubernetes.io/projected/07cbf59d-dabc-4ba4-92b8-ec63306521cf-kube-api-access-hqf67\") pod \"network-check-target-tn98z\" (UID: \"07cbf59d-dabc-4ba4-92b8-ec63306521cf\") " pod="openshift-network-diagnostics/network-check-target-tn98z" Jul 22 21:05:47.450690 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.450247 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a449924b-6d2d-40f9-97a1-584a066510a3-socket-dir\") pod \"aws-ebs-csi-driver-node-87wqc\" (UID: \"a449924b-6d2d-40f9-97a1-584a066510a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-87wqc" Jul 22 21:05:47.450690 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.450268 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4fe33109-ef55-4c16-935f-cc6280e9d58e-var-lib-kubelet\") pod \"tuned-dxs8z\" (UID: \"4fe33109-ef55-4c16-935f-cc6280e9d58e\") " pod="openshift-cluster-node-tuning-operator/tuned-dxs8z" Jul 22 21:05:47.450690 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.450274 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hwlt\" (UniqueName: \"kubernetes.io/projected/40f77f02-2bc6-4339-a948-7631a9603ce1-kube-api-access-8hwlt\") pod \"node-resolver-z2g8w\" (UID: \"40f77f02-2bc6-4339-a948-7631a9603ce1\") " pod="openshift-dns/node-resolver-z2g8w" Jul 22 21:05:47.450690 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.450299 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/45b1939d-7d91-4899-b489-7e2609a5878d-serviceca\") pod \"node-ca-4s9cs\" (UID: \"45b1939d-7d91-4899-b489-7e2609a5878d\") " pod="openshift-image-registry/node-ca-4s9cs" Jul 22 21:05:47.450690 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.450323 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c29841d1-b93c-4f1e-9455-477c2208081f-host-kubelet\") pod \"ovnkube-node-7w87p\" (UID: \"c29841d1-b93c-4f1e-9455-477c2208081f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w87p" Jul 22 21:05:47.450690 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.450314 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4fe33109-ef55-4c16-935f-cc6280e9d58e-host\") pod \"tuned-dxs8z\" (UID: \"4fe33109-ef55-4c16-935f-cc6280e9d58e\") " pod="openshift-cluster-node-tuning-operator/tuned-dxs8z" Jul 22 21:05:47.450690 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.450347 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c29841d1-b93c-4f1e-9455-477c2208081f-node-log\") pod \"ovnkube-node-7w87p\" (UID: \"c29841d1-b93c-4f1e-9455-477c2208081f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w87p" Jul 22 21:05:47.450690 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.450359 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4fe33109-ef55-4c16-935f-cc6280e9d58e-etc-sysctl-d\") pod \"tuned-dxs8z\" (UID: \"4fe33109-ef55-4c16-935f-cc6280e9d58e\") " pod="openshift-cluster-node-tuning-operator/tuned-dxs8z" Jul 22 21:05:47.450690 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.450371 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/40f77f02-2bc6-4339-a948-7631a9603ce1-hosts-file\") pod \"node-resolver-z2g8w\" (UID: \"40f77f02-2bc6-4339-a948-7631a9603ce1\") " pod="openshift-dns/node-resolver-z2g8w" Jul 22 21:05:47.450690 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.450395 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4fe33109-ef55-4c16-935f-cc6280e9d58e-etc-sysctl-conf\") pod \"tuned-dxs8z\" (UID: \"4fe33109-ef55-4c16-935f-cc6280e9d58e\") " pod="openshift-cluster-node-tuning-operator/tuned-dxs8z" Jul 22 21:05:47.450690 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.450418 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/45b1939d-7d91-4899-b489-7e2609a5878d-host\") pod \"node-ca-4s9cs\" (UID: \"45b1939d-7d91-4899-b489-7e2609a5878d\") " pod="openshift-image-registry/node-ca-4s9cs" Jul 22 21:05:47.451515 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.450444 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcr84\" (UniqueName: \"kubernetes.io/projected/c29841d1-b93c-4f1e-9455-477c2208081f-kube-api-access-tcr84\") pod \"ovnkube-node-7w87p\" (UID: \"c29841d1-b93c-4f1e-9455-477c2208081f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w87p" Jul 22 21:05:47.451515 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.450468 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cf2ac247-7e8e-46b0-991e-e82a384991c1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-42g4d\" (UID: \"cf2ac247-7e8e-46b0-991e-e82a384991c1\") " pod="openshift-multus/multus-additional-cni-plugins-42g4d" Jul 22 21:05:47.451515 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.450495 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4f3c7912-ed80-4a24-ada3-b2a3033bb210-original-pull-secret\") pod \"global-pull-secret-syncer-zrg2r\" (UID: \"4f3c7912-ed80-4a24-ada3-b2a3033bb210\") " pod="kube-system/global-pull-secret-syncer-zrg2r" Jul 22 21:05:47.451515 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.450518 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cf2ac247-7e8e-46b0-991e-e82a384991c1-os-release\") pod \"multus-additional-cni-plugins-42g4d\" (UID: \"cf2ac247-7e8e-46b0-991e-e82a384991c1\") " pod="openshift-multus/multus-additional-cni-plugins-42g4d" Jul 22 21:05:47.451515 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.450541 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cf2ac247-7e8e-46b0-991e-e82a384991c1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-42g4d\" (UID: \"cf2ac247-7e8e-46b0-991e-e82a384991c1\") " pod="openshift-multus/multus-additional-cni-plugins-42g4d" Jul 22 21:05:47.451515 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.450565 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4fe33109-ef55-4c16-935f-cc6280e9d58e-etc-modprobe-d\") pod \"tuned-dxs8z\" (UID: \"4fe33109-ef55-4c16-935f-cc6280e9d58e\") " pod="openshift-cluster-node-tuning-operator/tuned-dxs8z" Jul 22 21:05:47.451515 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.450592 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c29841d1-b93c-4f1e-9455-477c2208081f-host-run-netns\") pod \"ovnkube-node-7w87p\" (UID: \"c29841d1-b93c-4f1e-9455-477c2208081f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w87p" Jul 22 21:05:47.451515 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.450615 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c29841d1-b93c-4f1e-9455-477c2208081f-host-cni-netd\") pod \"ovnkube-node-7w87p\" (UID: \"c29841d1-b93c-4f1e-9455-477c2208081f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w87p" Jul 22 21:05:47.451515 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.450639 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a449924b-6d2d-40f9-97a1-584a066510a3-device-dir\") pod \"aws-ebs-csi-driver-node-87wqc\" (UID: \"a449924b-6d2d-40f9-97a1-584a066510a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-87wqc" Jul 22 21:05:47.451515 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.450653 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c29841d1-b93c-4f1e-9455-477c2208081f-ovnkube-script-lib\") pod \"ovnkube-node-7w87p\" (UID: \"c29841d1-b93c-4f1e-9455-477c2208081f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w87p" Jul 22 21:05:47.451515 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.450665 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c29841d1-b93c-4f1e-9455-477c2208081f-systemd-units\") pod \"ovnkube-node-7w87p\" (UID: \"c29841d1-b93c-4f1e-9455-477c2208081f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w87p" Jul 22 21:05:47.451515 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.450675 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c29841d1-b93c-4f1e-9455-477c2208081f-env-overrides\") pod \"ovnkube-node-7w87p\" (UID: \"c29841d1-b93c-4f1e-9455-477c2208081f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w87p" Jul 22 21:05:47.451515 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.450690 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a449924b-6d2d-40f9-97a1-584a066510a3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-87wqc\" (UID: \"a449924b-6d2d-40f9-97a1-584a066510a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-87wqc" Jul 22 21:05:47.451515 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.450715 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a449924b-6d2d-40f9-97a1-584a066510a3-etc-selinux\") pod \"aws-ebs-csi-driver-node-87wqc\" (UID: \"a449924b-6d2d-40f9-97a1-584a066510a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-87wqc" Jul 22 21:05:47.451515 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.450738 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4fe33109-ef55-4c16-935f-cc6280e9d58e-sys\") pod \"tuned-dxs8z\" (UID: \"4fe33109-ef55-4c16-935f-cc6280e9d58e\") " pod="openshift-cluster-node-tuning-operator/tuned-dxs8z" Jul 22 21:05:47.451515 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.450761 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cf2ac247-7e8e-46b0-991e-e82a384991c1-system-cni-dir\") pod \"multus-additional-cni-plugins-42g4d\" (UID: \"cf2ac247-7e8e-46b0-991e-e82a384991c1\") " pod="openshift-multus/multus-additional-cni-plugins-42g4d" Jul 22 21:05:47.451515 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.450784 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4f3c7912-ed80-4a24-ada3-b2a3033bb210-dbus\") pod \"global-pull-secret-syncer-zrg2r\" (UID: \"4f3c7912-ed80-4a24-ada3-b2a3033bb210\") " pod="kube-system/global-pull-secret-syncer-zrg2r" Jul 22 21:05:47.452388 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:05:47.450791 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 21:05:47.452388 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.450807 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4fe33109-ef55-4c16-935f-cc6280e9d58e-lib-modules\") pod \"tuned-dxs8z\" (UID: \"4fe33109-ef55-4c16-935f-cc6280e9d58e\") " pod="openshift-cluster-node-tuning-operator/tuned-dxs8z" Jul 22 21:05:47.452388 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.450833 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wskpm\" (UniqueName: \"kubernetes.io/projected/45b1939d-7d91-4899-b489-7e2609a5878d-kube-api-access-wskpm\") pod \"node-ca-4s9cs\" (UID: \"45b1939d-7d91-4899-b489-7e2609a5878d\") " pod="openshift-image-registry/node-ca-4s9cs" Jul 22 21:05:47.452824 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.452799 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4f3c7912-ed80-4a24-ada3-b2a3033bb210-dbus\") pod \"global-pull-secret-syncer-zrg2r\" (UID: \"4f3c7912-ed80-4a24-ada3-b2a3033bb210\") " pod="kube-system/global-pull-secret-syncer-zrg2r" Jul 22 21:05:47.452889 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.451405 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4fe33109-ef55-4c16-935f-cc6280e9d58e-etc-sysctl-conf\") pod \"tuned-dxs8z\" (UID: \"4fe33109-ef55-4c16-935f-cc6280e9d58e\") " pod="openshift-cluster-node-tuning-operator/tuned-dxs8z" Jul 22 21:05:47.452889 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.451469 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/45b1939d-7d91-4899-b489-7e2609a5878d-host\") pod \"node-ca-4s9cs\" (UID: \"45b1939d-7d91-4899-b489-7e2609a5878d\") " pod="openshift-image-registry/node-ca-4s9cs" Jul 22 21:05:47.452889 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.451691 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cf2ac247-7e8e-46b0-991e-e82a384991c1-os-release\") pod \"multus-additional-cni-plugins-42g4d\" (UID: \"cf2ac247-7e8e-46b0-991e-e82a384991c1\") " pod="openshift-multus/multus-additional-cni-plugins-42g4d" Jul 22 21:05:47.452889 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.451972 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cf2ac247-7e8e-46b0-991e-e82a384991c1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-42g4d\" (UID: \"cf2ac247-7e8e-46b0-991e-e82a384991c1\") " pod="openshift-multus/multus-additional-cni-plugins-42g4d" Jul 22 21:05:47.452889 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.452875 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c29841d1-b93c-4f1e-9455-477c2208081f-var-lib-openvswitch\") pod \"ovnkube-node-7w87p\" (UID: \"c29841d1-b93c-4f1e-9455-477c2208081f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w87p" Jul 22 21:05:47.453134 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.452130 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c29841d1-b93c-4f1e-9455-477c2208081f-host-run-netns\") pod \"ovnkube-node-7w87p\" (UID: \"c29841d1-b93c-4f1e-9455-477c2208081f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w87p" Jul 22 21:05:47.453134 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.452176 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c29841d1-b93c-4f1e-9455-477c2208081f-host-cni-netd\") pod \"ovnkube-node-7w87p\" (UID: \"c29841d1-b93c-4f1e-9455-477c2208081f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w87p" Jul 22 21:05:47.453134 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.452218 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a449924b-6d2d-40f9-97a1-584a066510a3-device-dir\") pod \"aws-ebs-csi-driver-node-87wqc\" (UID: \"a449924b-6d2d-40f9-97a1-584a066510a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-87wqc" Jul 22 21:05:47.453134 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.452402 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c29841d1-b93c-4f1e-9455-477c2208081f-systemd-units\") pod \"ovnkube-node-7w87p\" (UID: \"c29841d1-b93c-4f1e-9455-477c2208081f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w87p" Jul 22 21:05:47.453134 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.450805 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/40f77f02-2bc6-4339-a948-7631a9603ce1-hosts-file\") pod \"node-resolver-z2g8w\" (UID: \"40f77f02-2bc6-4339-a948-7631a9603ce1\") " pod="openshift-dns/node-resolver-z2g8w" Jul 22 21:05:47.453134 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.452585 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c29841d1-b93c-4f1e-9455-477c2208081f-host-kubelet\") pod \"ovnkube-node-7w87p\" (UID: \"c29841d1-b93c-4f1e-9455-477c2208081f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w87p" Jul 22 21:05:47.453134 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.452592 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4fe33109-ef55-4c16-935f-cc6280e9d58e-sys\") pod \"tuned-dxs8z\" (UID: \"4fe33109-ef55-4c16-935f-cc6280e9d58e\") " pod="openshift-cluster-node-tuning-operator/tuned-dxs8z" Jul 22 21:05:47.453134 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.453011 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c29841d1-b93c-4f1e-9455-477c2208081f-run-ovn\") pod \"ovnkube-node-7w87p\" (UID: \"c29841d1-b93c-4f1e-9455-477c2208081f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w87p" Jul 22 21:05:47.453134 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.452632 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a449924b-6d2d-40f9-97a1-584a066510a3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-87wqc\" (UID: \"a449924b-6d2d-40f9-97a1-584a066510a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-87wqc" Jul 22 21:05:47.453134 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.452733 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a449924b-6d2d-40f9-97a1-584a066510a3-etc-selinux\") pod \"aws-ebs-csi-driver-node-87wqc\" (UID: \"a449924b-6d2d-40f9-97a1-584a066510a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-87wqc" Jul 22 21:05:47.453134 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.452087 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4fe33109-ef55-4c16-935f-cc6280e9d58e-etc-modprobe-d\") pod \"tuned-dxs8z\" (UID: \"4fe33109-ef55-4c16-935f-cc6280e9d58e\") " pod="openshift-cluster-node-tuning-operator/tuned-dxs8z" Jul 22 21:05:47.453134 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.452700 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cf2ac247-7e8e-46b0-991e-e82a384991c1-system-cni-dir\") pod \"multus-additional-cni-plugins-42g4d\" (UID: \"cf2ac247-7e8e-46b0-991e-e82a384991c1\") " pod="openshift-multus/multus-additional-cni-plugins-42g4d" Jul 22 21:05:47.453134 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.450808 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a449924b-6d2d-40f9-97a1-584a066510a3-socket-dir\") pod \"aws-ebs-csi-driver-node-87wqc\" (UID: \"a449924b-6d2d-40f9-97a1-584a066510a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-87wqc" Jul 22 21:05:47.453134 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.452408 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cf2ac247-7e8e-46b0-991e-e82a384991c1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-42g4d\" (UID: \"cf2ac247-7e8e-46b0-991e-e82a384991c1\") " pod="openshift-multus/multus-additional-cni-plugins-42g4d" Jul 22 21:05:47.453756 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.453277 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cf2ac247-7e8e-46b0-991e-e82a384991c1-cnibin\") pod \"multus-additional-cni-plugins-42g4d\" (UID: \"cf2ac247-7e8e-46b0-991e-e82a384991c1\") " pod="openshift-multus/multus-additional-cni-plugins-42g4d" Jul 22 21:05:47.453756 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.453316 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4fe33109-ef55-4c16-935f-cc6280e9d58e-tmp\") pod \"tuned-dxs8z\" (UID: \"4fe33109-ef55-4c16-935f-cc6280e9d58e\") " pod="openshift-cluster-node-tuning-operator/tuned-dxs8z" Jul 22 21:05:47.453756 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.453357 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c29841d1-b93c-4f1e-9455-477c2208081f-var-lib-openvswitch\") pod \"ovnkube-node-7w87p\" (UID: \"c29841d1-b93c-4f1e-9455-477c2208081f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w87p" Jul 22 21:05:47.453756 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.453384 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c29841d1-b93c-4f1e-9455-477c2208081f-run-ovn\") pod \"ovnkube-node-7w87p\" (UID: \"c29841d1-b93c-4f1e-9455-477c2208081f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w87p" Jul 22 21:05:47.453756 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.453410 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c29841d1-b93c-4f1e-9455-477c2208081f-node-log\") pod \"ovnkube-node-7w87p\" (UID: \"c29841d1-b93c-4f1e-9455-477c2208081f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w87p" Jul 22 21:05:47.453756 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.453438 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cf2ac247-7e8e-46b0-991e-e82a384991c1-cnibin\") pod \"multus-additional-cni-plugins-42g4d\" (UID: \"cf2ac247-7e8e-46b0-991e-e82a384991c1\") " pod="openshift-multus/multus-additional-cni-plugins-42g4d" Jul 22 21:05:47.453756 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.453665 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c29841d1-b93c-4f1e-9455-477c2208081f-ovnkube-config\") pod \"ovnkube-node-7w87p\" (UID: \"c29841d1-b93c-4f1e-9455-477c2208081f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w87p" Jul 22 21:05:47.454504 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.454298 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c29841d1-b93c-4f1e-9455-477c2208081f-ovnkube-config\") pod \"ovnkube-node-7w87p\" (UID: \"c29841d1-b93c-4f1e-9455-477c2208081f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w87p" Jul 22 21:05:47.454504 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.454316 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4fe33109-ef55-4c16-935f-cc6280e9d58e-etc-tuned\") pod \"tuned-dxs8z\" (UID: \"4fe33109-ef55-4c16-935f-cc6280e9d58e\") " pod="openshift-cluster-node-tuning-operator/tuned-dxs8z" Jul 22 21:05:47.454504 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:05:47.454421 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f3c7912-ed80-4a24-ada3-b2a3033bb210-original-pull-secret podName:4f3c7912-ed80-4a24-ada3-b2a3033bb210 nodeName:}" failed. No retries permitted until 2026-07-22 21:05:47.954399345 +0000 UTC m=+3.068791690 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4f3c7912-ed80-4a24-ada3-b2a3033bb210-original-pull-secret") pod "global-pull-secret-syncer-zrg2r" (UID: "4f3c7912-ed80-4a24-ada3-b2a3033bb210") : object "kube-system"/"original-pull-secret" not registered Jul 22 21:05:47.455118 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.455097 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/45b1939d-7d91-4899-b489-7e2609a5878d-serviceca\") pod \"node-ca-4s9cs\" (UID: \"45b1939d-7d91-4899-b489-7e2609a5878d\") " pod="openshift-image-registry/node-ca-4s9cs" Jul 22 21:05:47.455303 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.455222 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4fe33109-ef55-4c16-935f-cc6280e9d58e-lib-modules\") pod \"tuned-dxs8z\" (UID: \"4fe33109-ef55-4c16-935f-cc6280e9d58e\") " pod="openshift-cluster-node-tuning-operator/tuned-dxs8z" Jul 22 21:05:47.458196 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:05:47.456319 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 21:05:47.458196 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:05:47.456350 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 21:05:47.458196 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:05:47.456366 2572 projected.go:196] Error preparing data for projected volume kube-api-access-hqf67 for pod openshift-network-diagnostics/network-check-target-tn98z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 21:05:47.458196 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:05:47.456434 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07cbf59d-dabc-4ba4-92b8-ec63306521cf-kube-api-access-hqf67 podName:07cbf59d-dabc-4ba4-92b8-ec63306521cf nodeName:}" failed. No retries permitted until 2026-07-22 21:05:47.956416121 +0000 UTC m=+3.070808473 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-hqf67" (UniqueName: "kubernetes.io/projected/07cbf59d-dabc-4ba4-92b8-ec63306521cf-kube-api-access-hqf67") pod "network-check-target-tn98z" (UID: "07cbf59d-dabc-4ba4-92b8-ec63306521cf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 21:05:47.458196 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.457365 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c29841d1-b93c-4f1e-9455-477c2208081f-ovn-node-metrics-cert\") pod \"ovnkube-node-7w87p\" (UID: \"c29841d1-b93c-4f1e-9455-477c2208081f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w87p" Jul 22 21:05:47.458196 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.457754 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4fe33109-ef55-4c16-935f-cc6280e9d58e-tmp\") pod \"tuned-dxs8z\" (UID: \"4fe33109-ef55-4c16-935f-cc6280e9d58e\") " pod="openshift-cluster-node-tuning-operator/tuned-dxs8z" Jul 22 21:05:47.459160 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.459030 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65c4s\" (UniqueName: \"kubernetes.io/projected/cf2ac247-7e8e-46b0-991e-e82a384991c1-kube-api-access-65c4s\") pod \"multus-additional-cni-plugins-42g4d\" (UID: \"cf2ac247-7e8e-46b0-991e-e82a384991c1\") " pod="openshift-multus/multus-additional-cni-plugins-42g4d" Jul 22 21:05:47.460847 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.460746 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hwlt\" (UniqueName: \"kubernetes.io/projected/40f77f02-2bc6-4339-a948-7631a9603ce1-kube-api-access-8hwlt\") pod \"node-resolver-z2g8w\" (UID: \"40f77f02-2bc6-4339-a948-7631a9603ce1\") " pod="openshift-dns/node-resolver-z2g8w" Jul 22 21:05:47.461281 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.461212 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6q5s\" (UniqueName: \"kubernetes.io/projected/4fe33109-ef55-4c16-935f-cc6280e9d58e-kube-api-access-z6q5s\") pod \"tuned-dxs8z\" (UID: \"4fe33109-ef55-4c16-935f-cc6280e9d58e\") " pod="openshift-cluster-node-tuning-operator/tuned-dxs8z" Jul 22 21:05:47.461669 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.461648 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wskpm\" (UniqueName: \"kubernetes.io/projected/45b1939d-7d91-4899-b489-7e2609a5878d-kube-api-access-wskpm\") pod \"node-ca-4s9cs\" (UID: \"45b1939d-7d91-4899-b489-7e2609a5878d\") " pod="openshift-image-registry/node-ca-4s9cs" Jul 22 21:05:47.461755 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.461688 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbcq7\" (UniqueName: \"kubernetes.io/projected/a449924b-6d2d-40f9-97a1-584a066510a3-kube-api-access-sbcq7\") pod \"aws-ebs-csi-driver-node-87wqc\" (UID: \"a449924b-6d2d-40f9-97a1-584a066510a3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-87wqc" Jul 22 21:05:47.462352 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.462333 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcr84\" (UniqueName: \"kubernetes.io/projected/c29841d1-b93c-4f1e-9455-477c2208081f-kube-api-access-tcr84\") pod \"ovnkube-node-7w87p\" (UID: \"c29841d1-b93c-4f1e-9455-477c2208081f\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w87p" Jul 22 21:05:47.535783 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.535718 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-m7pxh" Jul 22 21:05:47.547594 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.547568 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rdlwm" Jul 22 21:05:47.556590 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.556567 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-kpw4k" Jul 22 21:05:47.561195 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.561159 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7w87p" Jul 22 21:05:47.571156 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.571128 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-87wqc" Jul 22 21:05:47.575858 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.575836 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-z2g8w" Jul 22 21:05:47.582952 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.582929 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dxs8z" Jul 22 21:05:47.588890 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.588723 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4s9cs" Jul 22 21:05:47.593624 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.593604 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-42g4d" Jul 22 21:05:47.640535 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:47.640432 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45b1939d_7d91_4899_b489_7e2609a5878d.slice/crio-7fae6e86c586119e084839a2b5ef79d2ec846d53ba974ac5225c44d7f0ae98aa WatchSource:0}: Error finding container 7fae6e86c586119e084839a2b5ef79d2ec846d53ba974ac5225c44d7f0ae98aa: Status 404 returned error can't find the container with id 7fae6e86c586119e084839a2b5ef79d2ec846d53ba974ac5225c44d7f0ae98aa Jul 22 21:05:47.641834 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:47.641807 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fe33109_ef55_4c16_935f_cc6280e9d58e.slice/crio-516e7388226b984a37dce335ae045b83039c9e55fc93e78efef76780479fd9d3 WatchSource:0}: Error finding container 516e7388226b984a37dce335ae045b83039c9e55fc93e78efef76780479fd9d3: Status 404 returned error can't find the container with id 516e7388226b984a37dce335ae045b83039c9e55fc93e78efef76780479fd9d3 Jul 22 21:05:47.642706 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:47.642685 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d3ff930_d226_4a49_9cb5_aa7d02eb37af.slice/crio-f36a1a103d83f4026c302a088a3315217460e0937faf476fbb48d1cf2917b2f1 WatchSource:0}: Error finding container f36a1a103d83f4026c302a088a3315217460e0937faf476fbb48d1cf2917b2f1: Status 404 returned error can't find the container with id f36a1a103d83f4026c302a088a3315217460e0937faf476fbb48d1cf2917b2f1 Jul 22 21:05:47.645806 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:47.644665 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc29841d1_b93c_4f1e_9455_477c2208081f.slice/crio-e03d1f72011ab5eb69656f14cbe8bc4625dd2813dbd31ba6998783b87e58fbe6 WatchSource:0}: Error finding container e03d1f72011ab5eb69656f14cbe8bc4625dd2813dbd31ba6998783b87e58fbe6: Status 404 returned error can't find the container with id e03d1f72011ab5eb69656f14cbe8bc4625dd2813dbd31ba6998783b87e58fbe6 Jul 22 21:05:47.646643 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:47.646615 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf2ac247_7e8e_46b0_991e_e82a384991c1.slice/crio-b0729163ce7bab0d673772af713d16d68c235df66ae6ff62b61bbc908d3d4565 WatchSource:0}: Error finding container b0729163ce7bab0d673772af713d16d68c235df66ae6ff62b61bbc908d3d4565: Status 404 returned error can't find the container with id b0729163ce7bab0d673772af713d16d68c235df66ae6ff62b61bbc908d3d4565 Jul 22 21:05:47.673139 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:47.673119 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda449924b_6d2d_40f9_97a1_584a066510a3.slice/crio-bb281e8ee297bed58ccb641e4b59609de787314983d9403fa8a02d54f2249aa0 WatchSource:0}: Error finding container bb281e8ee297bed58ccb641e4b59609de787314983d9403fa8a02d54f2249aa0: Status 404 returned error can't find the container with id bb281e8ee297bed58ccb641e4b59609de787314983d9403fa8a02d54f2249aa0 Jul 22 21:05:47.674716 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:47.674697 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e6d7a69_6a07_40bf_9b57_c7a4828cb08a.slice/crio-6d2295b313f8f0d1d84b73f901ba6898b9582d101c0c31db50a9f6f2c16fddfc WatchSource:0}: Error finding container 6d2295b313f8f0d1d84b73f901ba6898b9582d101c0c31db50a9f6f2c16fddfc: Status 404 returned error can't find the container with id 6d2295b313f8f0d1d84b73f901ba6898b9582d101c0c31db50a9f6f2c16fddfc Jul 22 21:05:47.856194 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.856090 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a91c48d-374d-4fc3-9114-d5c88ea18f63-metrics-certs\") pod \"network-metrics-daemon-lj48g\" (UID: \"8a91c48d-374d-4fc3-9114-d5c88ea18f63\") " pod="openshift-multus/network-metrics-daemon-lj48g" Jul 22 21:05:47.856345 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:05:47.856270 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 21:05:47.856430 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:05:47.856347 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a91c48d-374d-4fc3-9114-d5c88ea18f63-metrics-certs podName:8a91c48d-374d-4fc3-9114-d5c88ea18f63 nodeName:}" failed. No retries permitted until 2026-07-22 21:05:48.856324813 +0000 UTC m=+3.970717142 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a91c48d-374d-4fc3-9114-d5c88ea18f63-metrics-certs") pod "network-metrics-daemon-lj48g" (UID: "8a91c48d-374d-4fc3-9114-d5c88ea18f63") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 21:05:47.957336 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.957271 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4f3c7912-ed80-4a24-ada3-b2a3033bb210-original-pull-secret\") pod \"global-pull-secret-syncer-zrg2r\" (UID: \"4f3c7912-ed80-4a24-ada3-b2a3033bb210\") " pod="kube-system/global-pull-secret-syncer-zrg2r" Jul 22 21:05:47.957429 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:47.957343 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqf67\" (UniqueName: \"kubernetes.io/projected/07cbf59d-dabc-4ba4-92b8-ec63306521cf-kube-api-access-hqf67\") pod \"network-check-target-tn98z\" (UID: \"07cbf59d-dabc-4ba4-92b8-ec63306521cf\") " pod="openshift-network-diagnostics/network-check-target-tn98z" Jul 22 21:05:47.957471 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:05:47.957428 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 21:05:47.957471 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:05:47.957439 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 21:05:47.957471 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:05:47.957455 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 21:05:47.957471 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:05:47.957468 2572 projected.go:196] Error preparing data for projected volume kube-api-access-hqf67 for pod openshift-network-diagnostics/network-check-target-tn98z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 21:05:47.957619 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:05:47.957492 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f3c7912-ed80-4a24-ada3-b2a3033bb210-original-pull-secret podName:4f3c7912-ed80-4a24-ada3-b2a3033bb210 nodeName:}" failed. No retries permitted until 2026-07-22 21:05:48.957471634 +0000 UTC m=+4.071863979 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4f3c7912-ed80-4a24-ada3-b2a3033bb210-original-pull-secret") pod "global-pull-secret-syncer-zrg2r" (UID: "4f3c7912-ed80-4a24-ada3-b2a3033bb210") : object "kube-system"/"original-pull-secret" not registered Jul 22 21:05:47.957619 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:05:47.957510 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07cbf59d-dabc-4ba4-92b8-ec63306521cf-kube-api-access-hqf67 podName:07cbf59d-dabc-4ba4-92b8-ec63306521cf nodeName:}" failed. No retries permitted until 2026-07-22 21:05:48.957499019 +0000 UTC m=+4.071891360 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-hqf67" (UniqueName: "kubernetes.io/projected/07cbf59d-dabc-4ba4-92b8-ec63306521cf-kube-api-access-hqf67") pod "network-check-target-tn98z" (UID: "07cbf59d-dabc-4ba4-92b8-ec63306521cf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 21:05:48.031997 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:05:48.031971 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod091f2ee13ccf617382e7e126cc5d9e12.slice/crio-81377f24f7fee2a8329e108c3e0c49748c106757409e90a241219d1887743454 WatchSource:0}: Error finding container 81377f24f7fee2a8329e108c3e0c49748c106757409e90a241219d1887743454: Status 404 returned error can't find the container with id 81377f24f7fee2a8329e108c3e0c49748c106757409e90a241219d1887743454 Jul 22 21:05:48.209131 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:48.208835 2572 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 21:05:48.277403 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:48.277334 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-21 21:00:47 +0000 UTC" deadline="2028-03-21 08:37:27.275978685 +0000 UTC" Jul 22 21:05:48.277403 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:48.277357 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14579h31m38.998625181s" Jul 22 21:05:48.378063 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:48.377304 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zrg2r" Jul 22 21:05:48.378063 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:05:48.377680 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zrg2r" podUID="4f3c7912-ed80-4a24-ada3-b2a3033bb210" Jul 22 21:05:48.388771 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:48.388735 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4s9cs" event={"ID":"45b1939d-7d91-4899-b489-7e2609a5878d","Type":"ContainerStarted","Data":"7fae6e86c586119e084839a2b5ef79d2ec846d53ba974ac5225c44d7f0ae98aa"} Jul 22 21:05:48.395257 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:48.395217 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-37.ec2.internal" event={"ID":"091f2ee13ccf617382e7e126cc5d9e12","Type":"ContainerStarted","Data":"81377f24f7fee2a8329e108c3e0c49748c106757409e90a241219d1887743454"} Jul 22 21:05:48.408662 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:48.408631 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-z2g8w" event={"ID":"40f77f02-2bc6-4339-a948-7631a9603ce1","Type":"ContainerStarted","Data":"aa3c908481bde34bbdefcb96ef833e03ad3e308300d0d8a58aa5260e6fdb97e8"} Jul 22 21:05:48.418539 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:48.418500 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-42g4d" event={"ID":"cf2ac247-7e8e-46b0-991e-e82a384991c1","Type":"ContainerStarted","Data":"b0729163ce7bab0d673772af713d16d68c235df66ae6ff62b61bbc908d3d4565"} Jul 22 21:05:48.436082 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:48.431966 2572 generic.go:356] "Generic (PLEG): container finished" podID="e462d762995f2dc89c7e851abf11f67f" containerID="f3385a6efb9481d99931c1e098253c0c4d7117cf61e0cb89433fa173bd1f9696" exitCode=0 Jul 22 21:05:48.436082 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:48.432057 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-37.ec2.internal" event={"ID":"e462d762995f2dc89c7e851abf11f67f","Type":"ContainerDied","Data":"f3385a6efb9481d99931c1e098253c0c4d7117cf61e0cb89433fa173bd1f9696"} Jul 22 21:05:48.438864 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:48.438815 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rdlwm" event={"ID":"3e6d7a69-6a07-40bf-9b57-c7a4828cb08a","Type":"ContainerStarted","Data":"6d2295b313f8f0d1d84b73f901ba6898b9582d101c0c31db50a9f6f2c16fddfc"} Jul 22 21:05:48.441359 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:48.441333 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-87wqc" event={"ID":"a449924b-6d2d-40f9-97a1-584a066510a3","Type":"ContainerStarted","Data":"bb281e8ee297bed58ccb641e4b59609de787314983d9403fa8a02d54f2249aa0"} Jul 22 21:05:48.445926 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:48.445768 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-m7pxh" event={"ID":"37b20c60-c50a-42df-acd7-2164653d94ea","Type":"ContainerStarted","Data":"35da1d59488e8ec88d845786330187ff72d967f93f852b9cb6b117a5bdffa382"} Jul 22 21:05:48.448169 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:48.448071 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7w87p" event={"ID":"c29841d1-b93c-4f1e-9455-477c2208081f","Type":"ContainerStarted","Data":"e03d1f72011ab5eb69656f14cbe8bc4625dd2813dbd31ba6998783b87e58fbe6"} Jul 22 21:05:48.450822 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:48.450798 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-kpw4k" event={"ID":"2d3ff930-d226-4a49-9cb5-aa7d02eb37af","Type":"ContainerStarted","Data":"f36a1a103d83f4026c302a088a3315217460e0937faf476fbb48d1cf2917b2f1"} Jul 22 21:05:48.454411 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:48.454385 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dxs8z" event={"ID":"4fe33109-ef55-4c16-935f-cc6280e9d58e","Type":"ContainerStarted","Data":"516e7388226b984a37dce335ae045b83039c9e55fc93e78efef76780479fd9d3"} Jul 22 21:05:48.864544 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:48.864419 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a91c48d-374d-4fc3-9114-d5c88ea18f63-metrics-certs\") pod \"network-metrics-daemon-lj48g\" (UID: \"8a91c48d-374d-4fc3-9114-d5c88ea18f63\") " pod="openshift-multus/network-metrics-daemon-lj48g" Jul 22 21:05:48.864713 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:05:48.864583 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 21:05:48.864713 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:05:48.864655 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a91c48d-374d-4fc3-9114-d5c88ea18f63-metrics-certs podName:8a91c48d-374d-4fc3-9114-d5c88ea18f63 nodeName:}" failed. No retries permitted until 2026-07-22 21:05:50.864635279 +0000 UTC m=+5.979027620 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a91c48d-374d-4fc3-9114-d5c88ea18f63-metrics-certs") pod "network-metrics-daemon-lj48g" (UID: "8a91c48d-374d-4fc3-9114-d5c88ea18f63") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 21:05:48.966569 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:48.965685 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4f3c7912-ed80-4a24-ada3-b2a3033bb210-original-pull-secret\") pod \"global-pull-secret-syncer-zrg2r\" (UID: \"4f3c7912-ed80-4a24-ada3-b2a3033bb210\") " pod="kube-system/global-pull-secret-syncer-zrg2r" Jul 22 21:05:48.966569 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:48.965799 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqf67\" (UniqueName: \"kubernetes.io/projected/07cbf59d-dabc-4ba4-92b8-ec63306521cf-kube-api-access-hqf67\") pod \"network-check-target-tn98z\" (UID: \"07cbf59d-dabc-4ba4-92b8-ec63306521cf\") " pod="openshift-network-diagnostics/network-check-target-tn98z" Jul 22 21:05:48.966569 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:05:48.965946 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 21:05:48.966569 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:05:48.965965 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 21:05:48.966569 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:05:48.965979 2572 projected.go:196] Error preparing data for projected volume kube-api-access-hqf67 for pod openshift-network-diagnostics/network-check-target-tn98z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 21:05:48.966569 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:05:48.966036 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07cbf59d-dabc-4ba4-92b8-ec63306521cf-kube-api-access-hqf67 podName:07cbf59d-dabc-4ba4-92b8-ec63306521cf nodeName:}" failed. No retries permitted until 2026-07-22 21:05:50.966017858 +0000 UTC m=+6.080410199 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-hqf67" (UniqueName: "kubernetes.io/projected/07cbf59d-dabc-4ba4-92b8-ec63306521cf-kube-api-access-hqf67") pod "network-check-target-tn98z" (UID: "07cbf59d-dabc-4ba4-92b8-ec63306521cf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 21:05:48.966569 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:05:48.966458 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 21:05:48.966569 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:05:48.966527 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f3c7912-ed80-4a24-ada3-b2a3033bb210-original-pull-secret podName:4f3c7912-ed80-4a24-ada3-b2a3033bb210 nodeName:}" failed. No retries permitted until 2026-07-22 21:05:50.966501018 +0000 UTC m=+6.080893357 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4f3c7912-ed80-4a24-ada3-b2a3033bb210-original-pull-secret") pod "global-pull-secret-syncer-zrg2r" (UID: "4f3c7912-ed80-4a24-ada3-b2a3033bb210") : object "kube-system"/"original-pull-secret" not registered Jul 22 21:05:49.278418 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:49.278340 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-21 21:00:47 +0000 UTC" deadline="2028-01-19 20:46:57.526780387 +0000 UTC" Jul 22 21:05:49.278418 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:49.278383 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13103h41m8.248402631s" Jul 22 21:05:49.380589 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:49.379870 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lj48g" Jul 22 21:05:49.380589 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:05:49.380034 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lj48g" podUID="8a91c48d-374d-4fc3-9114-d5c88ea18f63" Jul 22 21:05:49.380589 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:49.380120 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tn98z" Jul 22 21:05:49.380589 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:05:49.380196 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tn98z" podUID="07cbf59d-dabc-4ba4-92b8-ec63306521cf" Jul 22 21:05:49.466594 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:49.466477 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-37.ec2.internal" event={"ID":"e462d762995f2dc89c7e851abf11f67f","Type":"ContainerStarted","Data":"7e66709d77d5ee0f0d4b095052b1596d87ca8ddb87c027c8418faed44d5e1085"} Jul 22 21:05:50.377544 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:50.376593 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zrg2r" Jul 22 21:05:50.377544 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:05:50.376736 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zrg2r" podUID="4f3c7912-ed80-4a24-ada3-b2a3033bb210" Jul 22 21:05:50.880683 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:50.880647 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a91c48d-374d-4fc3-9114-d5c88ea18f63-metrics-certs\") pod \"network-metrics-daemon-lj48g\" (UID: \"8a91c48d-374d-4fc3-9114-d5c88ea18f63\") " pod="openshift-multus/network-metrics-daemon-lj48g" Jul 22 21:05:50.880920 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:05:50.880900 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 21:05:50.881057 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:05:50.880973 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a91c48d-374d-4fc3-9114-d5c88ea18f63-metrics-certs podName:8a91c48d-374d-4fc3-9114-d5c88ea18f63 nodeName:}" failed. No retries permitted until 2026-07-22 21:05:54.880951908 +0000 UTC m=+9.995344251 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a91c48d-374d-4fc3-9114-d5c88ea18f63-metrics-certs") pod "network-metrics-daemon-lj48g" (UID: "8a91c48d-374d-4fc3-9114-d5c88ea18f63") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 21:05:50.981812 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:50.981776 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqf67\" (UniqueName: \"kubernetes.io/projected/07cbf59d-dabc-4ba4-92b8-ec63306521cf-kube-api-access-hqf67\") pod \"network-check-target-tn98z\" (UID: \"07cbf59d-dabc-4ba4-92b8-ec63306521cf\") " pod="openshift-network-diagnostics/network-check-target-tn98z" Jul 22 21:05:50.982015 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:50.981827 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4f3c7912-ed80-4a24-ada3-b2a3033bb210-original-pull-secret\") pod \"global-pull-secret-syncer-zrg2r\" (UID: \"4f3c7912-ed80-4a24-ada3-b2a3033bb210\") " pod="kube-system/global-pull-secret-syncer-zrg2r" Jul 22 21:05:50.982015 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:05:50.981950 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 21:05:50.982015 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:05:50.982002 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f3c7912-ed80-4a24-ada3-b2a3033bb210-original-pull-secret podName:4f3c7912-ed80-4a24-ada3-b2a3033bb210 nodeName:}" failed. No retries permitted until 2026-07-22 21:05:54.981989971 +0000 UTC m=+10.096382299 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4f3c7912-ed80-4a24-ada3-b2a3033bb210-original-pull-secret") pod "global-pull-secret-syncer-zrg2r" (UID: "4f3c7912-ed80-4a24-ada3-b2a3033bb210") : object "kube-system"/"original-pull-secret" not registered Jul 22 21:05:50.982478 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:05:50.982454 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 21:05:50.982478 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:05:50.982482 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 21:05:50.982478 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:05:50.982495 2572 projected.go:196] Error preparing data for projected volume kube-api-access-hqf67 for pod openshift-network-diagnostics/network-check-target-tn98z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 21:05:50.982740 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:05:50.982543 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07cbf59d-dabc-4ba4-92b8-ec63306521cf-kube-api-access-hqf67 podName:07cbf59d-dabc-4ba4-92b8-ec63306521cf nodeName:}" failed. No retries permitted until 2026-07-22 21:05:54.982527873 +0000 UTC m=+10.096920200 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-hqf67" (UniqueName: "kubernetes.io/projected/07cbf59d-dabc-4ba4-92b8-ec63306521cf-kube-api-access-hqf67") pod "network-check-target-tn98z" (UID: "07cbf59d-dabc-4ba4-92b8-ec63306521cf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 21:05:51.378437 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:51.378309 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lj48g" Jul 22 21:05:51.378437 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:51.378401 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tn98z" Jul 22 21:05:51.379122 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:05:51.378494 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tn98z" podUID="07cbf59d-dabc-4ba4-92b8-ec63306521cf" Jul 22 21:05:51.379122 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:05:51.378841 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lj48g" podUID="8a91c48d-374d-4fc3-9114-d5c88ea18f63" Jul 22 21:05:52.377573 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:52.377542 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zrg2r" Jul 22 21:05:52.377765 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:05:52.377677 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zrg2r" podUID="4f3c7912-ed80-4a24-ada3-b2a3033bb210" Jul 22 21:05:53.377347 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:53.377313 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lj48g" Jul 22 21:05:53.377837 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:05:53.377458 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lj48g" podUID="8a91c48d-374d-4fc3-9114-d5c88ea18f63" Jul 22 21:05:53.377904 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:53.377893 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tn98z" Jul 22 21:05:53.378012 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:05:53.377990 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tn98z" podUID="07cbf59d-dabc-4ba4-92b8-ec63306521cf" Jul 22 21:05:54.377467 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:54.377390 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zrg2r" Jul 22 21:05:54.377934 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:05:54.377523 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zrg2r" podUID="4f3c7912-ed80-4a24-ada3-b2a3033bb210" Jul 22 21:05:54.915740 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:54.915696 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a91c48d-374d-4fc3-9114-d5c88ea18f63-metrics-certs\") pod \"network-metrics-daemon-lj48g\" (UID: \"8a91c48d-374d-4fc3-9114-d5c88ea18f63\") " pod="openshift-multus/network-metrics-daemon-lj48g" Jul 22 21:05:54.916016 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:05:54.915872 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 21:05:54.916016 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:05:54.915956 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a91c48d-374d-4fc3-9114-d5c88ea18f63-metrics-certs podName:8a91c48d-374d-4fc3-9114-d5c88ea18f63 nodeName:}" failed. No retries permitted until 2026-07-22 21:06:02.915934365 +0000 UTC m=+18.030326691 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a91c48d-374d-4fc3-9114-d5c88ea18f63-metrics-certs") pod "network-metrics-daemon-lj48g" (UID: "8a91c48d-374d-4fc3-9114-d5c88ea18f63") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 21:05:55.016360 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:55.016321 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqf67\" (UniqueName: \"kubernetes.io/projected/07cbf59d-dabc-4ba4-92b8-ec63306521cf-kube-api-access-hqf67\") pod \"network-check-target-tn98z\" (UID: \"07cbf59d-dabc-4ba4-92b8-ec63306521cf\") " pod="openshift-network-diagnostics/network-check-target-tn98z" Jul 22 21:05:55.016517 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:55.016379 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4f3c7912-ed80-4a24-ada3-b2a3033bb210-original-pull-secret\") pod \"global-pull-secret-syncer-zrg2r\" (UID: \"4f3c7912-ed80-4a24-ada3-b2a3033bb210\") " pod="kube-system/global-pull-secret-syncer-zrg2r" Jul 22 21:05:55.016591 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:05:55.016548 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 21:05:55.016643 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:05:55.016610 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f3c7912-ed80-4a24-ada3-b2a3033bb210-original-pull-secret podName:4f3c7912-ed80-4a24-ada3-b2a3033bb210 nodeName:}" failed. No retries permitted until 2026-07-22 21:06:03.016592106 +0000 UTC m=+18.130984432 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4f3c7912-ed80-4a24-ada3-b2a3033bb210-original-pull-secret") pod "global-pull-secret-syncer-zrg2r" (UID: "4f3c7912-ed80-4a24-ada3-b2a3033bb210") : object "kube-system"/"original-pull-secret" not registered Jul 22 21:05:55.017028 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:05:55.016991 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 21:05:55.017028 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:05:55.017020 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 21:05:55.017200 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:05:55.017033 2572 projected.go:196] Error preparing data for projected volume kube-api-access-hqf67 for pod openshift-network-diagnostics/network-check-target-tn98z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 21:05:55.017200 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:05:55.017084 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07cbf59d-dabc-4ba4-92b8-ec63306521cf-kube-api-access-hqf67 podName:07cbf59d-dabc-4ba4-92b8-ec63306521cf nodeName:}" failed. No retries permitted until 2026-07-22 21:06:03.017069737 +0000 UTC m=+18.131462065 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-hqf67" (UniqueName: "kubernetes.io/projected/07cbf59d-dabc-4ba4-92b8-ec63306521cf-kube-api-access-hqf67") pod "network-check-target-tn98z" (UID: "07cbf59d-dabc-4ba4-92b8-ec63306521cf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 21:05:55.377770 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:55.377666 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lj48g" Jul 22 21:05:55.378207 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:05:55.377788 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lj48g" podUID="8a91c48d-374d-4fc3-9114-d5c88ea18f63" Jul 22 21:05:55.378207 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:55.377852 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tn98z" Jul 22 21:05:55.378207 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:05:55.377985 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tn98z" podUID="07cbf59d-dabc-4ba4-92b8-ec63306521cf" Jul 22 21:05:56.377221 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:56.377190 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zrg2r" Jul 22 21:05:56.377415 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:05:56.377288 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zrg2r" podUID="4f3c7912-ed80-4a24-ada3-b2a3033bb210" Jul 22 21:05:57.378286 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:57.378254 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lj48g" Jul 22 21:05:57.378752 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:57.378296 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tn98z" Jul 22 21:05:57.378752 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:05:57.378392 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lj48g" podUID="8a91c48d-374d-4fc3-9114-d5c88ea18f63" Jul 22 21:05:57.378752 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:05:57.378466 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tn98z" podUID="07cbf59d-dabc-4ba4-92b8-ec63306521cf" Jul 22 21:05:58.377556 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:58.377524 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zrg2r" Jul 22 21:05:58.377711 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:05:58.377626 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zrg2r" podUID="4f3c7912-ed80-4a24-ada3-b2a3033bb210" Jul 22 21:05:59.376723 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:59.376690 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tn98z" Jul 22 21:05:59.377193 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:05:59.376816 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tn98z" podUID="07cbf59d-dabc-4ba4-92b8-ec63306521cf" Jul 22 21:05:59.377193 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:05:59.376862 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lj48g" Jul 22 21:05:59.377193 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:05:59.376967 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lj48g" podUID="8a91c48d-374d-4fc3-9114-d5c88ea18f63" Jul 22 21:06:00.377141 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:00.377110 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zrg2r" Jul 22 21:06:00.377630 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:06:00.377254 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zrg2r" podUID="4f3c7912-ed80-4a24-ada3-b2a3033bb210" Jul 22 21:06:01.377179 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:01.377132 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lj48g" Jul 22 21:06:01.377615 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:01.377132 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tn98z" Jul 22 21:06:01.377615 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:06:01.377327 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lj48g" podUID="8a91c48d-374d-4fc3-9114-d5c88ea18f63" Jul 22 21:06:01.377615 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:06:01.377362 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tn98z" podUID="07cbf59d-dabc-4ba4-92b8-ec63306521cf" Jul 22 21:06:02.379483 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:02.377220 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zrg2r" Jul 22 21:06:02.379483 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:06:02.377679 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zrg2r" podUID="4f3c7912-ed80-4a24-ada3-b2a3033bb210" Jul 22 21:06:02.981112 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:02.981079 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a91c48d-374d-4fc3-9114-d5c88ea18f63-metrics-certs\") pod \"network-metrics-daemon-lj48g\" (UID: \"8a91c48d-374d-4fc3-9114-d5c88ea18f63\") " pod="openshift-multus/network-metrics-daemon-lj48g" Jul 22 21:06:02.981284 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:06:02.981231 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 21:06:02.981338 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:06:02.981299 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a91c48d-374d-4fc3-9114-d5c88ea18f63-metrics-certs podName:8a91c48d-374d-4fc3-9114-d5c88ea18f63 nodeName:}" failed. No retries permitted until 2026-07-22 21:06:18.981282502 +0000 UTC m=+34.095674849 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a91c48d-374d-4fc3-9114-d5c88ea18f63-metrics-certs") pod "network-metrics-daemon-lj48g" (UID: "8a91c48d-374d-4fc3-9114-d5c88ea18f63") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 21:06:03.082353 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:03.082317 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4f3c7912-ed80-4a24-ada3-b2a3033bb210-original-pull-secret\") pod \"global-pull-secret-syncer-zrg2r\" (UID: \"4f3c7912-ed80-4a24-ada3-b2a3033bb210\") " pod="kube-system/global-pull-secret-syncer-zrg2r" Jul 22 21:06:03.082522 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:03.082399 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqf67\" (UniqueName: \"kubernetes.io/projected/07cbf59d-dabc-4ba4-92b8-ec63306521cf-kube-api-access-hqf67\") pod \"network-check-target-tn98z\" (UID: \"07cbf59d-dabc-4ba4-92b8-ec63306521cf\") " pod="openshift-network-diagnostics/network-check-target-tn98z" Jul 22 21:06:03.082522 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:06:03.082481 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 21:06:03.082522 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:06:03.082507 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 21:06:03.082522 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:06:03.082525 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 21:06:03.082679 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:06:03.082536 2572 projected.go:196] Error preparing data for projected volume kube-api-access-hqf67 for pod openshift-network-diagnostics/network-check-target-tn98z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 21:06:03.082679 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:06:03.082554 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f3c7912-ed80-4a24-ada3-b2a3033bb210-original-pull-secret podName:4f3c7912-ed80-4a24-ada3-b2a3033bb210 nodeName:}" failed. No retries permitted until 2026-07-22 21:06:19.082535081 +0000 UTC m=+34.196927421 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4f3c7912-ed80-4a24-ada3-b2a3033bb210-original-pull-secret") pod "global-pull-secret-syncer-zrg2r" (UID: "4f3c7912-ed80-4a24-ada3-b2a3033bb210") : object "kube-system"/"original-pull-secret" not registered Jul 22 21:06:03.082679 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:06:03.082572 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07cbf59d-dabc-4ba4-92b8-ec63306521cf-kube-api-access-hqf67 podName:07cbf59d-dabc-4ba4-92b8-ec63306521cf nodeName:}" failed. No retries permitted until 2026-07-22 21:06:19.082561776 +0000 UTC m=+34.196954098 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-hqf67" (UniqueName: "kubernetes.io/projected/07cbf59d-dabc-4ba4-92b8-ec63306521cf-kube-api-access-hqf67") pod "network-check-target-tn98z" (UID: "07cbf59d-dabc-4ba4-92b8-ec63306521cf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 21:06:03.377248 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:03.377154 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tn98z" Jul 22 21:06:03.377248 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:03.377210 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lj48g" Jul 22 21:06:03.377440 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:06:03.377299 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tn98z" podUID="07cbf59d-dabc-4ba4-92b8-ec63306521cf" Jul 22 21:06:03.377440 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:06:03.377424 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lj48g" podUID="8a91c48d-374d-4fc3-9114-d5c88ea18f63" Jul 22 21:06:04.376947 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:04.376904 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zrg2r" Jul 22 21:06:04.377393 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:06:04.377012 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zrg2r" podUID="4f3c7912-ed80-4a24-ada3-b2a3033bb210" Jul 22 21:06:05.377246 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:05.377211 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lj48g" Jul 22 21:06:05.377684 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:06:05.377325 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lj48g" podUID="8a91c48d-374d-4fc3-9114-d5c88ea18f63" Jul 22 21:06:05.377684 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:05.377391 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tn98z" Jul 22 21:06:05.377684 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:06:05.377502 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tn98z" podUID="07cbf59d-dabc-4ba4-92b8-ec63306521cf" Jul 22 21:06:06.376871 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:06.376839 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zrg2r" Jul 22 21:06:06.377067 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:06:06.376967 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zrg2r" podUID="4f3c7912-ed80-4a24-ada3-b2a3033bb210" Jul 22 21:06:07.377520 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:07.377497 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lj48g" Jul 22 21:06:07.378018 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:07.377536 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tn98z" Jul 22 21:06:07.378018 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:06:07.377631 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lj48g" podUID="8a91c48d-374d-4fc3-9114-d5c88ea18f63" Jul 22 21:06:07.378018 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:06:07.377707 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tn98z" podUID="07cbf59d-dabc-4ba4-92b8-ec63306521cf" Jul 22 21:06:08.377000 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:08.376840 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zrg2r" Jul 22 21:06:08.377154 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:06:08.377081 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zrg2r" podUID="4f3c7912-ed80-4a24-ada3-b2a3033bb210" Jul 22 21:06:08.509184 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:08.509087 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-87wqc" event={"ID":"a449924b-6d2d-40f9-97a1-584a066510a3","Type":"ContainerStarted","Data":"e9fa46f241c4b2a8e6ff0f3b7bf57cb328d3e1cf6f964d070fe3190c5f79ad33"} Jul 22 21:06:08.512303 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:08.512278 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-m7pxh" event={"ID":"37b20c60-c50a-42df-acd7-2164653d94ea","Type":"ContainerStarted","Data":"7d286c316344d8cdaddf78fb2b15ced19fcb3b878da9da7b201e8588049921c4"} Jul 22 21:06:08.514416 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:08.514395 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7w87p" event={"ID":"c29841d1-b93c-4f1e-9455-477c2208081f","Type":"ContainerStarted","Data":"4f3dfdc5606aa9517d2793051e4e232ad301c671081ce806d00d2db0a528afd5"} Jul 22 21:06:08.514521 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:08.514451 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7w87p" event={"ID":"c29841d1-b93c-4f1e-9455-477c2208081f","Type":"ContainerStarted","Data":"1ea5994f59d90bf00cd7befecbbf32773a0aca28d0433303047726949437bbfb"} Jul 22 21:06:08.514521 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:08.514496 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7w87p" event={"ID":"c29841d1-b93c-4f1e-9455-477c2208081f","Type":"ContainerStarted","Data":"637ef1d4768fb7c0df52f005daa08fdf17f1d976167cff8d67f93b3e257ff27d"} Jul 22 21:06:08.514613 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:08.514527 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7w87p" event={"ID":"c29841d1-b93c-4f1e-9455-477c2208081f","Type":"ContainerStarted","Data":"9f0a3a8652df8391c173d3c395fba458e29e453eee1c23d2e7559cfd6f2762e7"} Jul 22 21:06:08.515721 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:08.515697 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-kpw4k" event={"ID":"2d3ff930-d226-4a49-9cb5-aa7d02eb37af","Type":"ContainerStarted","Data":"a29d54e9690002834e15024e45f1db9baf7bb1ddf182dd0c634cf9a1ed56ac6b"} Jul 22 21:06:08.516881 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:08.516861 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dxs8z" event={"ID":"4fe33109-ef55-4c16-935f-cc6280e9d58e","Type":"ContainerStarted","Data":"d40589e59f32011256567ece875c200288996ef69f9bf18ae4e578f7c2185ed8"} Jul 22 21:06:08.518250 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:08.518230 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4s9cs" event={"ID":"45b1939d-7d91-4899-b489-7e2609a5878d","Type":"ContainerStarted","Data":"23553a4ad49add39e2e0ec9b7a5969d3b9747e6005b6c121a3240bd25fc71466"} Jul 22 21:06:08.519384 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:08.519365 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-37.ec2.internal" event={"ID":"091f2ee13ccf617382e7e126cc5d9e12","Type":"ContainerStarted","Data":"b436ce5bf2584e8b6ae7fa3842c18cf2643348c75ef8dd4b5bc4f8a779991bdb"} Jul 22 21:06:08.520598 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:08.520570 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-z2g8w" event={"ID":"40f77f02-2bc6-4339-a948-7631a9603ce1","Type":"ContainerStarted","Data":"ae702e34a2c8e3b827cfe7937881351035cfd474d291ec8caa24115a4fb65503"} Jul 22 21:06:08.521903 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:08.521880 2572 generic.go:356] "Generic (PLEG): container finished" podID="cf2ac247-7e8e-46b0-991e-e82a384991c1" containerID="dfb2bac048a5b1f01871babd2ebbfcbc24e0ef32cb429bccb263b2c0784d1c64" exitCode=0 Jul 22 21:06:08.521974 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:08.521912 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-42g4d" event={"ID":"cf2ac247-7e8e-46b0-991e-e82a384991c1","Type":"ContainerDied","Data":"dfb2bac048a5b1f01871babd2ebbfcbc24e0ef32cb429bccb263b2c0784d1c64"} Jul 22 21:06:08.542135 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:08.542100 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-37.ec2.internal" podStartSLOduration=21.542089026 podStartE2EDuration="21.542089026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 21:05:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 21:05:49.486321754 +0000 UTC m=+4.600714102" watchObservedRunningTime="2026-07-22 21:06:08.542089026 +0000 UTC m=+23.656481369" Jul 22 21:06:08.554752 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:08.554723 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-z2g8w" podStartSLOduration=3.185634875 podStartE2EDuration="23.554711795s" totalImagesPullingTime="20.36907692s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 21:05:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 21:06:08.554572037 +0000 UTC m=+23.668964381" watchObservedRunningTime="2026-07-22 21:06:08.554711795 +0000 UTC m=+23.669104139" Jul 22 21:06:08.582686 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:08.582649 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-37.ec2.internal" podStartSLOduration=21.582638044 podStartE2EDuration="21.582638044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 21:05:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 21:06:08.58245744 +0000 UTC m=+23.696849786" watchObservedRunningTime="2026-07-22 21:06:08.582638044 +0000 UTC m=+23.697030387" Jul 22 21:06:08.582780 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:08.582755 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-dxs8z" podStartSLOduration=3.190817764 podStartE2EDuration="23.582748922s" totalImagesPullingTime="20.391931158s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 21:05:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 21:06:08.570379226 +0000 UTC m=+23.684771570" watchObservedRunningTime="2026-07-22 21:06:08.582748922 +0000 UTC m=+23.697141266" Jul 22 21:06:08.598283 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:08.598248 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-kpw4k" podStartSLOduration=3.206929047 podStartE2EDuration="23.598236205s" totalImagesPullingTime="20.391307158s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 21:05:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 21:06:08.597942353 +0000 UTC m=+23.712334722" watchObservedRunningTime="2026-07-22 21:06:08.598236205 +0000 UTC m=+23.712628545" Jul 22 21:06:08.628183 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:08.628121 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-m7pxh" podStartSLOduration=4.442707394 podStartE2EDuration="23.628108346s" totalImagesPullingTime="19.185400952s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 21:05:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 21:06:08.627908644 +0000 UTC m=+23.742300989" watchObservedRunningTime="2026-07-22 21:06:08.628108346 +0000 UTC m=+23.742500689" Jul 22 21:06:08.628348 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:08.628327 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-4s9cs" podStartSLOduration=3.225088059 podStartE2EDuration="23.628318259s" totalImagesPullingTime="20.4032302s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 21:05:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 21:06:08.613397971 +0000 UTC m=+23.727790315" watchObservedRunningTime="2026-07-22 21:06:08.628318259 +0000 UTC m=+23.742710603" Jul 22 21:06:08.987345 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:08.987316 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-m7pxh" Jul 22 21:06:08.987909 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:08.987889 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-m7pxh" Jul 22 21:06:09.377526 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:09.377495 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tn98z" Jul 22 21:06:09.377732 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:06:09.377603 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tn98z" podUID="07cbf59d-dabc-4ba4-92b8-ec63306521cf" Jul 22 21:06:09.377898 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:09.377494 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lj48g" Jul 22 21:06:09.378021 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:06:09.377982 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lj48g" podUID="8a91c48d-374d-4fc3-9114-d5c88ea18f63" Jul 22 21:06:09.527857 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:09.527822 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7w87p" event={"ID":"c29841d1-b93c-4f1e-9455-477c2208081f","Type":"ContainerStarted","Data":"c9b4800cafc7fdcfc7c958b49352f046436b7857ebde3b93ec4bf835a55b764a"} Jul 22 21:06:09.528315 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:09.527916 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7w87p" event={"ID":"c29841d1-b93c-4f1e-9455-477c2208081f","Type":"ContainerStarted","Data":"6e9ff59c25eb6bd89e423c22a21bf477585d4ec4b6aed1e761a40b6142f567a7"} Jul 22 21:06:09.530181 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:09.530133 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rdlwm" event={"ID":"3e6d7a69-6a07-40bf-9b57-c7a4828cb08a","Type":"ContainerStarted","Data":"405e24c92fe7edb2c75163ff3fa98053fcf1fa9235a9b7eaf2ae059b706b3769"} Jul 22 21:06:09.530690 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:09.530388 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-m7pxh" Jul 22 21:06:09.530990 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:09.530970 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-m7pxh" Jul 22 21:06:09.543924 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:09.543852 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-rdlwm" podStartSLOduration=4.183881723 podStartE2EDuration="24.543838181s" totalImagesPullingTime="20.359956458s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 21:05:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 21:06:09.543079315 +0000 UTC m=+24.657471680" watchObservedRunningTime="2026-07-22 21:06:09.543838181 +0000 UTC m=+24.658230524" Jul 22 21:06:09.895997 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:09.895974 2572 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 22 21:06:10.304736 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:10.304604 2572 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-22T21:06:09.895992622Z","UUID":"aa70f2f6-357b-4ce4-85d9-f42867124c93","Handler":null,"Name":"","Endpoint":""} Jul 22 21:06:10.306397 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:10.306374 2572 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 22 21:06:10.306531 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:10.306421 2572 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 22 21:06:10.377124 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:10.377093 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zrg2r" Jul 22 21:06:10.377282 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:06:10.377262 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zrg2r" podUID="4f3c7912-ed80-4a24-ada3-b2a3033bb210" Jul 22 21:06:10.533794 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:10.533750 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-87wqc" event={"ID":"a449924b-6d2d-40f9-97a1-584a066510a3","Type":"ContainerStarted","Data":"1e3bee81630de5a58b2fa6630576d6a6619f67ff8a65a7143a327bc8ec7acbe2"} Jul 22 21:06:11.377674 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:11.377641 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tn98z" Jul 22 21:06:11.377873 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:11.377650 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lj48g" Jul 22 21:06:11.377873 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:06:11.377775 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tn98z" podUID="07cbf59d-dabc-4ba4-92b8-ec63306521cf" Jul 22 21:06:11.377998 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:06:11.377888 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lj48g" podUID="8a91c48d-374d-4fc3-9114-d5c88ea18f63" Jul 22 21:06:11.540657 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:11.540616 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7w87p" event={"ID":"c29841d1-b93c-4f1e-9455-477c2208081f","Type":"ContainerStarted","Data":"d0ab977eeda3e52095589ebfab0f3ed4e8a8d54fa3dd2e980356be2bf3cbd32c"} Jul 22 21:06:12.377008 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:12.376775 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zrg2r" Jul 22 21:06:12.377231 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:06:12.377127 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zrg2r" podUID="4f3c7912-ed80-4a24-ada3-b2a3033bb210" Jul 22 21:06:13.376891 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:13.376848 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tn98z" Jul 22 21:06:13.377671 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:13.376860 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lj48g" Jul 22 21:06:13.377671 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:06:13.376997 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tn98z" podUID="07cbf59d-dabc-4ba4-92b8-ec63306521cf" Jul 22 21:06:13.377671 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:06:13.377144 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lj48g" podUID="8a91c48d-374d-4fc3-9114-d5c88ea18f63" Jul 22 21:06:13.547536 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:13.547453 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7w87p" event={"ID":"c29841d1-b93c-4f1e-9455-477c2208081f","Type":"ContainerStarted","Data":"74515f51b17ac33f8807f71861ee1a2c7b3f73898df4636ddf1440012a6dfb84"} Jul 22 21:06:13.548201 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:13.548154 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7w87p" Jul 22 21:06:13.548322 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:13.548249 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7w87p" Jul 22 21:06:13.549411 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:13.549390 2572 generic.go:356] "Generic (PLEG): container finished" podID="cf2ac247-7e8e-46b0-991e-e82a384991c1" containerID="e41b90e1619b5970e65307bbd52899b07d68174019136a0f498ad6c7f790960e" exitCode=0 Jul 22 21:06:13.549508 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:13.549418 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-42g4d" event={"ID":"cf2ac247-7e8e-46b0-991e-e82a384991c1","Type":"ContainerDied","Data":"e41b90e1619b5970e65307bbd52899b07d68174019136a0f498ad6c7f790960e"} Jul 22 21:06:13.564125 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:13.564104 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7w87p" Jul 22 21:06:13.581480 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:13.581443 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-7w87p" podStartSLOduration=8.191855308 podStartE2EDuration="28.581432245s" totalImagesPullingTime="20.389576937s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 21:05:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 21:06:13.581348764 +0000 UTC m=+28.695741107" watchObservedRunningTime="2026-07-22 21:06:13.581432245 +0000 UTC m=+28.695824588" Jul 22 21:06:14.377879 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:14.377590 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zrg2r" Jul 22 21:06:14.378350 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:06:14.377961 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zrg2r" podUID="4f3c7912-ed80-4a24-ada3-b2a3033bb210" Jul 22 21:06:14.555649 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:14.555618 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-87wqc" event={"ID":"a449924b-6d2d-40f9-97a1-584a066510a3","Type":"ContainerStarted","Data":"2327837d624fe21b154961fcc8c8d9b8ddc1466f5856d5721f7bc476e70b31b5"} Jul 22 21:06:14.556078 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:14.556044 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7w87p" Jul 22 21:06:14.571129 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:14.571106 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7w87p" Jul 22 21:06:14.578305 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:14.578263 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-87wqc" podStartSLOduration=4.020787322 podStartE2EDuration="29.578250262s" totalImagesPullingTime="25.55746294s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 21:05:45 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 21:06:14.577241936 +0000 UTC m=+29.691634297" watchObservedRunningTime="2026-07-22 21:06:14.578250262 +0000 UTC m=+29.692642603" Jul 22 21:06:15.092488 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:15.092401 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zrg2r"] Jul 22 21:06:15.092746 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:15.092553 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zrg2r" Jul 22 21:06:15.092746 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:06:15.092668 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zrg2r" podUID="4f3c7912-ed80-4a24-ada3-b2a3033bb210" Jul 22 21:06:15.093144 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:15.093117 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lj48g"] Jul 22 21:06:15.093348 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:15.093322 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lj48g" Jul 22 21:06:15.093461 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:06:15.093427 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lj48g" podUID="8a91c48d-374d-4fc3-9114-d5c88ea18f63" Jul 22 21:06:15.093666 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:15.093648 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tn98z"] Jul 22 21:06:15.093737 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:15.093727 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tn98z" Jul 22 21:06:15.093806 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:06:15.093791 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tn98z" podUID="07cbf59d-dabc-4ba4-92b8-ec63306521cf" Jul 22 21:06:15.559305 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:15.559218 2572 generic.go:356] "Generic (PLEG): container finished" podID="cf2ac247-7e8e-46b0-991e-e82a384991c1" containerID="62c6af478f08445f528630faac637db3b8d4fd711c29a978b415bf6cd310d0c9" exitCode=0 Jul 22 21:06:15.559793 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:15.559308 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-42g4d" event={"ID":"cf2ac247-7e8e-46b0-991e-e82a384991c1","Type":"ContainerDied","Data":"62c6af478f08445f528630faac637db3b8d4fd711c29a978b415bf6cd310d0c9"} Jul 22 21:06:17.377453 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:17.377376 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tn98z" Jul 22 21:06:17.377453 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:17.377379 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zrg2r" Jul 22 21:06:17.377804 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:06:17.377457 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tn98z" podUID="07cbf59d-dabc-4ba4-92b8-ec63306521cf" Jul 22 21:06:17.377804 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:06:17.377538 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zrg2r" podUID="4f3c7912-ed80-4a24-ada3-b2a3033bb210" Jul 22 21:06:17.377804 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:17.377590 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lj48g" Jul 22 21:06:17.377804 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:06:17.377657 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lj48g" podUID="8a91c48d-374d-4fc3-9114-d5c88ea18f63" Jul 22 21:06:17.565649 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:17.565614 2572 generic.go:356] "Generic (PLEG): container finished" podID="cf2ac247-7e8e-46b0-991e-e82a384991c1" containerID="df07c7171ca516643dc6c9d9cb3dcddfd26e281f9bb114f2f4d9481653bc9cf0" exitCode=0 Jul 22 21:06:17.565787 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:17.565672 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-42g4d" event={"ID":"cf2ac247-7e8e-46b0-991e-e82a384991c1","Type":"ContainerDied","Data":"df07c7171ca516643dc6c9d9cb3dcddfd26e281f9bb114f2f4d9481653bc9cf0"} Jul 22 21:06:19.003033 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:19.002982 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a91c48d-374d-4fc3-9114-d5c88ea18f63-metrics-certs\") pod \"network-metrics-daemon-lj48g\" (UID: \"8a91c48d-374d-4fc3-9114-d5c88ea18f63\") " pod="openshift-multus/network-metrics-daemon-lj48g" Jul 22 21:06:19.003458 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:06:19.003130 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 21:06:19.003458 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:06:19.003216 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a91c48d-374d-4fc3-9114-d5c88ea18f63-metrics-certs podName:8a91c48d-374d-4fc3-9114-d5c88ea18f63 nodeName:}" failed. No retries permitted until 2026-07-22 21:06:51.003199549 +0000 UTC m=+66.117591871 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a91c48d-374d-4fc3-9114-d5c88ea18f63-metrics-certs") pod "network-metrics-daemon-lj48g" (UID: "8a91c48d-374d-4fc3-9114-d5c88ea18f63") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 21:06:19.103567 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:19.103534 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqf67\" (UniqueName: \"kubernetes.io/projected/07cbf59d-dabc-4ba4-92b8-ec63306521cf-kube-api-access-hqf67\") pod \"network-check-target-tn98z\" (UID: \"07cbf59d-dabc-4ba4-92b8-ec63306521cf\") " pod="openshift-network-diagnostics/network-check-target-tn98z" Jul 22 21:06:19.103716 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:19.103580 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4f3c7912-ed80-4a24-ada3-b2a3033bb210-original-pull-secret\") pod \"global-pull-secret-syncer-zrg2r\" (UID: \"4f3c7912-ed80-4a24-ada3-b2a3033bb210\") " pod="kube-system/global-pull-secret-syncer-zrg2r" Jul 22 21:06:19.103716 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:06:19.103675 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 21:06:19.103716 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:06:19.103683 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 21:06:19.103716 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:06:19.103695 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 21:06:19.103716 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:06:19.103706 2572 projected.go:196] Error preparing data for projected volume kube-api-access-hqf67 for pod openshift-network-diagnostics/network-check-target-tn98z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 21:06:19.103906 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:06:19.103738 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f3c7912-ed80-4a24-ada3-b2a3033bb210-original-pull-secret podName:4f3c7912-ed80-4a24-ada3-b2a3033bb210 nodeName:}" failed. No retries permitted until 2026-07-22 21:06:51.103722495 +0000 UTC m=+66.218114819 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4f3c7912-ed80-4a24-ada3-b2a3033bb210-original-pull-secret") pod "global-pull-secret-syncer-zrg2r" (UID: "4f3c7912-ed80-4a24-ada3-b2a3033bb210") : object "kube-system"/"original-pull-secret" not registered Jul 22 21:06:19.103906 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:06:19.103753 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07cbf59d-dabc-4ba4-92b8-ec63306521cf-kube-api-access-hqf67 podName:07cbf59d-dabc-4ba4-92b8-ec63306521cf nodeName:}" failed. No retries permitted until 2026-07-22 21:06:51.103747489 +0000 UTC m=+66.218139811 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-hqf67" (UniqueName: "kubernetes.io/projected/07cbf59d-dabc-4ba4-92b8-ec63306521cf-kube-api-access-hqf67") pod "network-check-target-tn98z" (UID: "07cbf59d-dabc-4ba4-92b8-ec63306521cf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 21:06:19.377714 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:19.377629 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zrg2r" Jul 22 21:06:19.377714 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:19.377706 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tn98z" Jul 22 21:06:19.377928 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:19.377711 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lj48g" Jul 22 21:06:19.377928 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:06:19.377807 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zrg2r" podUID="4f3c7912-ed80-4a24-ada3-b2a3033bb210" Jul 22 21:06:19.378304 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:06:19.378262 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tn98z" podUID="07cbf59d-dabc-4ba4-92b8-ec63306521cf" Jul 22 21:06:19.378438 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:06:19.378372 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lj48g" podUID="8a91c48d-374d-4fc3-9114-d5c88ea18f63" Jul 22 21:06:20.723212 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:20.723177 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-37.ec2.internal" event="NodeReady" Jul 22 21:06:20.723725 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:20.723327 2572 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 22 21:06:20.769713 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:20.769623 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-7pnql"] Jul 22 21:06:20.772556 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:20.772527 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7pnql" Jul 22 21:06:20.773032 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:20.773005 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-49svb"] Jul 22 21:06:20.775122 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:20.774725 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-49svb" Jul 22 21:06:20.775122 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:20.774996 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 22 21:06:20.775325 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:20.775308 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 22 21:06:20.775416 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:20.775400 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2svwz\"" Jul 22 21:06:20.777075 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:20.776803 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 22 21:06:20.777075 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:20.776927 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 22 21:06:20.777534 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:20.777508 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 22 21:06:20.777741 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:20.777724 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-8dq9z\"" Jul 22 21:06:20.781619 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:20.781599 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7pnql"] Jul 22 21:06:20.784789 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:20.784741 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-49svb"] Jul 22 21:06:20.919995 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:20.919958 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/25aab156-c282-46cd-bd40-bfbf4991f6dc-cert\") pod \"ingress-canary-49svb\" (UID: \"25aab156-c282-46cd-bd40-bfbf4991f6dc\") " pod="openshift-ingress-canary/ingress-canary-49svb" Jul 22 21:06:20.920197 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:20.920044 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d959e22d-dd51-47e5-8e9b-0a2b883aad04-config-volume\") pod \"dns-default-7pnql\" (UID: \"d959e22d-dd51-47e5-8e9b-0a2b883aad04\") " pod="openshift-dns/dns-default-7pnql" Jul 22 21:06:20.920197 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:20.920082 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vchsb\" (UniqueName: \"kubernetes.io/projected/d959e22d-dd51-47e5-8e9b-0a2b883aad04-kube-api-access-vchsb\") pod \"dns-default-7pnql\" (UID: \"d959e22d-dd51-47e5-8e9b-0a2b883aad04\") " pod="openshift-dns/dns-default-7pnql" Jul 22 21:06:20.920197 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:20.920101 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d959e22d-dd51-47e5-8e9b-0a2b883aad04-metrics-tls\") pod \"dns-default-7pnql\" (UID: \"d959e22d-dd51-47e5-8e9b-0a2b883aad04\") " pod="openshift-dns/dns-default-7pnql" Jul 22 21:06:20.920197 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:20.920118 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsqft\" (UniqueName: \"kubernetes.io/projected/25aab156-c282-46cd-bd40-bfbf4991f6dc-kube-api-access-jsqft\") pod \"ingress-canary-49svb\" (UID: \"25aab156-c282-46cd-bd40-bfbf4991f6dc\") " pod="openshift-ingress-canary/ingress-canary-49svb" Jul 22 21:06:20.920197 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:20.920175 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d959e22d-dd51-47e5-8e9b-0a2b883aad04-tmp-dir\") pod \"dns-default-7pnql\" (UID: \"d959e22d-dd51-47e5-8e9b-0a2b883aad04\") " pod="openshift-dns/dns-default-7pnql" Jul 22 21:06:21.021346 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:21.021266 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d959e22d-dd51-47e5-8e9b-0a2b883aad04-tmp-dir\") pod \"dns-default-7pnql\" (UID: \"d959e22d-dd51-47e5-8e9b-0a2b883aad04\") " pod="openshift-dns/dns-default-7pnql" Jul 22 21:06:21.021346 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:21.021309 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/25aab156-c282-46cd-bd40-bfbf4991f6dc-cert\") pod \"ingress-canary-49svb\" (UID: \"25aab156-c282-46cd-bd40-bfbf4991f6dc\") " pod="openshift-ingress-canary/ingress-canary-49svb" Jul 22 21:06:21.021541 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:21.021383 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d959e22d-dd51-47e5-8e9b-0a2b883aad04-config-volume\") pod \"dns-default-7pnql\" (UID: \"d959e22d-dd51-47e5-8e9b-0a2b883aad04\") " pod="openshift-dns/dns-default-7pnql" Jul 22 21:06:21.021541 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:21.021420 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vchsb\" (UniqueName: \"kubernetes.io/projected/d959e22d-dd51-47e5-8e9b-0a2b883aad04-kube-api-access-vchsb\") pod \"dns-default-7pnql\" (UID: \"d959e22d-dd51-47e5-8e9b-0a2b883aad04\") " pod="openshift-dns/dns-default-7pnql" Jul 22 21:06:21.021541 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:21.021447 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d959e22d-dd51-47e5-8e9b-0a2b883aad04-metrics-tls\") pod \"dns-default-7pnql\" (UID: \"d959e22d-dd51-47e5-8e9b-0a2b883aad04\") " pod="openshift-dns/dns-default-7pnql" Jul 22 21:06:21.021541 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:21.021467 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsqft\" (UniqueName: \"kubernetes.io/projected/25aab156-c282-46cd-bd40-bfbf4991f6dc-kube-api-access-jsqft\") pod \"ingress-canary-49svb\" (UID: \"25aab156-c282-46cd-bd40-bfbf4991f6dc\") " pod="openshift-ingress-canary/ingress-canary-49svb" Jul 22 21:06:21.021703 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:21.021680 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d959e22d-dd51-47e5-8e9b-0a2b883aad04-tmp-dir\") pod \"dns-default-7pnql\" (UID: \"d959e22d-dd51-47e5-8e9b-0a2b883aad04\") " pod="openshift-dns/dns-default-7pnql" Jul 22 21:06:21.021744 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:06:21.021694 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 21:06:21.021804 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:06:21.021792 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d959e22d-dd51-47e5-8e9b-0a2b883aad04-metrics-tls podName:d959e22d-dd51-47e5-8e9b-0a2b883aad04 nodeName:}" failed. No retries permitted until 2026-07-22 21:06:21.521769022 +0000 UTC m=+36.636161345 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d959e22d-dd51-47e5-8e9b-0a2b883aad04-metrics-tls") pod "dns-default-7pnql" (UID: "d959e22d-dd51-47e5-8e9b-0a2b883aad04") : secret "dns-default-metrics-tls" not found Jul 22 21:06:21.021959 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:21.021943 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d959e22d-dd51-47e5-8e9b-0a2b883aad04-config-volume\") pod \"dns-default-7pnql\" (UID: \"d959e22d-dd51-47e5-8e9b-0a2b883aad04\") " pod="openshift-dns/dns-default-7pnql" Jul 22 21:06:21.026212 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:21.026153 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/25aab156-c282-46cd-bd40-bfbf4991f6dc-cert\") pod \"ingress-canary-49svb\" (UID: \"25aab156-c282-46cd-bd40-bfbf4991f6dc\") " pod="openshift-ingress-canary/ingress-canary-49svb" Jul 22 21:06:21.030217 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:21.030155 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vchsb\" (UniqueName: \"kubernetes.io/projected/d959e22d-dd51-47e5-8e9b-0a2b883aad04-kube-api-access-vchsb\") pod \"dns-default-7pnql\" (UID: \"d959e22d-dd51-47e5-8e9b-0a2b883aad04\") " pod="openshift-dns/dns-default-7pnql" Jul 22 21:06:21.030420 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:21.030399 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsqft\" (UniqueName: \"kubernetes.io/projected/25aab156-c282-46cd-bd40-bfbf4991f6dc-kube-api-access-jsqft\") pod \"ingress-canary-49svb\" (UID: \"25aab156-c282-46cd-bd40-bfbf4991f6dc\") " pod="openshift-ingress-canary/ingress-canary-49svb" Jul 22 21:06:21.092275 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:21.092236 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-49svb" Jul 22 21:06:21.383102 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:21.383025 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lj48g" Jul 22 21:06:21.383102 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:21.383064 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zrg2r" Jul 22 21:06:21.383102 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:21.383028 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tn98z" Jul 22 21:06:21.386741 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:21.386717 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 22 21:06:21.386863 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:21.386769 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 22 21:06:21.386863 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:21.386781 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 22 21:06:21.386863 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:21.386828 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 22 21:06:21.386863 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:21.386772 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8629h\"" Jul 22 21:06:21.387068 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:21.387046 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-s2zj6\"" Jul 22 21:06:21.525761 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:21.524757 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d959e22d-dd51-47e5-8e9b-0a2b883aad04-metrics-tls\") pod \"dns-default-7pnql\" (UID: \"d959e22d-dd51-47e5-8e9b-0a2b883aad04\") " pod="openshift-dns/dns-default-7pnql" Jul 22 21:06:21.525761 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:06:21.524936 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 21:06:21.525761 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:06:21.525005 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d959e22d-dd51-47e5-8e9b-0a2b883aad04-metrics-tls podName:d959e22d-dd51-47e5-8e9b-0a2b883aad04 nodeName:}" failed. No retries permitted until 2026-07-22 21:06:22.524975835 +0000 UTC m=+37.639368174 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d959e22d-dd51-47e5-8e9b-0a2b883aad04-metrics-tls") pod "dns-default-7pnql" (UID: "d959e22d-dd51-47e5-8e9b-0a2b883aad04") : secret "dns-default-metrics-tls" not found Jul 22 21:06:21.607473 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:21.607313 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-49svb"] Jul 22 21:06:21.610991 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:06:21.610967 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25aab156_c282_46cd_bd40_bfbf4991f6dc.slice/crio-9a68090f7a8c0ffcd8531ee89e64f44affc3629fc5a153e78907e77448280dcc WatchSource:0}: Error finding container 9a68090f7a8c0ffcd8531ee89e64f44affc3629fc5a153e78907e77448280dcc: Status 404 returned error can't find the container with id 9a68090f7a8c0ffcd8531ee89e64f44affc3629fc5a153e78907e77448280dcc Jul 22 21:06:22.532025 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:22.531982 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d959e22d-dd51-47e5-8e9b-0a2b883aad04-metrics-tls\") pod \"dns-default-7pnql\" (UID: \"d959e22d-dd51-47e5-8e9b-0a2b883aad04\") " pod="openshift-dns/dns-default-7pnql" Jul 22 21:06:22.532774 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:06:22.532109 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 21:06:22.532774 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:06:22.532193 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d959e22d-dd51-47e5-8e9b-0a2b883aad04-metrics-tls podName:d959e22d-dd51-47e5-8e9b-0a2b883aad04 nodeName:}" failed. No retries permitted until 2026-07-22 21:06:24.532154951 +0000 UTC m=+39.646547272 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d959e22d-dd51-47e5-8e9b-0a2b883aad04-metrics-tls") pod "dns-default-7pnql" (UID: "d959e22d-dd51-47e5-8e9b-0a2b883aad04") : secret "dns-default-metrics-tls" not found Jul 22 21:06:22.580605 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:22.580569 2572 generic.go:356] "Generic (PLEG): container finished" podID="cf2ac247-7e8e-46b0-991e-e82a384991c1" containerID="d56e12c501a891abc7c4cd7e84eed84cadd5a2853fffe8c026d04970af6eca38" exitCode=0 Jul 22 21:06:22.580760 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:22.580638 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-42g4d" event={"ID":"cf2ac247-7e8e-46b0-991e-e82a384991c1","Type":"ContainerDied","Data":"d56e12c501a891abc7c4cd7e84eed84cadd5a2853fffe8c026d04970af6eca38"} Jul 22 21:06:22.581820 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:22.581799 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-49svb" event={"ID":"25aab156-c282-46cd-bd40-bfbf4991f6dc","Type":"ContainerStarted","Data":"9a68090f7a8c0ffcd8531ee89e64f44affc3629fc5a153e78907e77448280dcc"} Jul 22 21:06:23.587793 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:23.587758 2572 generic.go:356] "Generic (PLEG): container finished" podID="cf2ac247-7e8e-46b0-991e-e82a384991c1" containerID="41723bfc3d19db89ec191a9c72796125a4a566d4d47565f6786cb2749023d6b9" exitCode=0 Jul 22 21:06:23.588257 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:23.587828 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-42g4d" event={"ID":"cf2ac247-7e8e-46b0-991e-e82a384991c1","Type":"ContainerDied","Data":"41723bfc3d19db89ec191a9c72796125a4a566d4d47565f6786cb2749023d6b9"} Jul 22 21:06:24.548461 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:24.548426 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d959e22d-dd51-47e5-8e9b-0a2b883aad04-metrics-tls\") pod \"dns-default-7pnql\" (UID: \"d959e22d-dd51-47e5-8e9b-0a2b883aad04\") " pod="openshift-dns/dns-default-7pnql" Jul 22 21:06:24.548628 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:06:24.548548 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 21:06:24.548628 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:06:24.548605 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d959e22d-dd51-47e5-8e9b-0a2b883aad04-metrics-tls podName:d959e22d-dd51-47e5-8e9b-0a2b883aad04 nodeName:}" failed. No retries permitted until 2026-07-22 21:06:28.548588776 +0000 UTC m=+43.662981101 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d959e22d-dd51-47e5-8e9b-0a2b883aad04-metrics-tls") pod "dns-default-7pnql" (UID: "d959e22d-dd51-47e5-8e9b-0a2b883aad04") : secret "dns-default-metrics-tls" not found Jul 22 21:06:24.593102 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:24.593073 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-42g4d" event={"ID":"cf2ac247-7e8e-46b0-991e-e82a384991c1","Type":"ContainerStarted","Data":"6339bedc32a1d93791be1ae710a30390f8150caf9dd238bafdee413677997ee9"} Jul 22 21:06:24.618237 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:24.618195 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-42g4d" podStartSLOduration=8.485814421 podStartE2EDuration="39.618180755s" totalImagesPullingTime="31.132366334s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 21:05:45 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 21:06:24.6168355 +0000 UTC m=+39.731227843" watchObservedRunningTime="2026-07-22 21:06:24.618180755 +0000 UTC m=+39.732573096" Jul 22 21:06:26.598476 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:26.598435 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-49svb" event={"ID":"25aab156-c282-46cd-bd40-bfbf4991f6dc","Type":"ContainerStarted","Data":"bdaede17cd78b5708a5e23c117827be9a71986b8d3161dd2b2a00b5d479d5895"} Jul 22 21:06:26.616224 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:26.616152 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-49svb" podStartSLOduration=2.393255277 podStartE2EDuration="6.616137951s" totalImagesPullingTime="4.222882674s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 21:06:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 21:06:26.615432178 +0000 UTC m=+41.729824522" watchObservedRunningTime="2026-07-22 21:06:26.616137951 +0000 UTC m=+41.730530272" Jul 22 21:06:28.578942 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:28.578906 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d959e22d-dd51-47e5-8e9b-0a2b883aad04-metrics-tls\") pod \"dns-default-7pnql\" (UID: \"d959e22d-dd51-47e5-8e9b-0a2b883aad04\") " pod="openshift-dns/dns-default-7pnql" Jul 22 21:06:28.579337 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:06:28.579053 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 21:06:28.579337 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:06:28.579115 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d959e22d-dd51-47e5-8e9b-0a2b883aad04-metrics-tls podName:d959e22d-dd51-47e5-8e9b-0a2b883aad04 nodeName:}" failed. No retries permitted until 2026-07-22 21:06:36.579100004 +0000 UTC m=+51.693492326 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d959e22d-dd51-47e5-8e9b-0a2b883aad04-metrics-tls") pod "dns-default-7pnql" (UID: "d959e22d-dd51-47e5-8e9b-0a2b883aad04") : secret "dns-default-metrics-tls" not found Jul 22 21:06:36.631565 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:36.631528 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d959e22d-dd51-47e5-8e9b-0a2b883aad04-metrics-tls\") pod \"dns-default-7pnql\" (UID: \"d959e22d-dd51-47e5-8e9b-0a2b883aad04\") " pod="openshift-dns/dns-default-7pnql" Jul 22 21:06:36.632010 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:06:36.631660 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 21:06:36.632010 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:06:36.631707 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d959e22d-dd51-47e5-8e9b-0a2b883aad04-metrics-tls podName:d959e22d-dd51-47e5-8e9b-0a2b883aad04 nodeName:}" failed. No retries permitted until 2026-07-22 21:06:52.631693161 +0000 UTC m=+67.746085483 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d959e22d-dd51-47e5-8e9b-0a2b883aad04-metrics-tls") pod "dns-default-7pnql" (UID: "d959e22d-dd51-47e5-8e9b-0a2b883aad04") : secret "dns-default-metrics-tls" not found Jul 22 21:06:46.576481 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:46.576454 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7w87p" Jul 22 21:06:47.426086 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:47.426050 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79cfb89bb6-2xwbp"] Jul 22 21:06:47.458979 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:47.458952 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79cfb89bb6-2xwbp"] Jul 22 21:06:47.459111 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:47.459036 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79cfb89bb6-2xwbp" Jul 22 21:06:47.462415 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:47.462388 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 22 21:06:47.462551 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:47.462425 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 22 21:06:47.462551 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:47.462394 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 22 21:06:47.462670 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:47.462551 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 22 21:06:47.462670 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:47.462554 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-bncrp\"" Jul 22 21:06:47.505420 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:47.505393 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjjx8\" (UniqueName: \"kubernetes.io/projected/2b15107c-afd9-4d23-ac63-57ee6080877d-kube-api-access-jjjx8\") pod \"managed-serviceaccount-addon-agent-79cfb89bb6-2xwbp\" (UID: \"2b15107c-afd9-4d23-ac63-57ee6080877d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79cfb89bb6-2xwbp" Jul 22 21:06:47.505532 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:47.505435 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2b15107c-afd9-4d23-ac63-57ee6080877d-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-79cfb89bb6-2xwbp\" (UID: \"2b15107c-afd9-4d23-ac63-57ee6080877d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79cfb89bb6-2xwbp" Jul 22 21:06:47.606214 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:47.606181 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2b15107c-afd9-4d23-ac63-57ee6080877d-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-79cfb89bb6-2xwbp\" (UID: \"2b15107c-afd9-4d23-ac63-57ee6080877d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79cfb89bb6-2xwbp" Jul 22 21:06:47.606639 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:47.606291 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjjx8\" (UniqueName: \"kubernetes.io/projected/2b15107c-afd9-4d23-ac63-57ee6080877d-kube-api-access-jjjx8\") pod \"managed-serviceaccount-addon-agent-79cfb89bb6-2xwbp\" (UID: \"2b15107c-afd9-4d23-ac63-57ee6080877d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79cfb89bb6-2xwbp" Jul 22 21:06:47.610481 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:47.610452 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2b15107c-afd9-4d23-ac63-57ee6080877d-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-79cfb89bb6-2xwbp\" (UID: \"2b15107c-afd9-4d23-ac63-57ee6080877d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79cfb89bb6-2xwbp" Jul 22 21:06:47.615301 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:47.615279 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjjx8\" (UniqueName: \"kubernetes.io/projected/2b15107c-afd9-4d23-ac63-57ee6080877d-kube-api-access-jjjx8\") pod \"managed-serviceaccount-addon-agent-79cfb89bb6-2xwbp\" (UID: \"2b15107c-afd9-4d23-ac63-57ee6080877d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79cfb89bb6-2xwbp" Jul 22 21:06:47.781020 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:47.780942 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79cfb89bb6-2xwbp" Jul 22 21:06:47.921356 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:47.921324 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79cfb89bb6-2xwbp"] Jul 22 21:06:47.924470 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:06:47.924444 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2b15107c_afd9_4d23_ac63_57ee6080877d.slice/crio-56446291652c2691f00a9b66efe285556727e2d7bd3cfb893e8ce954d459fbbb WatchSource:0}: Error finding container 56446291652c2691f00a9b66efe285556727e2d7bd3cfb893e8ce954d459fbbb: Status 404 returned error can't find the container with id 56446291652c2691f00a9b66efe285556727e2d7bd3cfb893e8ce954d459fbbb Jul 22 21:06:48.649152 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:48.649113 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79cfb89bb6-2xwbp" event={"ID":"2b15107c-afd9-4d23-ac63-57ee6080877d","Type":"ContainerStarted","Data":"56446291652c2691f00a9b66efe285556727e2d7bd3cfb893e8ce954d459fbbb"} Jul 22 21:06:51.034880 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:51.034782 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a91c48d-374d-4fc3-9114-d5c88ea18f63-metrics-certs\") pod \"network-metrics-daemon-lj48g\" (UID: \"8a91c48d-374d-4fc3-9114-d5c88ea18f63\") " pod="openshift-multus/network-metrics-daemon-lj48g" Jul 22 21:06:51.037424 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:51.037403 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 22 21:06:51.045760 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:06:51.045746 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 22 21:06:51.045802 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:06:51.045797 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a91c48d-374d-4fc3-9114-d5c88ea18f63-metrics-certs podName:8a91c48d-374d-4fc3-9114-d5c88ea18f63 nodeName:}" failed. No retries permitted until 2026-07-22 21:07:55.045782697 +0000 UTC m=+130.160175019 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a91c48d-374d-4fc3-9114-d5c88ea18f63-metrics-certs") pod "network-metrics-daemon-lj48g" (UID: "8a91c48d-374d-4fc3-9114-d5c88ea18f63") : secret "metrics-daemon-secret" not found Jul 22 21:06:51.135847 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:51.135816 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqf67\" (UniqueName: \"kubernetes.io/projected/07cbf59d-dabc-4ba4-92b8-ec63306521cf-kube-api-access-hqf67\") pod \"network-check-target-tn98z\" (UID: \"07cbf59d-dabc-4ba4-92b8-ec63306521cf\") " pod="openshift-network-diagnostics/network-check-target-tn98z" Jul 22 21:06:51.135969 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:51.135852 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4f3c7912-ed80-4a24-ada3-b2a3033bb210-original-pull-secret\") pod \"global-pull-secret-syncer-zrg2r\" (UID: \"4f3c7912-ed80-4a24-ada3-b2a3033bb210\") " pod="kube-system/global-pull-secret-syncer-zrg2r" Jul 22 21:06:51.138765 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:51.138750 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 22 21:06:51.138806 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:51.138778 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 22 21:06:51.148748 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:51.148722 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 22 21:06:51.148920 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:51.148901 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4f3c7912-ed80-4a24-ada3-b2a3033bb210-original-pull-secret\") pod \"global-pull-secret-syncer-zrg2r\" (UID: \"4f3c7912-ed80-4a24-ada3-b2a3033bb210\") " pod="kube-system/global-pull-secret-syncer-zrg2r" Jul 22 21:06:51.160003 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:51.159981 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqf67\" (UniqueName: \"kubernetes.io/projected/07cbf59d-dabc-4ba4-92b8-ec63306521cf-kube-api-access-hqf67\") pod \"network-check-target-tn98z\" (UID: \"07cbf59d-dabc-4ba4-92b8-ec63306521cf\") " pod="openshift-network-diagnostics/network-check-target-tn98z" Jul 22 21:06:51.402199 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:51.402154 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zrg2r" Jul 22 21:06:51.410504 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:51.410481 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-s2zj6\"" Jul 22 21:06:51.418596 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:51.418572 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tn98z" Jul 22 21:06:51.551374 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:51.551321 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zrg2r"] Jul 22 21:06:51.555036 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:06:51.555010 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f3c7912_ed80_4a24_ada3_b2a3033bb210.slice/crio-3cd5ba4ee5ec774bc23c04667673b9d2dc8db17464379a036a37cf21956cbcfe WatchSource:0}: Error finding container 3cd5ba4ee5ec774bc23c04667673b9d2dc8db17464379a036a37cf21956cbcfe: Status 404 returned error can't find the container with id 3cd5ba4ee5ec774bc23c04667673b9d2dc8db17464379a036a37cf21956cbcfe Jul 22 21:06:51.575608 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:51.575585 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tn98z"] Jul 22 21:06:51.578236 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:06:51.578213 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07cbf59d_dabc_4ba4_92b8_ec63306521cf.slice/crio-d68fbc768f3630fd69ae12d730fb56529b28fd72b0ae4860f6bd169ba9f7aecc WatchSource:0}: Error finding container d68fbc768f3630fd69ae12d730fb56529b28fd72b0ae4860f6bd169ba9f7aecc: Status 404 returned error can't find the container with id d68fbc768f3630fd69ae12d730fb56529b28fd72b0ae4860f6bd169ba9f7aecc Jul 22 21:06:51.658369 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:51.658304 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79cfb89bb6-2xwbp" event={"ID":"2b15107c-afd9-4d23-ac63-57ee6080877d","Type":"ContainerStarted","Data":"29cbfb94b9a0252d18b3f9a3cb303f3bebc4b228267342b332e014dd21092969"} Jul 22 21:06:51.659599 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:51.659575 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tn98z" event={"ID":"07cbf59d-dabc-4ba4-92b8-ec63306521cf","Type":"ContainerStarted","Data":"d68fbc768f3630fd69ae12d730fb56529b28fd72b0ae4860f6bd169ba9f7aecc"} Jul 22 21:06:51.660698 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:51.660677 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zrg2r" event={"ID":"4f3c7912-ed80-4a24-ada3-b2a3033bb210","Type":"ContainerStarted","Data":"3cd5ba4ee5ec774bc23c04667673b9d2dc8db17464379a036a37cf21956cbcfe"} Jul 22 21:06:51.674157 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:51.674119 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79cfb89bb6-2xwbp" podStartSLOduration=1.958093619 podStartE2EDuration="4.674106706s" totalImagesPullingTime="2.716013087s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 21:06:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 21:06:51.673286651 +0000 UTC m=+66.787678996" watchObservedRunningTime="2026-07-22 21:06:51.674106706 +0000 UTC m=+66.788499049" Jul 22 21:06:52.647949 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:52.647911 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d959e22d-dd51-47e5-8e9b-0a2b883aad04-metrics-tls\") pod \"dns-default-7pnql\" (UID: \"d959e22d-dd51-47e5-8e9b-0a2b883aad04\") " pod="openshift-dns/dns-default-7pnql" Jul 22 21:06:52.648380 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:06:52.648052 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 21:06:52.648380 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:06:52.648113 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d959e22d-dd51-47e5-8e9b-0a2b883aad04-metrics-tls podName:d959e22d-dd51-47e5-8e9b-0a2b883aad04 nodeName:}" failed. No retries permitted until 2026-07-22 21:07:24.648094275 +0000 UTC m=+99.762486599 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d959e22d-dd51-47e5-8e9b-0a2b883aad04-metrics-tls") pod "dns-default-7pnql" (UID: "d959e22d-dd51-47e5-8e9b-0a2b883aad04") : secret "dns-default-metrics-tls" not found Jul 22 21:06:57.678375 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:57.678336 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zrg2r" event={"ID":"4f3c7912-ed80-4a24-ada3-b2a3033bb210","Type":"ContainerStarted","Data":"23281dd8c2336d68c344b76e9b3d514680bb7d81445909a4a201bfca1167ca57"} Jul 22 21:06:57.679758 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:57.679735 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tn98z" event={"ID":"07cbf59d-dabc-4ba4-92b8-ec63306521cf","Type":"ContainerStarted","Data":"e051230a529aa6f05c93974f7c4435468544ce932410c7d5647f3f8867143e72"} Jul 22 21:06:57.679884 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:57.679870 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-tn98z" Jul 22 21:06:57.695224 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:57.695180 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-zrg2r" podStartSLOduration=67.619899057 podStartE2EDuration="1m12.695144738s" totalImagesPullingTime="5.075245681s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 21:05:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 21:06:57.694528207 +0000 UTC m=+72.808920564" watchObservedRunningTime="2026-07-22 21:06:57.695144738 +0000 UTC m=+72.809537060" Jul 22 21:06:57.708239 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:06:57.708202 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-tn98z" podStartSLOduration=67.666364181 podStartE2EDuration="1m12.7081919s" totalImagesPullingTime="5.041827719s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 21:05:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 21:06:57.707671373 +0000 UTC m=+72.822063717" watchObservedRunningTime="2026-07-22 21:06:57.7081919 +0000 UTC m=+72.822584245" Jul 22 21:07:24.667849 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:07:24.667822 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d959e22d-dd51-47e5-8e9b-0a2b883aad04-metrics-tls\") pod \"dns-default-7pnql\" (UID: \"d959e22d-dd51-47e5-8e9b-0a2b883aad04\") " pod="openshift-dns/dns-default-7pnql" Jul 22 21:07:24.668185 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:07:24.667937 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 21:07:24.668185 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:07:24.667994 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d959e22d-dd51-47e5-8e9b-0a2b883aad04-metrics-tls podName:d959e22d-dd51-47e5-8e9b-0a2b883aad04 nodeName:}" failed. No retries permitted until 2026-07-22 21:08:28.667980991 +0000 UTC m=+163.782373314 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d959e22d-dd51-47e5-8e9b-0a2b883aad04-metrics-tls") pod "dns-default-7pnql" (UID: "d959e22d-dd51-47e5-8e9b-0a2b883aad04") : secret "dns-default-metrics-tls" not found Jul 22 21:07:28.684758 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:07:28.684726 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-tn98z" Jul 22 21:07:55.071567 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:07:55.071518 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a91c48d-374d-4fc3-9114-d5c88ea18f63-metrics-certs\") pod \"network-metrics-daemon-lj48g\" (UID: \"8a91c48d-374d-4fc3-9114-d5c88ea18f63\") " pod="openshift-multus/network-metrics-daemon-lj48g" Jul 22 21:07:55.072048 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:07:55.071648 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 22 21:07:55.072048 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:07:55.071718 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a91c48d-374d-4fc3-9114-d5c88ea18f63-metrics-certs podName:8a91c48d-374d-4fc3-9114-d5c88ea18f63 nodeName:}" failed. No retries permitted until 2026-07-22 21:09:57.071700032 +0000 UTC m=+252.186092372 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a91c48d-374d-4fc3-9114-d5c88ea18f63-metrics-certs") pod "network-metrics-daemon-lj48g" (UID: "8a91c48d-374d-4fc3-9114-d5c88ea18f63") : secret "metrics-daemon-secret" not found Jul 22 21:08:23.785534 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:08:23.785480 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-7pnql" podUID="d959e22d-dd51-47e5-8e9b-0a2b883aad04" Jul 22 21:08:23.885030 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:08:23.885002 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7pnql" Jul 22 21:08:24.394233 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:08:24.394190 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-lj48g" podUID="8a91c48d-374d-4fc3-9114-d5c88ea18f63" Jul 22 21:08:28.698182 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:08:28.698130 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d959e22d-dd51-47e5-8e9b-0a2b883aad04-metrics-tls\") pod \"dns-default-7pnql\" (UID: \"d959e22d-dd51-47e5-8e9b-0a2b883aad04\") " pod="openshift-dns/dns-default-7pnql" Jul 22 21:08:28.698660 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:08:28.698277 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 21:08:28.698660 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:08:28.698357 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d959e22d-dd51-47e5-8e9b-0a2b883aad04-metrics-tls podName:d959e22d-dd51-47e5-8e9b-0a2b883aad04 nodeName:}" failed. No retries permitted until 2026-07-22 21:10:30.698337166 +0000 UTC m=+285.812729492 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d959e22d-dd51-47e5-8e9b-0a2b883aad04-metrics-tls") pod "dns-default-7pnql" (UID: "d959e22d-dd51-47e5-8e9b-0a2b883aad04") : secret "dns-default-metrics-tls" not found Jul 22 21:08:38.377216 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:08:38.377112 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lj48g" Jul 22 21:08:50.951129 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:08:50.951094 2572 generic.go:356] "Generic (PLEG): container finished" podID="2b15107c-afd9-4d23-ac63-57ee6080877d" containerID="29cbfb94b9a0252d18b3f9a3cb303f3bebc4b228267342b332e014dd21092969" exitCode=255 Jul 22 21:08:50.951506 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:08:50.951157 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79cfb89bb6-2xwbp" event={"ID":"2b15107c-afd9-4d23-ac63-57ee6080877d","Type":"ContainerDied","Data":"29cbfb94b9a0252d18b3f9a3cb303f3bebc4b228267342b332e014dd21092969"} Jul 22 21:08:50.956659 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:08:50.956635 2572 scope.go:117] "RemoveContainer" containerID="29cbfb94b9a0252d18b3f9a3cb303f3bebc4b228267342b332e014dd21092969" Jul 22 21:08:51.954698 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:08:51.954666 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79cfb89bb6-2xwbp" event={"ID":"2b15107c-afd9-4d23-ac63-57ee6080877d","Type":"ContainerStarted","Data":"61816d8b6374eb5734fb1f9efde669eca44f3bacb19177e64cac55757d127fe2"} Jul 22 21:09:48.821876 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:48.821844 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-66b4d7486b-9q9kc"] Jul 22 21:09:48.823555 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:48.823541 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-66b4d7486b-9q9kc" Jul 22 21:09:48.826598 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:48.826577 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 22 21:09:48.826729 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:48.826680 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-t2c4n\"" Jul 22 21:09:48.826729 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:48.826720 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 22 21:09:48.826936 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:48.826923 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 22 21:09:48.827277 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:48.827263 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 22 21:09:48.827477 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:48.827462 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 22 21:09:48.830141 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:48.829961 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 22 21:09:48.851247 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:48.851228 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-66b4d7486b-9q9kc"] Jul 22 21:09:48.914265 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:48.914237 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-nxdpm"] Jul 22 21:09:48.915912 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:48.915897 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-zbtpg"] Jul 22 21:09:48.916043 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:48.916027 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-nxdpm" Jul 22 21:09:48.917489 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:48.917473 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-zbtpg" Jul 22 21:09:48.918569 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:48.918551 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 22 21:09:48.918648 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:48.918568 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 22 21:09:48.918985 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:48.918843 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-g7qrf\"" Jul 22 21:09:48.924054 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:48.924036 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 22 21:09:48.924209 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:48.924041 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 22 21:09:48.924313 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:48.924074 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 22 21:09:48.924313 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:48.924115 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 22 21:09:48.924415 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:48.924189 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-p2nkl\"" Jul 22 21:09:48.925292 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:48.925269 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/14d2ff8c-748e-4a6f-801f-34cccafb93a8-stats-auth\") pod \"router-default-66b4d7486b-9q9kc\" (UID: \"14d2ff8c-748e-4a6f-801f-34cccafb93a8\") " pod="openshift-ingress/router-default-66b4d7486b-9q9kc" Jul 22 21:09:48.925388 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:48.925312 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14d2ff8c-748e-4a6f-801f-34cccafb93a8-metrics-certs\") pod \"router-default-66b4d7486b-9q9kc\" (UID: \"14d2ff8c-748e-4a6f-801f-34cccafb93a8\") " pod="openshift-ingress/router-default-66b4d7486b-9q9kc" Jul 22 21:09:48.925388 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:48.925369 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-rkwzk"] Jul 22 21:09:48.925496 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:48.925411 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/14d2ff8c-748e-4a6f-801f-34cccafb93a8-default-certificate\") pod \"router-default-66b4d7486b-9q9kc\" (UID: \"14d2ff8c-748e-4a6f-801f-34cccafb93a8\") " pod="openshift-ingress/router-default-66b4d7486b-9q9kc" Jul 22 21:09:48.925496 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:48.925454 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gdzn\" (UniqueName: \"kubernetes.io/projected/14d2ff8c-748e-4a6f-801f-34cccafb93a8-kube-api-access-7gdzn\") pod \"router-default-66b4d7486b-9q9kc\" (UID: \"14d2ff8c-748e-4a6f-801f-34cccafb93a8\") " pod="openshift-ingress/router-default-66b4d7486b-9q9kc" Jul 22 21:09:48.925601 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:48.925521 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14d2ff8c-748e-4a6f-801f-34cccafb93a8-service-ca-bundle\") pod \"router-default-66b4d7486b-9q9kc\" (UID: \"14d2ff8c-748e-4a6f-801f-34cccafb93a8\") " pod="openshift-ingress/router-default-66b4d7486b-9q9kc" Jul 22 21:09:48.927981 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:48.927522 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-66564cbd6f-4djq8"] Jul 22 21:09:48.927981 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:48.927655 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-rkwzk" Jul 22 21:09:48.929763 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:48.929742 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-nxdpm"] Jul 22 21:09:48.929914 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:48.929895 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-zbtpg"] Jul 22 21:09:48.930011 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:48.929842 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66564cbd6f-4djq8" Jul 22 21:09:48.930298 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:48.930282 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-plm98\"" Jul 22 21:09:48.930429 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:48.930316 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 22 21:09:48.930528 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:48.930512 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 22 21:09:48.930595 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:48.930583 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 22 21:09:48.934958 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:48.934939 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 22 21:09:48.935070 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:48.934969 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 22 21:09:48.935185 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:48.935152 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-7thd2\"" Jul 22 21:09:48.936987 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:48.936971 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 22 21:09:48.938633 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:48.938613 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-rkwzk"] Jul 22 21:09:48.940692 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:48.940671 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 22 21:09:48.954740 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:48.954724 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-66564cbd6f-4djq8"] Jul 22 21:09:49.026296 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:49.026270 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/14d2ff8c-748e-4a6f-801f-34cccafb93a8-default-certificate\") pod \"router-default-66b4d7486b-9q9kc\" (UID: \"14d2ff8c-748e-4a6f-801f-34cccafb93a8\") " pod="openshift-ingress/router-default-66b4d7486b-9q9kc" Jul 22 21:09:49.026420 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:49.026300 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfpg4\" (UniqueName: \"kubernetes.io/projected/53fa2e94-a71d-451a-921c-e36a2973b37f-kube-api-access-rfpg4\") pod \"volume-data-source-validator-78cf4c6f6-nxdpm\" (UID: \"53fa2e94-a71d-451a-921c-e36a2973b37f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-nxdpm" Jul 22 21:09:49.026420 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:49.026327 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bcf6675a-56d1-4606-9a1d-0330d7c6df44-installation-pull-secrets\") pod \"image-registry-66564cbd6f-4djq8\" (UID: \"bcf6675a-56d1-4606-9a1d-0330d7c6df44\") " pod="openshift-image-registry/image-registry-66564cbd6f-4djq8" Jul 22 21:09:49.026420 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:49.026347 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gdzn\" (UniqueName: \"kubernetes.io/projected/14d2ff8c-748e-4a6f-801f-34cccafb93a8-kube-api-access-7gdzn\") pod \"router-default-66b4d7486b-9q9kc\" (UID: \"14d2ff8c-748e-4a6f-801f-34cccafb93a8\") " pod="openshift-ingress/router-default-66b4d7486b-9q9kc" Jul 22 21:09:49.026420 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:49.026371 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3036abe7-96a8-4ae5-be89-97d3071f96e6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-zbtpg\" (UID: \"3036abe7-96a8-4ae5-be89-97d3071f96e6\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-zbtpg" Jul 22 21:09:49.026420 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:49.026389 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bcf6675a-56d1-4606-9a1d-0330d7c6df44-bound-sa-token\") pod \"image-registry-66564cbd6f-4djq8\" (UID: \"bcf6675a-56d1-4606-9a1d-0330d7c6df44\") " pod="openshift-image-registry/image-registry-66564cbd6f-4djq8" Jul 22 21:09:49.026420 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:49.026405 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bcf6675a-56d1-4606-9a1d-0330d7c6df44-registry-tls\") pod \"image-registry-66564cbd6f-4djq8\" (UID: \"bcf6675a-56d1-4606-9a1d-0330d7c6df44\") " pod="openshift-image-registry/image-registry-66564cbd6f-4djq8" Jul 22 21:09:49.026674 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:49.026444 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wtgz\" (UniqueName: \"kubernetes.io/projected/3036abe7-96a8-4ae5-be89-97d3071f96e6-kube-api-access-5wtgz\") pod \"cluster-monitoring-operator-76cf8966f8-zbtpg\" (UID: \"3036abe7-96a8-4ae5-be89-97d3071f96e6\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-zbtpg" Jul 22 21:09:49.026674 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:49.026523 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bcf6675a-56d1-4606-9a1d-0330d7c6df44-trusted-ca\") pod \"image-registry-66564cbd6f-4djq8\" (UID: \"bcf6675a-56d1-4606-9a1d-0330d7c6df44\") " pod="openshift-image-registry/image-registry-66564cbd6f-4djq8" Jul 22 21:09:49.026674 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:49.026570 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzhdr\" (UniqueName: \"kubernetes.io/projected/87326356-f1c4-4f61-a908-8a0d984d4b3c-kube-api-access-rzhdr\") pod \"cluster-samples-operator-79b7c55f87-rkwzk\" (UID: \"87326356-f1c4-4f61-a908-8a0d984d4b3c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-rkwzk" Jul 22 21:09:49.026674 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:49.026600 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bcf6675a-56d1-4606-9a1d-0330d7c6df44-registry-certificates\") pod \"image-registry-66564cbd6f-4djq8\" (UID: \"bcf6675a-56d1-4606-9a1d-0330d7c6df44\") " pod="openshift-image-registry/image-registry-66564cbd6f-4djq8" Jul 22 21:09:49.026674 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:49.026635 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bcf6675a-56d1-4606-9a1d-0330d7c6df44-image-registry-private-configuration\") pod \"image-registry-66564cbd6f-4djq8\" (UID: \"bcf6675a-56d1-4606-9a1d-0330d7c6df44\") " pod="openshift-image-registry/image-registry-66564cbd6f-4djq8" Jul 22 21:09:49.026674 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:49.026669 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14d2ff8c-748e-4a6f-801f-34cccafb93a8-service-ca-bundle\") pod \"router-default-66b4d7486b-9q9kc\" (UID: \"14d2ff8c-748e-4a6f-801f-34cccafb93a8\") " pod="openshift-ingress/router-default-66b4d7486b-9q9kc" Jul 22 21:09:49.026938 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:49.026694 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9k25\" (UniqueName: \"kubernetes.io/projected/bcf6675a-56d1-4606-9a1d-0330d7c6df44-kube-api-access-t9k25\") pod \"image-registry-66564cbd6f-4djq8\" (UID: \"bcf6675a-56d1-4606-9a1d-0330d7c6df44\") " pod="openshift-image-registry/image-registry-66564cbd6f-4djq8" Jul 22 21:09:49.026938 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:49.026719 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bcf6675a-56d1-4606-9a1d-0330d7c6df44-ca-trust-extracted\") pod \"image-registry-66564cbd6f-4djq8\" (UID: \"bcf6675a-56d1-4606-9a1d-0330d7c6df44\") " pod="openshift-image-registry/image-registry-66564cbd6f-4djq8" Jul 22 21:09:49.026938 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:49.026760 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/14d2ff8c-748e-4a6f-801f-34cccafb93a8-stats-auth\") pod \"router-default-66b4d7486b-9q9kc\" (UID: \"14d2ff8c-748e-4a6f-801f-34cccafb93a8\") " pod="openshift-ingress/router-default-66b4d7486b-9q9kc" Jul 22 21:09:49.026938 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:09:49.026797 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/14d2ff8c-748e-4a6f-801f-34cccafb93a8-service-ca-bundle podName:14d2ff8c-748e-4a6f-801f-34cccafb93a8 nodeName:}" failed. No retries permitted until 2026-07-22 21:09:49.52678245 +0000 UTC m=+244.641174772 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/14d2ff8c-748e-4a6f-801f-34cccafb93a8-service-ca-bundle") pod "router-default-66b4d7486b-9q9kc" (UID: "14d2ff8c-748e-4a6f-801f-34cccafb93a8") : configmap references non-existent config key: service-ca.crt Jul 22 21:09:49.026938 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:49.026813 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14d2ff8c-748e-4a6f-801f-34cccafb93a8-metrics-certs\") pod \"router-default-66b4d7486b-9q9kc\" (UID: \"14d2ff8c-748e-4a6f-801f-34cccafb93a8\") " pod="openshift-ingress/router-default-66b4d7486b-9q9kc" Jul 22 21:09:49.026938 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:49.026836 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/3036abe7-96a8-4ae5-be89-97d3071f96e6-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-zbtpg\" (UID: \"3036abe7-96a8-4ae5-be89-97d3071f96e6\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-zbtpg" Jul 22 21:09:49.026938 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:49.026862 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/87326356-f1c4-4f61-a908-8a0d984d4b3c-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-rkwzk\" (UID: \"87326356-f1c4-4f61-a908-8a0d984d4b3c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-rkwzk" Jul 22 21:09:49.026938 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:09:49.026913 2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 22 21:09:49.027290 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:09:49.026966 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14d2ff8c-748e-4a6f-801f-34cccafb93a8-metrics-certs podName:14d2ff8c-748e-4a6f-801f-34cccafb93a8 nodeName:}" failed. No retries permitted until 2026-07-22 21:09:49.526953311 +0000 UTC m=+244.641345637 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/14d2ff8c-748e-4a6f-801f-34cccafb93a8-metrics-certs") pod "router-default-66b4d7486b-9q9kc" (UID: "14d2ff8c-748e-4a6f-801f-34cccafb93a8") : secret "router-metrics-certs-default" not found Jul 22 21:09:49.028762 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:49.028742 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/14d2ff8c-748e-4a6f-801f-34cccafb93a8-default-certificate\") pod \"router-default-66b4d7486b-9q9kc\" (UID: \"14d2ff8c-748e-4a6f-801f-34cccafb93a8\") " pod="openshift-ingress/router-default-66b4d7486b-9q9kc" Jul 22 21:09:49.029053 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:49.029037 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/14d2ff8c-748e-4a6f-801f-34cccafb93a8-stats-auth\") pod \"router-default-66b4d7486b-9q9kc\" (UID: \"14d2ff8c-748e-4a6f-801f-34cccafb93a8\") " pod="openshift-ingress/router-default-66b4d7486b-9q9kc" Jul 22 21:09:49.035283 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:49.035264 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gdzn\" (UniqueName: \"kubernetes.io/projected/14d2ff8c-748e-4a6f-801f-34cccafb93a8-kube-api-access-7gdzn\") pod \"router-default-66b4d7486b-9q9kc\" (UID: \"14d2ff8c-748e-4a6f-801f-34cccafb93a8\") " pod="openshift-ingress/router-default-66b4d7486b-9q9kc" Jul 22 21:09:49.127157 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:49.127099 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wtgz\" (UniqueName: \"kubernetes.io/projected/3036abe7-96a8-4ae5-be89-97d3071f96e6-kube-api-access-5wtgz\") pod \"cluster-monitoring-operator-76cf8966f8-zbtpg\" (UID: \"3036abe7-96a8-4ae5-be89-97d3071f96e6\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-zbtpg" Jul 22 21:09:49.127157 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:49.127130 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bcf6675a-56d1-4606-9a1d-0330d7c6df44-trusted-ca\") pod \"image-registry-66564cbd6f-4djq8\" (UID: \"bcf6675a-56d1-4606-9a1d-0330d7c6df44\") " pod="openshift-image-registry/image-registry-66564cbd6f-4djq8" Jul 22 21:09:49.127339 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:49.127157 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzhdr\" (UniqueName: \"kubernetes.io/projected/87326356-f1c4-4f61-a908-8a0d984d4b3c-kube-api-access-rzhdr\") pod \"cluster-samples-operator-79b7c55f87-rkwzk\" (UID: \"87326356-f1c4-4f61-a908-8a0d984d4b3c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-rkwzk" Jul 22 21:09:49.127339 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:49.127200 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bcf6675a-56d1-4606-9a1d-0330d7c6df44-registry-certificates\") pod \"image-registry-66564cbd6f-4djq8\" (UID: \"bcf6675a-56d1-4606-9a1d-0330d7c6df44\") " pod="openshift-image-registry/image-registry-66564cbd6f-4djq8" Jul 22 21:09:49.127339 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:49.127216 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bcf6675a-56d1-4606-9a1d-0330d7c6df44-image-registry-private-configuration\") pod \"image-registry-66564cbd6f-4djq8\" (UID: \"bcf6675a-56d1-4606-9a1d-0330d7c6df44\") " pod="openshift-image-registry/image-registry-66564cbd6f-4djq8" Jul 22 21:09:49.127339 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:49.127255 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9k25\" (UniqueName: \"kubernetes.io/projected/bcf6675a-56d1-4606-9a1d-0330d7c6df44-kube-api-access-t9k25\") pod \"image-registry-66564cbd6f-4djq8\" (UID: \"bcf6675a-56d1-4606-9a1d-0330d7c6df44\") " pod="openshift-image-registry/image-registry-66564cbd6f-4djq8" Jul 22 21:09:49.127339 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:49.127272 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bcf6675a-56d1-4606-9a1d-0330d7c6df44-ca-trust-extracted\") pod \"image-registry-66564cbd6f-4djq8\" (UID: \"bcf6675a-56d1-4606-9a1d-0330d7c6df44\") " pod="openshift-image-registry/image-registry-66564cbd6f-4djq8" Jul 22 21:09:49.127339 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:49.127300 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/3036abe7-96a8-4ae5-be89-97d3071f96e6-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-zbtpg\" (UID: \"3036abe7-96a8-4ae5-be89-97d3071f96e6\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-zbtpg" Jul 22 21:09:49.127630 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:49.127356 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/87326356-f1c4-4f61-a908-8a0d984d4b3c-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-rkwzk\" (UID: \"87326356-f1c4-4f61-a908-8a0d984d4b3c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-rkwzk" Jul 22 21:09:49.127630 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:49.127414 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfpg4\" (UniqueName: \"kubernetes.io/projected/53fa2e94-a71d-451a-921c-e36a2973b37f-kube-api-access-rfpg4\") pod \"volume-data-source-validator-78cf4c6f6-nxdpm\" (UID: \"53fa2e94-a71d-451a-921c-e36a2973b37f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-nxdpm" Jul 22 21:09:49.127630 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:09:49.127503 2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 22 21:09:49.127630 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:49.127547 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bcf6675a-56d1-4606-9a1d-0330d7c6df44-installation-pull-secrets\") pod \"image-registry-66564cbd6f-4djq8\" (UID: \"bcf6675a-56d1-4606-9a1d-0330d7c6df44\") " pod="openshift-image-registry/image-registry-66564cbd6f-4djq8" Jul 22 21:09:49.127630 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:09:49.127576 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87326356-f1c4-4f61-a908-8a0d984d4b3c-samples-operator-tls podName:87326356-f1c4-4f61-a908-8a0d984d4b3c nodeName:}" failed. No retries permitted until 2026-07-22 21:09:49.627557105 +0000 UTC m=+244.741949447 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/87326356-f1c4-4f61-a908-8a0d984d4b3c-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-rkwzk" (UID: "87326356-f1c4-4f61-a908-8a0d984d4b3c") : secret "samples-operator-tls" not found Jul 22 21:09:49.127873 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:49.127629 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3036abe7-96a8-4ae5-be89-97d3071f96e6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-zbtpg\" (UID: \"3036abe7-96a8-4ae5-be89-97d3071f96e6\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-zbtpg" Jul 22 21:09:49.127873 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:49.127668 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bcf6675a-56d1-4606-9a1d-0330d7c6df44-bound-sa-token\") pod \"image-registry-66564cbd6f-4djq8\" (UID: \"bcf6675a-56d1-4606-9a1d-0330d7c6df44\") " pod="openshift-image-registry/image-registry-66564cbd6f-4djq8" Jul 22 21:09:49.127873 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:49.127693 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bcf6675a-56d1-4606-9a1d-0330d7c6df44-registry-tls\") pod \"image-registry-66564cbd6f-4djq8\" (UID: \"bcf6675a-56d1-4606-9a1d-0330d7c6df44\") " pod="openshift-image-registry/image-registry-66564cbd6f-4djq8" Jul 22 21:09:49.127873 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:09:49.127749 2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 22 21:09:49.127873 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:09:49.127794 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3036abe7-96a8-4ae5-be89-97d3071f96e6-cluster-monitoring-operator-tls podName:3036abe7-96a8-4ae5-be89-97d3071f96e6 nodeName:}" failed. No retries permitted until 2026-07-22 21:09:49.627779256 +0000 UTC m=+244.742171596 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3036abe7-96a8-4ae5-be89-97d3071f96e6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-zbtpg" (UID: "3036abe7-96a8-4ae5-be89-97d3071f96e6") : secret "cluster-monitoring-operator-tls" not found Jul 22 21:09:49.127873 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:09:49.127820 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 21:09:49.127873 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:09:49.127834 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66564cbd6f-4djq8: secret "image-registry-tls" not found Jul 22 21:09:49.128313 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:09:49.127880 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bcf6675a-56d1-4606-9a1d-0330d7c6df44-registry-tls podName:bcf6675a-56d1-4606-9a1d-0330d7c6df44 nodeName:}" failed. No retries permitted until 2026-07-22 21:09:49.627865052 +0000 UTC m=+244.742257394 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bcf6675a-56d1-4606-9a1d-0330d7c6df44-registry-tls") pod "image-registry-66564cbd6f-4djq8" (UID: "bcf6675a-56d1-4606-9a1d-0330d7c6df44") : secret "image-registry-tls" not found Jul 22 21:09:49.128313 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:49.128099 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bcf6675a-56d1-4606-9a1d-0330d7c6df44-registry-certificates\") pod \"image-registry-66564cbd6f-4djq8\" (UID: \"bcf6675a-56d1-4606-9a1d-0330d7c6df44\") " pod="openshift-image-registry/image-registry-66564cbd6f-4djq8" Jul 22 21:09:49.128313 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:49.128140 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bcf6675a-56d1-4606-9a1d-0330d7c6df44-ca-trust-extracted\") pod \"image-registry-66564cbd6f-4djq8\" (UID: \"bcf6675a-56d1-4606-9a1d-0330d7c6df44\") " pod="openshift-image-registry/image-registry-66564cbd6f-4djq8" Jul 22 21:09:49.128313 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:49.128141 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/3036abe7-96a8-4ae5-be89-97d3071f96e6-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-zbtpg\" (UID: \"3036abe7-96a8-4ae5-be89-97d3071f96e6\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-zbtpg" Jul 22 21:09:49.128313 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:49.128254 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bcf6675a-56d1-4606-9a1d-0330d7c6df44-trusted-ca\") pod \"image-registry-66564cbd6f-4djq8\" (UID: \"bcf6675a-56d1-4606-9a1d-0330d7c6df44\") " pod="openshift-image-registry/image-registry-66564cbd6f-4djq8" Jul 22 21:09:49.130180 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:49.130143 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bcf6675a-56d1-4606-9a1d-0330d7c6df44-installation-pull-secrets\") pod \"image-registry-66564cbd6f-4djq8\" (UID: \"bcf6675a-56d1-4606-9a1d-0330d7c6df44\") " pod="openshift-image-registry/image-registry-66564cbd6f-4djq8" Jul 22 21:09:49.130447 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:49.130428 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bcf6675a-56d1-4606-9a1d-0330d7c6df44-image-registry-private-configuration\") pod \"image-registry-66564cbd6f-4djq8\" (UID: \"bcf6675a-56d1-4606-9a1d-0330d7c6df44\") " pod="openshift-image-registry/image-registry-66564cbd6f-4djq8" Jul 22 21:09:49.137595 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:49.137563 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzhdr\" (UniqueName: \"kubernetes.io/projected/87326356-f1c4-4f61-a908-8a0d984d4b3c-kube-api-access-rzhdr\") pod \"cluster-samples-operator-79b7c55f87-rkwzk\" (UID: \"87326356-f1c4-4f61-a908-8a0d984d4b3c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-rkwzk" Jul 22 21:09:49.137704 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:49.137683 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wtgz\" (UniqueName: \"kubernetes.io/projected/3036abe7-96a8-4ae5-be89-97d3071f96e6-kube-api-access-5wtgz\") pod \"cluster-monitoring-operator-76cf8966f8-zbtpg\" (UID: \"3036abe7-96a8-4ae5-be89-97d3071f96e6\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-zbtpg" Jul 22 21:09:49.137750 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:49.137717 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bcf6675a-56d1-4606-9a1d-0330d7c6df44-bound-sa-token\") pod \"image-registry-66564cbd6f-4djq8\" (UID: \"bcf6675a-56d1-4606-9a1d-0330d7c6df44\") " pod="openshift-image-registry/image-registry-66564cbd6f-4djq8" Jul 22 21:09:49.137790 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:49.137748 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9k25\" (UniqueName: \"kubernetes.io/projected/bcf6675a-56d1-4606-9a1d-0330d7c6df44-kube-api-access-t9k25\") pod \"image-registry-66564cbd6f-4djq8\" (UID: \"bcf6675a-56d1-4606-9a1d-0330d7c6df44\") " pod="openshift-image-registry/image-registry-66564cbd6f-4djq8" Jul 22 21:09:49.138389 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:49.138374 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfpg4\" (UniqueName: \"kubernetes.io/projected/53fa2e94-a71d-451a-921c-e36a2973b37f-kube-api-access-rfpg4\") pod \"volume-data-source-validator-78cf4c6f6-nxdpm\" (UID: \"53fa2e94-a71d-451a-921c-e36a2973b37f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-nxdpm" Jul 22 21:09:49.231169 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:49.231146 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-nxdpm" Jul 22 21:09:49.347118 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:49.347093 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-nxdpm"] Jul 22 21:09:49.349938 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:09:49.349912 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53fa2e94_a71d_451a_921c_e36a2973b37f.slice/crio-76e14542966edd2861eedade6bc490889d0f9100b9d0e0574c459bf907868b3b WatchSource:0}: Error finding container 76e14542966edd2861eedade6bc490889d0f9100b9d0e0574c459bf907868b3b: Status 404 returned error can't find the container with id 76e14542966edd2861eedade6bc490889d0f9100b9d0e0574c459bf907868b3b Jul 22 21:09:49.530076 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:49.530010 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14d2ff8c-748e-4a6f-801f-34cccafb93a8-service-ca-bundle\") pod \"router-default-66b4d7486b-9q9kc\" (UID: \"14d2ff8c-748e-4a6f-801f-34cccafb93a8\") " pod="openshift-ingress/router-default-66b4d7486b-9q9kc" Jul 22 21:09:49.530076 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:49.530045 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14d2ff8c-748e-4a6f-801f-34cccafb93a8-metrics-certs\") pod \"router-default-66b4d7486b-9q9kc\" (UID: \"14d2ff8c-748e-4a6f-801f-34cccafb93a8\") " pod="openshift-ingress/router-default-66b4d7486b-9q9kc" Jul 22 21:09:49.530271 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:09:49.530138 2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 22 21:09:49.530271 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:09:49.530191 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/14d2ff8c-748e-4a6f-801f-34cccafb93a8-service-ca-bundle podName:14d2ff8c-748e-4a6f-801f-34cccafb93a8 nodeName:}" failed. No retries permitted until 2026-07-22 21:09:50.530148371 +0000 UTC m=+245.644540695 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/14d2ff8c-748e-4a6f-801f-34cccafb93a8-service-ca-bundle") pod "router-default-66b4d7486b-9q9kc" (UID: "14d2ff8c-748e-4a6f-801f-34cccafb93a8") : configmap references non-existent config key: service-ca.crt Jul 22 21:09:49.530366 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:09:49.530277 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14d2ff8c-748e-4a6f-801f-34cccafb93a8-metrics-certs podName:14d2ff8c-748e-4a6f-801f-34cccafb93a8 nodeName:}" failed. No retries permitted until 2026-07-22 21:09:50.530267128 +0000 UTC m=+245.644659451 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/14d2ff8c-748e-4a6f-801f-34cccafb93a8-metrics-certs") pod "router-default-66b4d7486b-9q9kc" (UID: "14d2ff8c-748e-4a6f-801f-34cccafb93a8") : secret "router-metrics-certs-default" not found Jul 22 21:09:49.631034 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:49.630998 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/87326356-f1c4-4f61-a908-8a0d984d4b3c-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-rkwzk\" (UID: \"87326356-f1c4-4f61-a908-8a0d984d4b3c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-rkwzk" Jul 22 21:09:49.631205 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:49.631069 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3036abe7-96a8-4ae5-be89-97d3071f96e6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-zbtpg\" (UID: \"3036abe7-96a8-4ae5-be89-97d3071f96e6\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-zbtpg" Jul 22 21:09:49.631205 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:49.631097 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bcf6675a-56d1-4606-9a1d-0330d7c6df44-registry-tls\") pod \"image-registry-66564cbd6f-4djq8\" (UID: \"bcf6675a-56d1-4606-9a1d-0330d7c6df44\") " pod="openshift-image-registry/image-registry-66564cbd6f-4djq8" Jul 22 21:09:49.631205 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:09:49.631137 2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 22 21:09:49.631366 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:09:49.631220 2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 22 21:09:49.631366 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:09:49.631230 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 21:09:49.631366 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:09:49.631244 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66564cbd6f-4djq8: secret "image-registry-tls" not found Jul 22 21:09:49.631366 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:09:49.631225 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87326356-f1c4-4f61-a908-8a0d984d4b3c-samples-operator-tls podName:87326356-f1c4-4f61-a908-8a0d984d4b3c nodeName:}" failed. No retries permitted until 2026-07-22 21:09:50.631207219 +0000 UTC m=+245.745599550 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/87326356-f1c4-4f61-a908-8a0d984d4b3c-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-rkwzk" (UID: "87326356-f1c4-4f61-a908-8a0d984d4b3c") : secret "samples-operator-tls" not found Jul 22 21:09:49.631366 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:09:49.631269 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3036abe7-96a8-4ae5-be89-97d3071f96e6-cluster-monitoring-operator-tls podName:3036abe7-96a8-4ae5-be89-97d3071f96e6 nodeName:}" failed. No retries permitted until 2026-07-22 21:09:50.631258435 +0000 UTC m=+245.745650757 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3036abe7-96a8-4ae5-be89-97d3071f96e6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-zbtpg" (UID: "3036abe7-96a8-4ae5-be89-97d3071f96e6") : secret "cluster-monitoring-operator-tls" not found Jul 22 21:09:49.631366 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:09:49.631299 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bcf6675a-56d1-4606-9a1d-0330d7c6df44-registry-tls podName:bcf6675a-56d1-4606-9a1d-0330d7c6df44 nodeName:}" failed. No retries permitted until 2026-07-22 21:09:50.63128172 +0000 UTC m=+245.745674062 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bcf6675a-56d1-4606-9a1d-0330d7c6df44-registry-tls") pod "image-registry-66564cbd6f-4djq8" (UID: "bcf6675a-56d1-4606-9a1d-0330d7c6df44") : secret "image-registry-tls" not found Jul 22 21:09:50.091380 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:50.091340 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-nxdpm" event={"ID":"53fa2e94-a71d-451a-921c-e36a2973b37f","Type":"ContainerStarted","Data":"76e14542966edd2861eedade6bc490889d0f9100b9d0e0574c459bf907868b3b"} Jul 22 21:09:50.539028 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:50.538995 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14d2ff8c-748e-4a6f-801f-34cccafb93a8-service-ca-bundle\") pod \"router-default-66b4d7486b-9q9kc\" (UID: \"14d2ff8c-748e-4a6f-801f-34cccafb93a8\") " pod="openshift-ingress/router-default-66b4d7486b-9q9kc" Jul 22 21:09:50.539028 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:50.539032 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14d2ff8c-748e-4a6f-801f-34cccafb93a8-metrics-certs\") pod \"router-default-66b4d7486b-9q9kc\" (UID: \"14d2ff8c-748e-4a6f-801f-34cccafb93a8\") " pod="openshift-ingress/router-default-66b4d7486b-9q9kc" Jul 22 21:09:50.539238 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:09:50.539182 2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 22 21:09:50.539238 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:09:50.539210 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/14d2ff8c-748e-4a6f-801f-34cccafb93a8-service-ca-bundle podName:14d2ff8c-748e-4a6f-801f-34cccafb93a8 nodeName:}" failed. No retries permitted until 2026-07-22 21:09:52.539194153 +0000 UTC m=+247.653586475 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/14d2ff8c-748e-4a6f-801f-34cccafb93a8-service-ca-bundle") pod "router-default-66b4d7486b-9q9kc" (UID: "14d2ff8c-748e-4a6f-801f-34cccafb93a8") : configmap references non-existent config key: service-ca.crt Jul 22 21:09:50.539238 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:09:50.539233 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14d2ff8c-748e-4a6f-801f-34cccafb93a8-metrics-certs podName:14d2ff8c-748e-4a6f-801f-34cccafb93a8 nodeName:}" failed. No retries permitted until 2026-07-22 21:09:52.539224858 +0000 UTC m=+247.653617179 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/14d2ff8c-748e-4a6f-801f-34cccafb93a8-metrics-certs") pod "router-default-66b4d7486b-9q9kc" (UID: "14d2ff8c-748e-4a6f-801f-34cccafb93a8") : secret "router-metrics-certs-default" not found Jul 22 21:09:50.639450 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:50.639421 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3036abe7-96a8-4ae5-be89-97d3071f96e6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-zbtpg\" (UID: \"3036abe7-96a8-4ae5-be89-97d3071f96e6\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-zbtpg" Jul 22 21:09:50.639450 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:50.639452 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bcf6675a-56d1-4606-9a1d-0330d7c6df44-registry-tls\") pod \"image-registry-66564cbd6f-4djq8\" (UID: \"bcf6675a-56d1-4606-9a1d-0330d7c6df44\") " pod="openshift-image-registry/image-registry-66564cbd6f-4djq8" Jul 22 21:09:50.639624 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:50.639506 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/87326356-f1c4-4f61-a908-8a0d984d4b3c-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-rkwzk\" (UID: \"87326356-f1c4-4f61-a908-8a0d984d4b3c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-rkwzk" Jul 22 21:09:50.639624 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:09:50.639559 2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 22 21:09:50.639624 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:09:50.639597 2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 22 21:09:50.639624 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:09:50.639616 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 21:09:50.639737 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:09:50.639632 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66564cbd6f-4djq8: secret "image-registry-tls" not found Jul 22 21:09:50.639737 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:09:50.639623 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3036abe7-96a8-4ae5-be89-97d3071f96e6-cluster-monitoring-operator-tls podName:3036abe7-96a8-4ae5-be89-97d3071f96e6 nodeName:}" failed. No retries permitted until 2026-07-22 21:09:52.639606464 +0000 UTC m=+247.753998789 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3036abe7-96a8-4ae5-be89-97d3071f96e6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-zbtpg" (UID: "3036abe7-96a8-4ae5-be89-97d3071f96e6") : secret "cluster-monitoring-operator-tls" not found Jul 22 21:09:50.639737 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:09:50.639678 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87326356-f1c4-4f61-a908-8a0d984d4b3c-samples-operator-tls podName:87326356-f1c4-4f61-a908-8a0d984d4b3c nodeName:}" failed. No retries permitted until 2026-07-22 21:09:52.639666557 +0000 UTC m=+247.754058878 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/87326356-f1c4-4f61-a908-8a0d984d4b3c-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-rkwzk" (UID: "87326356-f1c4-4f61-a908-8a0d984d4b3c") : secret "samples-operator-tls" not found Jul 22 21:09:50.639737 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:09:50.639691 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bcf6675a-56d1-4606-9a1d-0330d7c6df44-registry-tls podName:bcf6675a-56d1-4606-9a1d-0330d7c6df44 nodeName:}" failed. No retries permitted until 2026-07-22 21:09:52.639684959 +0000 UTC m=+247.754077281 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bcf6675a-56d1-4606-9a1d-0330d7c6df44-registry-tls") pod "image-registry-66564cbd6f-4djq8" (UID: "bcf6675a-56d1-4606-9a1d-0330d7c6df44") : secret "image-registry-tls" not found Jul 22 21:09:51.099596 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:51.099561 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-nxdpm" event={"ID":"53fa2e94-a71d-451a-921c-e36a2973b37f","Type":"ContainerStarted","Data":"6fe571482c0d26b5641f4c7ee7863456827b51b9f498f17975eb37728d8840cc"} Jul 22 21:09:51.116377 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:51.116334 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-nxdpm" podStartSLOduration=2.07299721 podStartE2EDuration="3.116321521s" totalImagesPullingTime="1.043324311s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 21:09:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 21:09:51.11481134 +0000 UTC m=+246.229203686" watchObservedRunningTime="2026-07-22 21:09:51.116321521 +0000 UTC m=+246.230713865" Jul 22 21:09:52.555512 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:52.555466 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14d2ff8c-748e-4a6f-801f-34cccafb93a8-service-ca-bundle\") pod \"router-default-66b4d7486b-9q9kc\" (UID: \"14d2ff8c-748e-4a6f-801f-34cccafb93a8\") " pod="openshift-ingress/router-default-66b4d7486b-9q9kc" Jul 22 21:09:52.555512 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:52.555519 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14d2ff8c-748e-4a6f-801f-34cccafb93a8-metrics-certs\") pod \"router-default-66b4d7486b-9q9kc\" (UID: \"14d2ff8c-748e-4a6f-801f-34cccafb93a8\") " pod="openshift-ingress/router-default-66b4d7486b-9q9kc" Jul 22 21:09:52.555993 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:09:52.555613 2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 22 21:09:52.555993 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:09:52.555648 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/14d2ff8c-748e-4a6f-801f-34cccafb93a8-service-ca-bundle podName:14d2ff8c-748e-4a6f-801f-34cccafb93a8 nodeName:}" failed. No retries permitted until 2026-07-22 21:09:56.555623698 +0000 UTC m=+251.670016026 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/14d2ff8c-748e-4a6f-801f-34cccafb93a8-service-ca-bundle") pod "router-default-66b4d7486b-9q9kc" (UID: "14d2ff8c-748e-4a6f-801f-34cccafb93a8") : configmap references non-existent config key: service-ca.crt Jul 22 21:09:52.555993 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:09:52.555681 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14d2ff8c-748e-4a6f-801f-34cccafb93a8-metrics-certs podName:14d2ff8c-748e-4a6f-801f-34cccafb93a8 nodeName:}" failed. No retries permitted until 2026-07-22 21:09:56.555668663 +0000 UTC m=+251.670060992 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/14d2ff8c-748e-4a6f-801f-34cccafb93a8-metrics-certs") pod "router-default-66b4d7486b-9q9kc" (UID: "14d2ff8c-748e-4a6f-801f-34cccafb93a8") : secret "router-metrics-certs-default" not found Jul 22 21:09:52.657082 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:52.657052 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/87326356-f1c4-4f61-a908-8a0d984d4b3c-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-rkwzk\" (UID: \"87326356-f1c4-4f61-a908-8a0d984d4b3c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-rkwzk" Jul 22 21:09:52.657251 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:52.657124 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3036abe7-96a8-4ae5-be89-97d3071f96e6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-zbtpg\" (UID: \"3036abe7-96a8-4ae5-be89-97d3071f96e6\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-zbtpg" Jul 22 21:09:52.657251 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:52.657155 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bcf6675a-56d1-4606-9a1d-0330d7c6df44-registry-tls\") pod \"image-registry-66564cbd6f-4djq8\" (UID: \"bcf6675a-56d1-4606-9a1d-0330d7c6df44\") " pod="openshift-image-registry/image-registry-66564cbd6f-4djq8" Jul 22 21:09:52.657251 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:09:52.657220 2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 22 21:09:52.657354 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:09:52.657269 2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 22 21:09:52.657354 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:09:52.657306 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 21:09:52.657354 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:09:52.657278 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87326356-f1c4-4f61-a908-8a0d984d4b3c-samples-operator-tls podName:87326356-f1c4-4f61-a908-8a0d984d4b3c nodeName:}" failed. No retries permitted until 2026-07-22 21:09:56.657261091 +0000 UTC m=+251.771653428 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/87326356-f1c4-4f61-a908-8a0d984d4b3c-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-rkwzk" (UID: "87326356-f1c4-4f61-a908-8a0d984d4b3c") : secret "samples-operator-tls" not found Jul 22 21:09:52.657354 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:09:52.657318 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66564cbd6f-4djq8: secret "image-registry-tls" not found Jul 22 21:09:52.657354 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:09:52.657333 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3036abe7-96a8-4ae5-be89-97d3071f96e6-cluster-monitoring-operator-tls podName:3036abe7-96a8-4ae5-be89-97d3071f96e6 nodeName:}" failed. No retries permitted until 2026-07-22 21:09:56.657319046 +0000 UTC m=+251.771711369 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3036abe7-96a8-4ae5-be89-97d3071f96e6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-zbtpg" (UID: "3036abe7-96a8-4ae5-be89-97d3071f96e6") : secret "cluster-monitoring-operator-tls" not found Jul 22 21:09:52.657354 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:09:52.657351 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bcf6675a-56d1-4606-9a1d-0330d7c6df44-registry-tls podName:bcf6675a-56d1-4606-9a1d-0330d7c6df44 nodeName:}" failed. No retries permitted until 2026-07-22 21:09:56.657340266 +0000 UTC m=+251.771732588 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bcf6675a-56d1-4606-9a1d-0330d7c6df44-registry-tls") pod "image-registry-66564cbd6f-4djq8" (UID: "bcf6675a-56d1-4606-9a1d-0330d7c6df44") : secret "image-registry-tls" not found Jul 22 21:09:54.399208 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:54.399170 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-qk2st"] Jul 22 21:09:54.401555 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:54.401530 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qk2st" Jul 22 21:09:54.403984 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:54.403962 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 22 21:09:54.404116 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:54.403989 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 22 21:09:54.404116 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:54.404004 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 22 21:09:54.405222 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:54.405202 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-f4mgb\"" Jul 22 21:09:54.405327 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:54.405232 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 22 21:09:54.411120 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:54.411099 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qk2st"] Jul 22 21:09:54.572587 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:54.572549 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05-data-volume\") pod \"insights-runtime-extractor-qk2st\" (UID: \"ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05\") " pod="openshift-insights/insights-runtime-extractor-qk2st" Jul 22 21:09:54.572587 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:54.572588 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05-crio-socket\") pod \"insights-runtime-extractor-qk2st\" (UID: \"ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05\") " pod="openshift-insights/insights-runtime-extractor-qk2st" Jul 22 21:09:54.572776 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:54.572609 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qk2st\" (UID: \"ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05\") " pod="openshift-insights/insights-runtime-extractor-qk2st" Jul 22 21:09:54.572776 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:54.572741 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qk2st\" (UID: \"ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05\") " pod="openshift-insights/insights-runtime-extractor-qk2st" Jul 22 21:09:54.572839 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:54.572790 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zz257\" (UniqueName: \"kubernetes.io/projected/ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05-kube-api-access-zz257\") pod \"insights-runtime-extractor-qk2st\" (UID: \"ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05\") " pod="openshift-insights/insights-runtime-extractor-qk2st" Jul 22 21:09:54.674003 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:54.673923 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zz257\" (UniqueName: \"kubernetes.io/projected/ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05-kube-api-access-zz257\") pod \"insights-runtime-extractor-qk2st\" (UID: \"ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05\") " pod="openshift-insights/insights-runtime-extractor-qk2st" Jul 22 21:09:54.674003 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:54.673968 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05-data-volume\") pod \"insights-runtime-extractor-qk2st\" (UID: \"ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05\") " pod="openshift-insights/insights-runtime-extractor-qk2st" Jul 22 21:09:54.674003 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:54.673994 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05-crio-socket\") pod \"insights-runtime-extractor-qk2st\" (UID: \"ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05\") " pod="openshift-insights/insights-runtime-extractor-qk2st" Jul 22 21:09:54.674295 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:54.674062 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05-crio-socket\") pod \"insights-runtime-extractor-qk2st\" (UID: \"ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05\") " pod="openshift-insights/insights-runtime-extractor-qk2st" Jul 22 21:09:54.674295 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:54.674095 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qk2st\" (UID: \"ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05\") " pod="openshift-insights/insights-runtime-extractor-qk2st" Jul 22 21:09:54.674295 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:09:54.674228 2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 21:09:54.674295 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:54.674241 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qk2st\" (UID: \"ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05\") " pod="openshift-insights/insights-runtime-extractor-qk2st" Jul 22 21:09:54.674295 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:09:54.674294 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05-kube-rbac-proxy-cm podName:ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05 nodeName:}" failed. No retries permitted until 2026-07-22 21:09:55.174275074 +0000 UTC m=+250.288667402 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qk2st" (UID: "ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05") : configmap "kube-rbac-proxy" not found Jul 22 21:09:54.674500 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:54.674297 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05-data-volume\") pod \"insights-runtime-extractor-qk2st\" (UID: \"ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05\") " pod="openshift-insights/insights-runtime-extractor-qk2st" Jul 22 21:09:54.674500 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:09:54.674348 2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 21:09:54.674500 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:09:54.674409 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05-insights-runtime-extractor-tls podName:ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05 nodeName:}" failed. No retries permitted until 2026-07-22 21:09:55.174390352 +0000 UTC m=+250.288782691 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qk2st" (UID: "ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05") : secret "insights-runtime-extractor-tls" not found Jul 22 21:09:54.682631 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:54.682610 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz257\" (UniqueName: \"kubernetes.io/projected/ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05-kube-api-access-zz257\") pod \"insights-runtime-extractor-qk2st\" (UID: \"ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05\") " pod="openshift-insights/insights-runtime-extractor-qk2st" Jul 22 21:09:55.178437 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:55.178401 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qk2st\" (UID: \"ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05\") " pod="openshift-insights/insights-runtime-extractor-qk2st" Jul 22 21:09:55.178602 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:55.178466 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qk2st\" (UID: \"ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05\") " pod="openshift-insights/insights-runtime-extractor-qk2st" Jul 22 21:09:55.178602 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:09:55.178541 2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 21:09:55.178602 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:09:55.178591 2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 21:09:55.178694 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:09:55.178600 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05-insights-runtime-extractor-tls podName:ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05 nodeName:}" failed. No retries permitted until 2026-07-22 21:09:56.178585182 +0000 UTC m=+251.292977507 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qk2st" (UID: "ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05") : secret "insights-runtime-extractor-tls" not found Jul 22 21:09:55.178694 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:09:55.178648 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05-kube-rbac-proxy-cm podName:ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05 nodeName:}" failed. No retries permitted until 2026-07-22 21:09:56.178633466 +0000 UTC m=+251.293025789 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qk2st" (UID: "ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05") : configmap "kube-rbac-proxy" not found Jul 22 21:09:56.185130 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:56.185091 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qk2st\" (UID: \"ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05\") " pod="openshift-insights/insights-runtime-extractor-qk2st" Jul 22 21:09:56.185563 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:56.185155 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qk2st\" (UID: \"ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05\") " pod="openshift-insights/insights-runtime-extractor-qk2st" Jul 22 21:09:56.185563 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:09:56.185234 2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 21:09:56.185563 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:09:56.185269 2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 21:09:56.185563 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:09:56.185301 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05-kube-rbac-proxy-cm podName:ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05 nodeName:}" failed. No retries permitted until 2026-07-22 21:09:58.185286722 +0000 UTC m=+253.299679044 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qk2st" (UID: "ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05") : configmap "kube-rbac-proxy" not found Jul 22 21:09:56.185563 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:09:56.185342 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05-insights-runtime-extractor-tls podName:ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05 nodeName:}" failed. No retries permitted until 2026-07-22 21:09:58.185324407 +0000 UTC m=+253.299716733 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qk2st" (UID: "ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05") : secret "insights-runtime-extractor-tls" not found Jul 22 21:09:56.588558 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:56.588530 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14d2ff8c-748e-4a6f-801f-34cccafb93a8-service-ca-bundle\") pod \"router-default-66b4d7486b-9q9kc\" (UID: \"14d2ff8c-748e-4a6f-801f-34cccafb93a8\") " pod="openshift-ingress/router-default-66b4d7486b-9q9kc" Jul 22 21:09:56.588699 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:56.588569 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14d2ff8c-748e-4a6f-801f-34cccafb93a8-metrics-certs\") pod \"router-default-66b4d7486b-9q9kc\" (UID: \"14d2ff8c-748e-4a6f-801f-34cccafb93a8\") " pod="openshift-ingress/router-default-66b4d7486b-9q9kc" Jul 22 21:09:56.588699 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:09:56.588653 2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 22 21:09:56.588699 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:09:56.588696 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14d2ff8c-748e-4a6f-801f-34cccafb93a8-metrics-certs podName:14d2ff8c-748e-4a6f-801f-34cccafb93a8 nodeName:}" failed. No retries permitted until 2026-07-22 21:10:04.58868371 +0000 UTC m=+259.703076032 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/14d2ff8c-748e-4a6f-801f-34cccafb93a8-metrics-certs") pod "router-default-66b4d7486b-9q9kc" (UID: "14d2ff8c-748e-4a6f-801f-34cccafb93a8") : secret "router-metrics-certs-default" not found Jul 22 21:09:56.588863 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:09:56.588816 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/14d2ff8c-748e-4a6f-801f-34cccafb93a8-service-ca-bundle podName:14d2ff8c-748e-4a6f-801f-34cccafb93a8 nodeName:}" failed. No retries permitted until 2026-07-22 21:10:04.588797909 +0000 UTC m=+259.703190251 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/14d2ff8c-748e-4a6f-801f-34cccafb93a8-service-ca-bundle") pod "router-default-66b4d7486b-9q9kc" (UID: "14d2ff8c-748e-4a6f-801f-34cccafb93a8") : configmap references non-existent config key: service-ca.crt Jul 22 21:09:56.588940 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:56.588924 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-z2g8w_40f77f02-2bc6-4339-a948-7631a9603ce1/dns-node-resolver/0.log" Jul 22 21:09:56.689123 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:56.689095 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/87326356-f1c4-4f61-a908-8a0d984d4b3c-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-rkwzk\" (UID: \"87326356-f1c4-4f61-a908-8a0d984d4b3c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-rkwzk" Jul 22 21:09:56.689265 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:56.689145 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3036abe7-96a8-4ae5-be89-97d3071f96e6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-zbtpg\" (UID: \"3036abe7-96a8-4ae5-be89-97d3071f96e6\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-zbtpg" Jul 22 21:09:56.689265 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:56.689186 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bcf6675a-56d1-4606-9a1d-0330d7c6df44-registry-tls\") pod \"image-registry-66564cbd6f-4djq8\" (UID: \"bcf6675a-56d1-4606-9a1d-0330d7c6df44\") " pod="openshift-image-registry/image-registry-66564cbd6f-4djq8" Jul 22 21:09:56.689265 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:09:56.689240 2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 22 21:09:56.689358 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:09:56.689296 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 21:09:56.689358 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:09:56.689302 2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 22 21:09:56.689358 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:09:56.689305 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66564cbd6f-4djq8: secret "image-registry-tls" not found Jul 22 21:09:56.689358 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:09:56.689314 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87326356-f1c4-4f61-a908-8a0d984d4b3c-samples-operator-tls podName:87326356-f1c4-4f61-a908-8a0d984d4b3c nodeName:}" failed. No retries permitted until 2026-07-22 21:10:04.689296493 +0000 UTC m=+259.803688815 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/87326356-f1c4-4f61-a908-8a0d984d4b3c-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-rkwzk" (UID: "87326356-f1c4-4f61-a908-8a0d984d4b3c") : secret "samples-operator-tls" not found Jul 22 21:09:56.689358 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:09:56.689338 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3036abe7-96a8-4ae5-be89-97d3071f96e6-cluster-monitoring-operator-tls podName:3036abe7-96a8-4ae5-be89-97d3071f96e6 nodeName:}" failed. No retries permitted until 2026-07-22 21:10:04.689328316 +0000 UTC m=+259.803720638 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3036abe7-96a8-4ae5-be89-97d3071f96e6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-zbtpg" (UID: "3036abe7-96a8-4ae5-be89-97d3071f96e6") : secret "cluster-monitoring-operator-tls" not found Jul 22 21:09:56.689358 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:09:56.689355 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bcf6675a-56d1-4606-9a1d-0330d7c6df44-registry-tls podName:bcf6675a-56d1-4606-9a1d-0330d7c6df44 nodeName:}" failed. No retries permitted until 2026-07-22 21:10:04.689344956 +0000 UTC m=+259.803737278 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bcf6675a-56d1-4606-9a1d-0330d7c6df44-registry-tls") pod "image-registry-66564cbd6f-4djq8" (UID: "bcf6675a-56d1-4606-9a1d-0330d7c6df44") : secret "image-registry-tls" not found Jul 22 21:09:57.093222 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:57.093186 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a91c48d-374d-4fc3-9114-d5c88ea18f63-metrics-certs\") pod \"network-metrics-daemon-lj48g\" (UID: \"8a91c48d-374d-4fc3-9114-d5c88ea18f63\") " pod="openshift-multus/network-metrics-daemon-lj48g" Jul 22 21:09:57.093397 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:09:57.093312 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 22 21:09:57.093397 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:09:57.093375 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a91c48d-374d-4fc3-9114-d5c88ea18f63-metrics-certs podName:8a91c48d-374d-4fc3-9114-d5c88ea18f63 nodeName:}" failed. No retries permitted until 2026-07-22 21:11:59.093361015 +0000 UTC m=+374.207753337 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a91c48d-374d-4fc3-9114-d5c88ea18f63-metrics-certs") pod "network-metrics-daemon-lj48g" (UID: "8a91c48d-374d-4fc3-9114-d5c88ea18f63") : secret "metrics-daemon-secret" not found Jul 22 21:09:57.187806 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:57.187782 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4s9cs_45b1939d-7d91-4899-b489-7e2609a5878d/node-ca/0.log" Jul 22 21:09:58.188969 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:58.188935 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-49svb_25aab156-c282-46cd-bd40-bfbf4991f6dc/serve-healthcheck-canary/0.log" Jul 22 21:09:58.202144 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:58.202122 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qk2st\" (UID: \"ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05\") " pod="openshift-insights/insights-runtime-extractor-qk2st" Jul 22 21:09:58.202231 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:09:58.202213 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qk2st\" (UID: \"ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05\") " pod="openshift-insights/insights-runtime-extractor-qk2st" Jul 22 21:09:58.202288 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:09:58.202273 2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 21:09:58.202342 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:09:58.202325 2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 21:09:58.202381 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:09:58.202335 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05-insights-runtime-extractor-tls podName:ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05 nodeName:}" failed. No retries permitted until 2026-07-22 21:10:02.202319363 +0000 UTC m=+257.316711701 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qk2st" (UID: "ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05") : secret "insights-runtime-extractor-tls" not found Jul 22 21:09:58.202431 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:09:58.202393 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05-kube-rbac-proxy-cm podName:ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05 nodeName:}" failed. No retries permitted until 2026-07-22 21:10:02.202381244 +0000 UTC m=+257.316773566 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qk2st" (UID: "ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05") : configmap "kube-rbac-proxy" not found Jul 22 21:10:02.228962 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:02.228922 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qk2st\" (UID: \"ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05\") " pod="openshift-insights/insights-runtime-extractor-qk2st" Jul 22 21:10:02.229367 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:02.229024 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qk2st\" (UID: \"ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05\") " pod="openshift-insights/insights-runtime-extractor-qk2st" Jul 22 21:10:02.229367 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:10:02.229042 2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 21:10:02.229367 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:10:02.229116 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05-kube-rbac-proxy-cm podName:ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05 nodeName:}" failed. No retries permitted until 2026-07-22 21:10:10.229098197 +0000 UTC m=+265.343490521 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qk2st" (UID: "ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05") : configmap "kube-rbac-proxy" not found Jul 22 21:10:02.229367 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:10:02.229130 2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 21:10:02.229367 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:10:02.229186 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05-insights-runtime-extractor-tls podName:ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05 nodeName:}" failed. No retries permitted until 2026-07-22 21:10:10.229156919 +0000 UTC m=+265.343549242 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qk2st" (UID: "ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05") : secret "insights-runtime-extractor-tls" not found Jul 22 21:10:04.647430 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:04.647394 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14d2ff8c-748e-4a6f-801f-34cccafb93a8-service-ca-bundle\") pod \"router-default-66b4d7486b-9q9kc\" (UID: \"14d2ff8c-748e-4a6f-801f-34cccafb93a8\") " pod="openshift-ingress/router-default-66b4d7486b-9q9kc" Jul 22 21:10:04.647430 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:04.647435 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14d2ff8c-748e-4a6f-801f-34cccafb93a8-metrics-certs\") pod \"router-default-66b4d7486b-9q9kc\" (UID: \"14d2ff8c-748e-4a6f-801f-34cccafb93a8\") " pod="openshift-ingress/router-default-66b4d7486b-9q9kc" Jul 22 21:10:04.647804 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:10:04.647576 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/14d2ff8c-748e-4a6f-801f-34cccafb93a8-service-ca-bundle podName:14d2ff8c-748e-4a6f-801f-34cccafb93a8 nodeName:}" failed. No retries permitted until 2026-07-22 21:10:20.647554581 +0000 UTC m=+275.761946917 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/14d2ff8c-748e-4a6f-801f-34cccafb93a8-service-ca-bundle") pod "router-default-66b4d7486b-9q9kc" (UID: "14d2ff8c-748e-4a6f-801f-34cccafb93a8") : configmap references non-existent config key: service-ca.crt Jul 22 21:10:04.649693 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:04.649674 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14d2ff8c-748e-4a6f-801f-34cccafb93a8-metrics-certs\") pod \"router-default-66b4d7486b-9q9kc\" (UID: \"14d2ff8c-748e-4a6f-801f-34cccafb93a8\") " pod="openshift-ingress/router-default-66b4d7486b-9q9kc" Jul 22 21:10:04.748645 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:04.748619 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/87326356-f1c4-4f61-a908-8a0d984d4b3c-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-rkwzk\" (UID: \"87326356-f1c4-4f61-a908-8a0d984d4b3c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-rkwzk" Jul 22 21:10:04.748783 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:04.748668 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3036abe7-96a8-4ae5-be89-97d3071f96e6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-zbtpg\" (UID: \"3036abe7-96a8-4ae5-be89-97d3071f96e6\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-zbtpg" Jul 22 21:10:04.748783 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:04.748694 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bcf6675a-56d1-4606-9a1d-0330d7c6df44-registry-tls\") pod \"image-registry-66564cbd6f-4djq8\" (UID: \"bcf6675a-56d1-4606-9a1d-0330d7c6df44\") " pod="openshift-image-registry/image-registry-66564cbd6f-4djq8" Jul 22 21:10:04.748783 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:10:04.748767 2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 22 21:10:04.748910 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:10:04.748823 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3036abe7-96a8-4ae5-be89-97d3071f96e6-cluster-monitoring-operator-tls podName:3036abe7-96a8-4ae5-be89-97d3071f96e6 nodeName:}" failed. No retries permitted until 2026-07-22 21:10:20.748809556 +0000 UTC m=+275.863201880 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3036abe7-96a8-4ae5-be89-97d3071f96e6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-zbtpg" (UID: "3036abe7-96a8-4ae5-be89-97d3071f96e6") : secret "cluster-monitoring-operator-tls" not found Jul 22 21:10:04.751071 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:04.751048 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bcf6675a-56d1-4606-9a1d-0330d7c6df44-registry-tls\") pod \"image-registry-66564cbd6f-4djq8\" (UID: \"bcf6675a-56d1-4606-9a1d-0330d7c6df44\") " pod="openshift-image-registry/image-registry-66564cbd6f-4djq8" Jul 22 21:10:04.751197 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:04.751149 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/87326356-f1c4-4f61-a908-8a0d984d4b3c-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-rkwzk\" (UID: \"87326356-f1c4-4f61-a908-8a0d984d4b3c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-rkwzk" Jul 22 21:10:04.845417 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:04.845361 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-rkwzk" Jul 22 21:10:04.849623 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:04.849597 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66564cbd6f-4djq8" Jul 22 21:10:04.981599 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:04.981563 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-rkwzk"] Jul 22 21:10:04.997405 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:04.997378 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-66564cbd6f-4djq8"] Jul 22 21:10:04.999791 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:10:04.999767 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcf6675a_56d1_4606_9a1d_0330d7c6df44.slice/crio-f03f8c784d7650458cc86aaf2114a4187c32567ba13addf38e7205d1904110aa WatchSource:0}: Error finding container f03f8c784d7650458cc86aaf2114a4187c32567ba13addf38e7205d1904110aa: Status 404 returned error can't find the container with id f03f8c784d7650458cc86aaf2114a4187c32567ba13addf38e7205d1904110aa Jul 22 21:10:05.140355 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:05.140318 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-66564cbd6f-4djq8" event={"ID":"bcf6675a-56d1-4606-9a1d-0330d7c6df44","Type":"ContainerStarted","Data":"e7cef73b0bb3c2025e82d6eea0de0fcd8990156596e84ca031f99ae5670e95ed"} Jul 22 21:10:05.140572 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:05.140379 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-66564cbd6f-4djq8" event={"ID":"bcf6675a-56d1-4606-9a1d-0330d7c6df44","Type":"ContainerStarted","Data":"f03f8c784d7650458cc86aaf2114a4187c32567ba13addf38e7205d1904110aa"} Jul 22 21:10:05.140572 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:05.140419 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-66564cbd6f-4djq8" Jul 22 21:10:05.141527 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:05.141503 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-rkwzk" event={"ID":"87326356-f1c4-4f61-a908-8a0d984d4b3c","Type":"ContainerStarted","Data":"c9259516d2996e03e5df90ebca64e603f2c1747a84c838c1ffc27c911eaff776"} Jul 22 21:10:08.152414 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:08.152389 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-rkwzk" event={"ID":"87326356-f1c4-4f61-a908-8a0d984d4b3c","Type":"ContainerStarted","Data":"84a7dc6522b43b90960f691eee1e0dc35f2cb95a5b966478aa66a1fc762e70d9"} Jul 22 21:10:09.156914 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:09.156877 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-rkwzk" event={"ID":"87326356-f1c4-4f61-a908-8a0d984d4b3c","Type":"ContainerStarted","Data":"40b50a196a9ad7daaa2b33e03834db5797d17cbc420ef6896aff46989bf4f85f"} Jul 22 21:10:09.172953 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:09.172905 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-rkwzk" podStartSLOduration=18.16403125 podStartE2EDuration="21.172891108s" totalImagesPullingTime="3.008859858s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 21:09:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 21:10:09.172523574 +0000 UTC m=+264.286915919" watchObservedRunningTime="2026-07-22 21:10:09.172891108 +0000 UTC m=+264.287283452" Jul 22 21:10:09.173201 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:09.173153 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-66564cbd6f-4djq8" podStartSLOduration=21.173148524 podStartE2EDuration="21.173148524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 21:09:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 21:10:05.157006145 +0000 UTC m=+260.271398488" watchObservedRunningTime="2026-07-22 21:10:09.173148524 +0000 UTC m=+264.287540867" Jul 22 21:10:10.286420 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:10.286373 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qk2st\" (UID: \"ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05\") " pod="openshift-insights/insights-runtime-extractor-qk2st" Jul 22 21:10:10.286817 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:10.286446 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qk2st\" (UID: \"ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05\") " pod="openshift-insights/insights-runtime-extractor-qk2st" Jul 22 21:10:10.286941 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:10.286925 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qk2st\" (UID: \"ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05\") " pod="openshift-insights/insights-runtime-extractor-qk2st" Jul 22 21:10:10.288785 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:10.288765 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qk2st\" (UID: \"ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05\") " pod="openshift-insights/insights-runtime-extractor-qk2st" Jul 22 21:10:10.310794 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:10.310764 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qk2st" Jul 22 21:10:10.436718 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:10.436667 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qk2st"] Jul 22 21:10:10.439073 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:10:10.439046 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef9f20ca_5f6d_44bd_bc95_f7bd98a8ff05.slice/crio-f79e59d55e22a6d43ddff4641083cc134418656136debdf61e394ec27e36db5c WatchSource:0}: Error finding container f79e59d55e22a6d43ddff4641083cc134418656136debdf61e394ec27e36db5c: Status 404 returned error can't find the container with id f79e59d55e22a6d43ddff4641083cc134418656136debdf61e394ec27e36db5c Jul 22 21:10:11.166119 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:11.166085 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qk2st" event={"ID":"ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05","Type":"ContainerStarted","Data":"f79e59d55e22a6d43ddff4641083cc134418656136debdf61e394ec27e36db5c"} Jul 22 21:10:16.184795 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:16.184752 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qk2st" event={"ID":"ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05","Type":"ContainerStarted","Data":"68a4bd8eee74ff4969f2b963669464ec441d193f8e103d653fa49604929a7fc0"} Jul 22 21:10:18.180135 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:18.180099 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-66564cbd6f-4djq8"] Jul 22 21:10:19.196004 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:19.195961 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qk2st" event={"ID":"ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05","Type":"ContainerStarted","Data":"87f5b2472f918f2d65105c6bf48c7b65301e9b9956ccbcba3f5291a393875dec"} Jul 22 21:10:20.671788 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:20.671754 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14d2ff8c-748e-4a6f-801f-34cccafb93a8-service-ca-bundle\") pod \"router-default-66b4d7486b-9q9kc\" (UID: \"14d2ff8c-748e-4a6f-801f-34cccafb93a8\") " pod="openshift-ingress/router-default-66b4d7486b-9q9kc" Jul 22 21:10:20.672469 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:20.672442 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14d2ff8c-748e-4a6f-801f-34cccafb93a8-service-ca-bundle\") pod \"router-default-66b4d7486b-9q9kc\" (UID: \"14d2ff8c-748e-4a6f-801f-34cccafb93a8\") " pod="openshift-ingress/router-default-66b4d7486b-9q9kc" Jul 22 21:10:20.772294 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:20.772251 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3036abe7-96a8-4ae5-be89-97d3071f96e6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-zbtpg\" (UID: \"3036abe7-96a8-4ae5-be89-97d3071f96e6\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-zbtpg" Jul 22 21:10:20.775477 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:20.775452 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3036abe7-96a8-4ae5-be89-97d3071f96e6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-zbtpg\" (UID: \"3036abe7-96a8-4ae5-be89-97d3071f96e6\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-zbtpg" Jul 22 21:10:20.932394 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:20.932306 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-66b4d7486b-9q9kc" Jul 22 21:10:21.038686 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:21.038545 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-zbtpg" Jul 22 21:10:21.083472 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:21.083438 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-66b4d7486b-9q9kc"] Jul 22 21:10:21.088263 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:10:21.088234 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14d2ff8c_748e_4a6f_801f_34cccafb93a8.slice/crio-32ec6f8069f66fd6eba55d07c515a7dc941d2b685d4a67e478c6e67c7f3bcb50 WatchSource:0}: Error finding container 32ec6f8069f66fd6eba55d07c515a7dc941d2b685d4a67e478c6e67c7f3bcb50: Status 404 returned error can't find the container with id 32ec6f8069f66fd6eba55d07c515a7dc941d2b685d4a67e478c6e67c7f3bcb50 Jul 22 21:10:21.185195 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:21.185083 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-zbtpg"] Jul 22 21:10:21.188723 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:10:21.188692 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3036abe7_96a8_4ae5_be89_97d3071f96e6.slice/crio-48a30e0e10d5ec4b2aa9b64f9eea8146373c60616efe9af29282eddcbd320d9b WatchSource:0}: Error finding container 48a30e0e10d5ec4b2aa9b64f9eea8146373c60616efe9af29282eddcbd320d9b: Status 404 returned error can't find the container with id 48a30e0e10d5ec4b2aa9b64f9eea8146373c60616efe9af29282eddcbd320d9b Jul 22 21:10:21.204731 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:21.204499 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-66b4d7486b-9q9kc" event={"ID":"14d2ff8c-748e-4a6f-801f-34cccafb93a8","Type":"ContainerStarted","Data":"6c627886c3644c817fcea405fb83a36658cb304236c10665e2cf2df8e529840f"} Jul 22 21:10:21.204731 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:21.204563 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-66b4d7486b-9q9kc" event={"ID":"14d2ff8c-748e-4a6f-801f-34cccafb93a8","Type":"ContainerStarted","Data":"32ec6f8069f66fd6eba55d07c515a7dc941d2b685d4a67e478c6e67c7f3bcb50"} Jul 22 21:10:21.205833 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:21.205806 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-zbtpg" event={"ID":"3036abe7-96a8-4ae5-be89-97d3071f96e6","Type":"ContainerStarted","Data":"48a30e0e10d5ec4b2aa9b64f9eea8146373c60616efe9af29282eddcbd320d9b"} Jul 22 21:10:21.225656 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:21.225605 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-66b4d7486b-9q9kc" podStartSLOduration=33.22558708 podStartE2EDuration="33.22558708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 21:09:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 21:10:21.224123733 +0000 UTC m=+276.338516114" watchObservedRunningTime="2026-07-22 21:10:21.22558708 +0000 UTC m=+276.339979429" Jul 22 21:10:21.933251 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:21.933211 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-66b4d7486b-9q9kc" Jul 22 21:10:21.936135 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:21.936114 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-66b4d7486b-9q9kc" Jul 22 21:10:22.210929 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:22.210854 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-66b4d7486b-9q9kc" Jul 22 21:10:22.212302 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:22.212278 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-66b4d7486b-9q9kc" Jul 22 21:10:23.216480 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:23.216441 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qk2st" event={"ID":"ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05","Type":"ContainerStarted","Data":"19db0a2f5fd9022c8863c5f2ea92753586202c53c4bd8529185b0dc739256e4e"} Jul 22 21:10:23.233624 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:23.233578 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-qk2st" podStartSLOduration=16.753482003 podStartE2EDuration="29.233562245s" totalImagesPullingTime="12.480080242s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 21:09:54 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 21:10:23.231982396 +0000 UTC m=+278.346374741" watchObservedRunningTime="2026-07-22 21:10:23.233562245 +0000 UTC m=+278.347954588" Jul 22 21:10:24.221307 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:24.221278 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-zbtpg" event={"ID":"3036abe7-96a8-4ae5-be89-97d3071f96e6","Type":"ContainerStarted","Data":"a778e1e7c105c661f2d43b90f47310d4aa0f0a0522c9b14a31a2545dc0eca4f9"} Jul 22 21:10:24.239779 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:24.239726 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-zbtpg" podStartSLOduration=33.274720337 podStartE2EDuration="36.239706724s" totalImagesPullingTime="2.964986387s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 21:09:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 21:10:24.238946069 +0000 UTC m=+279.353338415" watchObservedRunningTime="2026-07-22 21:10:24.239706724 +0000 UTC m=+279.354099070" Jul 22 21:10:24.629830 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:24.629801 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-v9mwz"] Jul 22 21:10:24.632841 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:24.632825 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-v9mwz" Jul 22 21:10:24.635209 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:24.635181 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 22 21:10:24.635209 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:24.635185 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-cf4nx\"" Jul 22 21:10:24.642680 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:24.642659 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-v9mwz"] Jul 22 21:10:24.708130 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:24.708099 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/41bb58e8-872b-458c-a119-5059515e61d8-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-v9mwz\" (UID: \"41bb58e8-872b-458c-a119-5059515e61d8\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-v9mwz" Jul 22 21:10:24.808681 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:24.808646 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/41bb58e8-872b-458c-a119-5059515e61d8-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-v9mwz\" (UID: \"41bb58e8-872b-458c-a119-5059515e61d8\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-v9mwz" Jul 22 21:10:24.808848 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:10:24.808779 2572 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Jul 22 21:10:24.808848 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:10:24.808835 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41bb58e8-872b-458c-a119-5059515e61d8-tls-certificates podName:41bb58e8-872b-458c-a119-5059515e61d8 nodeName:}" failed. No retries permitted until 2026-07-22 21:10:25.308820438 +0000 UTC m=+280.423212759 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/41bb58e8-872b-458c-a119-5059515e61d8-tls-certificates") pod "prometheus-operator-admission-webhook-85574b5f49-v9mwz" (UID: "41bb58e8-872b-458c-a119-5059515e61d8") : secret "prometheus-operator-admission-webhook-tls" not found Jul 22 21:10:25.313270 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:25.313230 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/41bb58e8-872b-458c-a119-5059515e61d8-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-v9mwz\" (UID: \"41bb58e8-872b-458c-a119-5059515e61d8\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-v9mwz" Jul 22 21:10:25.315867 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:25.315843 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/41bb58e8-872b-458c-a119-5059515e61d8-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-v9mwz\" (UID: \"41bb58e8-872b-458c-a119-5059515e61d8\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-v9mwz" Jul 22 21:10:25.541962 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:25.541926 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-v9mwz" Jul 22 21:10:25.673113 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:25.673081 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-v9mwz"] Jul 22 21:10:25.675880 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:10:25.675854 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41bb58e8_872b_458c_a119_5059515e61d8.slice/crio-eacc8105ce2ee73a5ffeccf0d90df3a9a66a00e778777e254b328429f4bf4f3e WatchSource:0}: Error finding container eacc8105ce2ee73a5ffeccf0d90df3a9a66a00e778777e254b328429f4bf4f3e: Status 404 returned error can't find the container with id eacc8105ce2ee73a5ffeccf0d90df3a9a66a00e778777e254b328429f4bf4f3e Jul 22 21:10:26.228808 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:26.228770 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-v9mwz" event={"ID":"41bb58e8-872b-458c-a119-5059515e61d8","Type":"ContainerStarted","Data":"eacc8105ce2ee73a5ffeccf0d90df3a9a66a00e778777e254b328429f4bf4f3e"} Jul 22 21:10:26.549621 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:26.549535 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-f896b9d68-ndrf7"] Jul 22 21:10:26.554129 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:26.554104 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f896b9d68-ndrf7" Jul 22 21:10:26.556600 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:26.556575 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 22 21:10:26.557592 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:26.557573 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-rvbmx\"" Jul 22 21:10:26.557860 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:26.557838 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 22 21:10:26.557860 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:26.557849 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 22 21:10:26.558026 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:26.557907 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 22 21:10:26.558026 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:26.557838 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 22 21:10:26.558026 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:26.557921 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 22 21:10:26.558026 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:26.557953 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 22 21:10:26.560470 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:26.560446 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f896b9d68-ndrf7"] Jul 22 21:10:26.621980 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:26.621948 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e591af84-a844-4ba8-8855-3dc41a45b70b-oauth-serving-cert\") pod \"console-f896b9d68-ndrf7\" (UID: \"e591af84-a844-4ba8-8855-3dc41a45b70b\") " pod="openshift-console/console-f896b9d68-ndrf7" Jul 22 21:10:26.622187 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:26.621991 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e591af84-a844-4ba8-8855-3dc41a45b70b-console-config\") pod \"console-f896b9d68-ndrf7\" (UID: \"e591af84-a844-4ba8-8855-3dc41a45b70b\") " pod="openshift-console/console-f896b9d68-ndrf7" Jul 22 21:10:26.622187 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:26.622010 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e591af84-a844-4ba8-8855-3dc41a45b70b-service-ca\") pod \"console-f896b9d68-ndrf7\" (UID: \"e591af84-a844-4ba8-8855-3dc41a45b70b\") " pod="openshift-console/console-f896b9d68-ndrf7" Jul 22 21:10:26.622187 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:26.622095 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e591af84-a844-4ba8-8855-3dc41a45b70b-console-serving-cert\") pod \"console-f896b9d68-ndrf7\" (UID: \"e591af84-a844-4ba8-8855-3dc41a45b70b\") " pod="openshift-console/console-f896b9d68-ndrf7" Jul 22 21:10:26.622187 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:26.622140 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e591af84-a844-4ba8-8855-3dc41a45b70b-console-oauth-config\") pod \"console-f896b9d68-ndrf7\" (UID: \"e591af84-a844-4ba8-8855-3dc41a45b70b\") " pod="openshift-console/console-f896b9d68-ndrf7" Jul 22 21:10:26.622364 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:26.622189 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5l5v\" (UniqueName: \"kubernetes.io/projected/e591af84-a844-4ba8-8855-3dc41a45b70b-kube-api-access-x5l5v\") pod \"console-f896b9d68-ndrf7\" (UID: \"e591af84-a844-4ba8-8855-3dc41a45b70b\") " pod="openshift-console/console-f896b9d68-ndrf7" Jul 22 21:10:26.722810 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:26.722775 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e591af84-a844-4ba8-8855-3dc41a45b70b-oauth-serving-cert\") pod \"console-f896b9d68-ndrf7\" (UID: \"e591af84-a844-4ba8-8855-3dc41a45b70b\") " pod="openshift-console/console-f896b9d68-ndrf7" Jul 22 21:10:26.722935 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:26.722824 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e591af84-a844-4ba8-8855-3dc41a45b70b-console-config\") pod \"console-f896b9d68-ndrf7\" (UID: \"e591af84-a844-4ba8-8855-3dc41a45b70b\") " pod="openshift-console/console-f896b9d68-ndrf7" Jul 22 21:10:26.722935 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:26.722842 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e591af84-a844-4ba8-8855-3dc41a45b70b-service-ca\") pod \"console-f896b9d68-ndrf7\" (UID: \"e591af84-a844-4ba8-8855-3dc41a45b70b\") " pod="openshift-console/console-f896b9d68-ndrf7" Jul 22 21:10:26.722935 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:26.722882 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e591af84-a844-4ba8-8855-3dc41a45b70b-console-serving-cert\") pod \"console-f896b9d68-ndrf7\" (UID: \"e591af84-a844-4ba8-8855-3dc41a45b70b\") " pod="openshift-console/console-f896b9d68-ndrf7" Jul 22 21:10:26.722935 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:26.722899 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e591af84-a844-4ba8-8855-3dc41a45b70b-console-oauth-config\") pod \"console-f896b9d68-ndrf7\" (UID: \"e591af84-a844-4ba8-8855-3dc41a45b70b\") " pod="openshift-console/console-f896b9d68-ndrf7" Jul 22 21:10:26.722935 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:26.722914 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5l5v\" (UniqueName: \"kubernetes.io/projected/e591af84-a844-4ba8-8855-3dc41a45b70b-kube-api-access-x5l5v\") pod \"console-f896b9d68-ndrf7\" (UID: \"e591af84-a844-4ba8-8855-3dc41a45b70b\") " pod="openshift-console/console-f896b9d68-ndrf7" Jul 22 21:10:26.723630 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:26.723602 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e591af84-a844-4ba8-8855-3dc41a45b70b-console-config\") pod \"console-f896b9d68-ndrf7\" (UID: \"e591af84-a844-4ba8-8855-3dc41a45b70b\") " pod="openshift-console/console-f896b9d68-ndrf7" Jul 22 21:10:26.723908 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:26.723603 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e591af84-a844-4ba8-8855-3dc41a45b70b-service-ca\") pod \"console-f896b9d68-ndrf7\" (UID: \"e591af84-a844-4ba8-8855-3dc41a45b70b\") " pod="openshift-console/console-f896b9d68-ndrf7" Jul 22 21:10:26.723908 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:26.723602 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e591af84-a844-4ba8-8855-3dc41a45b70b-oauth-serving-cert\") pod \"console-f896b9d68-ndrf7\" (UID: \"e591af84-a844-4ba8-8855-3dc41a45b70b\") " pod="openshift-console/console-f896b9d68-ndrf7" Jul 22 21:10:26.725361 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:26.725335 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e591af84-a844-4ba8-8855-3dc41a45b70b-console-oauth-config\") pod \"console-f896b9d68-ndrf7\" (UID: \"e591af84-a844-4ba8-8855-3dc41a45b70b\") " pod="openshift-console/console-f896b9d68-ndrf7" Jul 22 21:10:26.725550 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:26.725533 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e591af84-a844-4ba8-8855-3dc41a45b70b-console-serving-cert\") pod \"console-f896b9d68-ndrf7\" (UID: \"e591af84-a844-4ba8-8855-3dc41a45b70b\") " pod="openshift-console/console-f896b9d68-ndrf7" Jul 22 21:10:26.731686 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:26.731666 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5l5v\" (UniqueName: \"kubernetes.io/projected/e591af84-a844-4ba8-8855-3dc41a45b70b-kube-api-access-x5l5v\") pod \"console-f896b9d68-ndrf7\" (UID: \"e591af84-a844-4ba8-8855-3dc41a45b70b\") " pod="openshift-console/console-f896b9d68-ndrf7" Jul 22 21:10:26.863968 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:26.863927 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f896b9d68-ndrf7" Jul 22 21:10:26.886291 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:10:26.886247 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-7pnql" podUID="d959e22d-dd51-47e5-8e9b-0a2b883aad04" Jul 22 21:10:27.009208 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:27.009176 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f896b9d68-ndrf7"] Jul 22 21:10:27.011897 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:10:27.011873 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode591af84_a844_4ba8_8855_3dc41a45b70b.slice/crio-e7fe55b014d8883155596f442a6b27c40de87cb41bcacc820219d759fab2b603 WatchSource:0}: Error finding container e7fe55b014d8883155596f442a6b27c40de87cb41bcacc820219d759fab2b603: Status 404 returned error can't find the container with id e7fe55b014d8883155596f442a6b27c40de87cb41bcacc820219d759fab2b603 Jul 22 21:10:27.234154 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:27.234071 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f896b9d68-ndrf7" event={"ID":"e591af84-a844-4ba8-8855-3dc41a45b70b","Type":"ContainerStarted","Data":"e7fe55b014d8883155596f442a6b27c40de87cb41bcacc820219d759fab2b603"} Jul 22 21:10:27.234154 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:27.234075 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7pnql" Jul 22 21:10:28.186514 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:28.186489 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-66564cbd6f-4djq8" Jul 22 21:10:28.239266 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:28.239236 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-v9mwz" event={"ID":"41bb58e8-872b-458c-a119-5059515e61d8","Type":"ContainerStarted","Data":"3d70a10865abaa19dfdddd18883aed1f6661340b1c6e7b4b759264a025ccbbbc"} Jul 22 21:10:28.239504 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:28.239446 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-v9mwz" Jul 22 21:10:28.245518 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:28.245494 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-v9mwz" Jul 22 21:10:28.256005 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:28.255969 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-v9mwz" podStartSLOduration=2.480289613 podStartE2EDuration="4.25595599s" totalImagesPullingTime="1.775666377s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 21:10:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 21:10:28.25475171 +0000 UTC m=+283.369144073" watchObservedRunningTime="2026-07-22 21:10:28.25595599 +0000 UTC m=+283.370348334" Jul 22 21:10:30.755856 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:30.755829 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d959e22d-dd51-47e5-8e9b-0a2b883aad04-metrics-tls\") pod \"dns-default-7pnql\" (UID: \"d959e22d-dd51-47e5-8e9b-0a2b883aad04\") " pod="openshift-dns/dns-default-7pnql" Jul 22 21:10:30.758197 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:30.758155 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d959e22d-dd51-47e5-8e9b-0a2b883aad04-metrics-tls\") pod \"dns-default-7pnql\" (UID: \"d959e22d-dd51-47e5-8e9b-0a2b883aad04\") " pod="openshift-dns/dns-default-7pnql" Jul 22 21:10:30.837243 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:30.837139 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2svwz\"" Jul 22 21:10:30.845491 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:30.845474 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7pnql" Jul 22 21:10:30.967999 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:30.967969 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7pnql"] Jul 22 21:10:30.970648 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:10:30.970624 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd959e22d_dd51_47e5_8e9b_0a2b883aad04.slice/crio-723b8b644d38beb5449928b97a86bc0133c408d9f2408f3c7d7b07895777638c WatchSource:0}: Error finding container 723b8b644d38beb5449928b97a86bc0133c408d9f2408f3c7d7b07895777638c: Status 404 returned error can't find the container with id 723b8b644d38beb5449928b97a86bc0133c408d9f2408f3c7d7b07895777638c Jul 22 21:10:31.251404 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:31.251315 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7pnql" event={"ID":"d959e22d-dd51-47e5-8e9b-0a2b883aad04","Type":"ContainerStarted","Data":"723b8b644d38beb5449928b97a86bc0133c408d9f2408f3c7d7b07895777638c"} Jul 22 21:10:31.252741 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:31.252717 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f896b9d68-ndrf7" event={"ID":"e591af84-a844-4ba8-8855-3dc41a45b70b","Type":"ContainerStarted","Data":"fd6fbbf02fd7b3c0035a449c244bcdd2cda1283c819c3b5b3aa6ec488834b13a"} Jul 22 21:10:31.270314 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:31.270277 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-f896b9d68-ndrf7" podStartSLOduration=1.700721766 podStartE2EDuration="5.270261925s" totalImagesPullingTime="3.569540159s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 21:10:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 21:10:31.269839352 +0000 UTC m=+286.384231695" watchObservedRunningTime="2026-07-22 21:10:31.270261925 +0000 UTC m=+286.384654269" Jul 22 21:10:33.261056 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:33.261019 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7pnql" event={"ID":"d959e22d-dd51-47e5-8e9b-0a2b883aad04","Type":"ContainerStarted","Data":"979bb5c2417cc3da1621987a40c02161510d7dbcc4770cb47eb36901a228ba28"} Jul 22 21:10:33.261468 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:33.261077 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7pnql" event={"ID":"d959e22d-dd51-47e5-8e9b-0a2b883aad04","Type":"ContainerStarted","Data":"cf895add48c42b0875b9c23e3df136aea844717624c40552b038ba9cfb585039"} Jul 22 21:10:33.261468 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:33.261158 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-7pnql" Jul 22 21:10:33.278635 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:33.278594 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-7pnql" podStartSLOduration=251.589443058 podStartE2EDuration="4m13.2785779s" totalImagesPullingTime="1.689134842s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 21:06:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 21:10:33.277718582 +0000 UTC m=+288.392110923" watchObservedRunningTime="2026-07-22 21:10:33.2785779 +0000 UTC m=+288.392970243" Jul 22 21:10:35.033707 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:35.033673 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-ckz8k"] Jul 22 21:10:35.037069 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:35.037041 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ckz8k" Jul 22 21:10:35.039793 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:35.039771 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 22 21:10:35.040806 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:35.040781 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 22 21:10:35.040908 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:35.040847 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 22 21:10:35.040908 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:35.040857 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-fp9vk\"" Jul 22 21:10:35.049132 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:35.049111 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-ckz8k"] Jul 22 21:10:35.077839 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:35.077807 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-6q24d"] Jul 22 21:10:35.081045 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:35.081030 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6q24d" Jul 22 21:10:35.084552 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:35.084528 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 22 21:10:35.084689 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:35.084562 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 22 21:10:35.084689 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:35.084541 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xbmlg\"" Jul 22 21:10:35.084804 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:35.084761 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 22 21:10:35.089577 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:35.089559 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0274094c-5b35-4e27-872c-f8a1df2d7f79-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-ckz8k\" (UID: \"0274094c-5b35-4e27-872c-f8a1df2d7f79\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ckz8k" Jul 22 21:10:35.089670 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:35.089583 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0274094c-5b35-4e27-872c-f8a1df2d7f79-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-ckz8k\" (UID: \"0274094c-5b35-4e27-872c-f8a1df2d7f79\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ckz8k" Jul 22 21:10:35.089670 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:35.089625 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76bz7\" (UniqueName: \"kubernetes.io/projected/0274094c-5b35-4e27-872c-f8a1df2d7f79-kube-api-access-76bz7\") pod \"openshift-state-metrics-79f6958cdf-ckz8k\" (UID: \"0274094c-5b35-4e27-872c-f8a1df2d7f79\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ckz8k" Jul 22 21:10:35.089794 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:35.089666 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0274094c-5b35-4e27-872c-f8a1df2d7f79-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-ckz8k\" (UID: \"0274094c-5b35-4e27-872c-f8a1df2d7f79\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ckz8k" Jul 22 21:10:35.190710 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:35.190661 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0274094c-5b35-4e27-872c-f8a1df2d7f79-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-ckz8k\" (UID: \"0274094c-5b35-4e27-872c-f8a1df2d7f79\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ckz8k" Jul 22 21:10:35.190710 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:35.190697 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2e6c99e9-55f5-4cf5-b429-23c08bc76207-node-exporter-textfile\") pod \"node-exporter-6q24d\" (UID: \"2e6c99e9-55f5-4cf5-b429-23c08bc76207\") " pod="openshift-monitoring/node-exporter-6q24d" Jul 22 21:10:35.190710 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:35.190715 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2e6c99e9-55f5-4cf5-b429-23c08bc76207-root\") pod \"node-exporter-6q24d\" (UID: \"2e6c99e9-55f5-4cf5-b429-23c08bc76207\") " pod="openshift-monitoring/node-exporter-6q24d" Jul 22 21:10:35.190990 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:35.190733 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2e6c99e9-55f5-4cf5-b429-23c08bc76207-metrics-client-ca\") pod \"node-exporter-6q24d\" (UID: \"2e6c99e9-55f5-4cf5-b429-23c08bc76207\") " pod="openshift-monitoring/node-exporter-6q24d" Jul 22 21:10:35.190990 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:35.190765 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnd2t\" (UniqueName: \"kubernetes.io/projected/2e6c99e9-55f5-4cf5-b429-23c08bc76207-kube-api-access-tnd2t\") pod \"node-exporter-6q24d\" (UID: \"2e6c99e9-55f5-4cf5-b429-23c08bc76207\") " pod="openshift-monitoring/node-exporter-6q24d" Jul 22 21:10:35.190990 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:35.190807 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2e6c99e9-55f5-4cf5-b429-23c08bc76207-node-exporter-wtmp\") pod \"node-exporter-6q24d\" (UID: \"2e6c99e9-55f5-4cf5-b429-23c08bc76207\") " pod="openshift-monitoring/node-exporter-6q24d" Jul 22 21:10:35.190990 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:35.190865 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2e6c99e9-55f5-4cf5-b429-23c08bc76207-sys\") pod \"node-exporter-6q24d\" (UID: \"2e6c99e9-55f5-4cf5-b429-23c08bc76207\") " pod="openshift-monitoring/node-exporter-6q24d" Jul 22 21:10:35.190990 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:35.190887 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2e6c99e9-55f5-4cf5-b429-23c08bc76207-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6q24d\" (UID: \"2e6c99e9-55f5-4cf5-b429-23c08bc76207\") " pod="openshift-monitoring/node-exporter-6q24d" Jul 22 21:10:35.190990 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:35.190940 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0274094c-5b35-4e27-872c-f8a1df2d7f79-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-ckz8k\" (UID: \"0274094c-5b35-4e27-872c-f8a1df2d7f79\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ckz8k" Jul 22 21:10:35.190990 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:35.190958 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0274094c-5b35-4e27-872c-f8a1df2d7f79-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-ckz8k\" (UID: \"0274094c-5b35-4e27-872c-f8a1df2d7f79\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ckz8k" Jul 22 21:10:35.191377 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:35.190996 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2e6c99e9-55f5-4cf5-b429-23c08bc76207-node-exporter-tls\") pod \"node-exporter-6q24d\" (UID: \"2e6c99e9-55f5-4cf5-b429-23c08bc76207\") " pod="openshift-monitoring/node-exporter-6q24d" Jul 22 21:10:35.191377 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:35.191026 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2e6c99e9-55f5-4cf5-b429-23c08bc76207-node-exporter-accelerators-collector-config\") pod \"node-exporter-6q24d\" (UID: \"2e6c99e9-55f5-4cf5-b429-23c08bc76207\") " pod="openshift-monitoring/node-exporter-6q24d" Jul 22 21:10:35.191377 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:35.191055 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76bz7\" (UniqueName: \"kubernetes.io/projected/0274094c-5b35-4e27-872c-f8a1df2d7f79-kube-api-access-76bz7\") pod \"openshift-state-metrics-79f6958cdf-ckz8k\" (UID: \"0274094c-5b35-4e27-872c-f8a1df2d7f79\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ckz8k" Jul 22 21:10:35.191910 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:35.191887 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0274094c-5b35-4e27-872c-f8a1df2d7f79-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-ckz8k\" (UID: \"0274094c-5b35-4e27-872c-f8a1df2d7f79\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ckz8k" Jul 22 21:10:35.193429 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:35.193400 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0274094c-5b35-4e27-872c-f8a1df2d7f79-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-ckz8k\" (UID: \"0274094c-5b35-4e27-872c-f8a1df2d7f79\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ckz8k" Jul 22 21:10:35.194023 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:35.194001 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0274094c-5b35-4e27-872c-f8a1df2d7f79-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-ckz8k\" (UID: \"0274094c-5b35-4e27-872c-f8a1df2d7f79\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ckz8k" Jul 22 21:10:35.199885 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:35.199861 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76bz7\" (UniqueName: \"kubernetes.io/projected/0274094c-5b35-4e27-872c-f8a1df2d7f79-kube-api-access-76bz7\") pod \"openshift-state-metrics-79f6958cdf-ckz8k\" (UID: \"0274094c-5b35-4e27-872c-f8a1df2d7f79\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ckz8k" Jul 22 21:10:35.291682 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:35.291594 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnd2t\" (UniqueName: \"kubernetes.io/projected/2e6c99e9-55f5-4cf5-b429-23c08bc76207-kube-api-access-tnd2t\") pod \"node-exporter-6q24d\" (UID: \"2e6c99e9-55f5-4cf5-b429-23c08bc76207\") " pod="openshift-monitoring/node-exporter-6q24d" Jul 22 21:10:35.291682 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:35.291638 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2e6c99e9-55f5-4cf5-b429-23c08bc76207-node-exporter-wtmp\") pod \"node-exporter-6q24d\" (UID: \"2e6c99e9-55f5-4cf5-b429-23c08bc76207\") " pod="openshift-monitoring/node-exporter-6q24d" Jul 22 21:10:35.291682 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:35.291662 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2e6c99e9-55f5-4cf5-b429-23c08bc76207-sys\") pod \"node-exporter-6q24d\" (UID: \"2e6c99e9-55f5-4cf5-b429-23c08bc76207\") " pod="openshift-monitoring/node-exporter-6q24d" Jul 22 21:10:35.291932 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:35.291716 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2e6c99e9-55f5-4cf5-b429-23c08bc76207-sys\") pod \"node-exporter-6q24d\" (UID: \"2e6c99e9-55f5-4cf5-b429-23c08bc76207\") " pod="openshift-monitoring/node-exporter-6q24d" Jul 22 21:10:35.291932 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:35.291798 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2e6c99e9-55f5-4cf5-b429-23c08bc76207-node-exporter-wtmp\") pod \"node-exporter-6q24d\" (UID: \"2e6c99e9-55f5-4cf5-b429-23c08bc76207\") " pod="openshift-monitoring/node-exporter-6q24d" Jul 22 21:10:35.291932 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:35.291844 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2e6c99e9-55f5-4cf5-b429-23c08bc76207-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6q24d\" (UID: \"2e6c99e9-55f5-4cf5-b429-23c08bc76207\") " pod="openshift-monitoring/node-exporter-6q24d" Jul 22 21:10:35.291932 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:35.291908 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2e6c99e9-55f5-4cf5-b429-23c08bc76207-node-exporter-tls\") pod \"node-exporter-6q24d\" (UID: \"2e6c99e9-55f5-4cf5-b429-23c08bc76207\") " pod="openshift-monitoring/node-exporter-6q24d" Jul 22 21:10:35.292143 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:35.291946 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2e6c99e9-55f5-4cf5-b429-23c08bc76207-node-exporter-accelerators-collector-config\") pod \"node-exporter-6q24d\" (UID: \"2e6c99e9-55f5-4cf5-b429-23c08bc76207\") " pod="openshift-monitoring/node-exporter-6q24d" Jul 22 21:10:35.292143 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:35.291976 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2e6c99e9-55f5-4cf5-b429-23c08bc76207-node-exporter-textfile\") pod \"node-exporter-6q24d\" (UID: \"2e6c99e9-55f5-4cf5-b429-23c08bc76207\") " pod="openshift-monitoring/node-exporter-6q24d" Jul 22 21:10:35.292143 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:35.291999 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2e6c99e9-55f5-4cf5-b429-23c08bc76207-root\") pod \"node-exporter-6q24d\" (UID: \"2e6c99e9-55f5-4cf5-b429-23c08bc76207\") " pod="openshift-monitoring/node-exporter-6q24d" Jul 22 21:10:35.292143 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:35.292023 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2e6c99e9-55f5-4cf5-b429-23c08bc76207-metrics-client-ca\") pod \"node-exporter-6q24d\" (UID: \"2e6c99e9-55f5-4cf5-b429-23c08bc76207\") " pod="openshift-monitoring/node-exporter-6q24d" Jul 22 21:10:35.292143 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:10:35.292050 2572 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 22 21:10:35.292143 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:35.292073 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2e6c99e9-55f5-4cf5-b429-23c08bc76207-root\") pod \"node-exporter-6q24d\" (UID: \"2e6c99e9-55f5-4cf5-b429-23c08bc76207\") " pod="openshift-monitoring/node-exporter-6q24d" Jul 22 21:10:35.292143 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:10:35.292119 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e6c99e9-55f5-4cf5-b429-23c08bc76207-node-exporter-tls podName:2e6c99e9-55f5-4cf5-b429-23c08bc76207 nodeName:}" failed. No retries permitted until 2026-07-22 21:10:35.792098857 +0000 UTC m=+290.906491183 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/2e6c99e9-55f5-4cf5-b429-23c08bc76207-node-exporter-tls") pod "node-exporter-6q24d" (UID: "2e6c99e9-55f5-4cf5-b429-23c08bc76207") : secret "node-exporter-tls" not found Jul 22 21:10:35.292416 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:35.292392 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2e6c99e9-55f5-4cf5-b429-23c08bc76207-node-exporter-textfile\") pod \"node-exporter-6q24d\" (UID: \"2e6c99e9-55f5-4cf5-b429-23c08bc76207\") " pod="openshift-monitoring/node-exporter-6q24d" Jul 22 21:10:35.292575 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:35.292558 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2e6c99e9-55f5-4cf5-b429-23c08bc76207-node-exporter-accelerators-collector-config\") pod \"node-exporter-6q24d\" (UID: \"2e6c99e9-55f5-4cf5-b429-23c08bc76207\") " pod="openshift-monitoring/node-exporter-6q24d" Jul 22 21:10:35.292609 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:35.292580 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2e6c99e9-55f5-4cf5-b429-23c08bc76207-metrics-client-ca\") pod \"node-exporter-6q24d\" (UID: \"2e6c99e9-55f5-4cf5-b429-23c08bc76207\") " pod="openshift-monitoring/node-exporter-6q24d" Jul 22 21:10:35.294356 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:35.294340 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2e6c99e9-55f5-4cf5-b429-23c08bc76207-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6q24d\" (UID: \"2e6c99e9-55f5-4cf5-b429-23c08bc76207\") " pod="openshift-monitoring/node-exporter-6q24d" Jul 22 21:10:35.300459 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:35.300435 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnd2t\" (UniqueName: \"kubernetes.io/projected/2e6c99e9-55f5-4cf5-b429-23c08bc76207-kube-api-access-tnd2t\") pod \"node-exporter-6q24d\" (UID: \"2e6c99e9-55f5-4cf5-b429-23c08bc76207\") " pod="openshift-monitoring/node-exporter-6q24d" Jul 22 21:10:35.345926 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:35.345896 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ckz8k" Jul 22 21:10:35.478408 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:35.478373 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-ckz8k"] Jul 22 21:10:35.481003 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:10:35.480979 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0274094c_5b35_4e27_872c_f8a1df2d7f79.slice/crio-0cd49b9f825af1239464dbf76cbe714b199e8b08e811b619423cfc7710ff7b31 WatchSource:0}: Error finding container 0cd49b9f825af1239464dbf76cbe714b199e8b08e811b619423cfc7710ff7b31: Status 404 returned error can't find the container with id 0cd49b9f825af1239464dbf76cbe714b199e8b08e811b619423cfc7710ff7b31 Jul 22 21:10:35.796916 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:35.796829 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2e6c99e9-55f5-4cf5-b429-23c08bc76207-node-exporter-tls\") pod \"node-exporter-6q24d\" (UID: \"2e6c99e9-55f5-4cf5-b429-23c08bc76207\") " pod="openshift-monitoring/node-exporter-6q24d" Jul 22 21:10:35.799492 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:35.799467 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2e6c99e9-55f5-4cf5-b429-23c08bc76207-node-exporter-tls\") pod \"node-exporter-6q24d\" (UID: \"2e6c99e9-55f5-4cf5-b429-23c08bc76207\") " pod="openshift-monitoring/node-exporter-6q24d" Jul 22 21:10:35.989830 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:35.989796 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6q24d" Jul 22 21:10:36.003003 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:10:36.002978 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e6c99e9_55f5_4cf5_b429_23c08bc76207.slice/crio-35b67b15743afa6ba6548732f7d6e939bb914f9cb7fbfc8514d5e0a2f4be187f WatchSource:0}: Error finding container 35b67b15743afa6ba6548732f7d6e939bb914f9cb7fbfc8514d5e0a2f4be187f: Status 404 returned error can't find the container with id 35b67b15743afa6ba6548732f7d6e939bb914f9cb7fbfc8514d5e0a2f4be187f Jul 22 21:10:36.092030 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:36.091993 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 21:10:36.096849 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:36.096823 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 22 21:10:36.099724 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:36.099702 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-8p5gr\"" Jul 22 21:10:36.099864 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:36.099735 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 22 21:10:36.099864 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:36.099701 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 22 21:10:36.099864 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:36.099817 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 22 21:10:36.100073 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:36.100057 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 22 21:10:36.100123 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:36.100085 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 22 21:10:36.100123 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:36.100087 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 22 21:10:36.100263 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:36.100092 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 22 21:10:36.100534 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:36.100514 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 22 21:10:36.100589 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:36.100534 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 22 21:10:36.110541 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:36.110518 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 21:10:36.199307 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:36.199275 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fb5f129e-7b15-49fa-9f02-4978dc5e60aa-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"fb5f129e-7b15-49fa-9f02-4978dc5e60aa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 21:10:36.199564 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:36.199319 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fb5f129e-7b15-49fa-9f02-4978dc5e60aa-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"fb5f129e-7b15-49fa-9f02-4978dc5e60aa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 21:10:36.199564 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:36.199349 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fb5f129e-7b15-49fa-9f02-4978dc5e60aa-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"fb5f129e-7b15-49fa-9f02-4978dc5e60aa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 21:10:36.199564 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:36.199380 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fb5f129e-7b15-49fa-9f02-4978dc5e60aa-web-config\") pod \"alertmanager-main-0\" (UID: \"fb5f129e-7b15-49fa-9f02-4978dc5e60aa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 21:10:36.199564 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:36.199414 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv4d2\" (UniqueName: \"kubernetes.io/projected/fb5f129e-7b15-49fa-9f02-4978dc5e60aa-kube-api-access-pv4d2\") pod \"alertmanager-main-0\" (UID: \"fb5f129e-7b15-49fa-9f02-4978dc5e60aa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 21:10:36.199564 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:36.199461 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fb5f129e-7b15-49fa-9f02-4978dc5e60aa-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"fb5f129e-7b15-49fa-9f02-4978dc5e60aa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 21:10:36.199564 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:36.199484 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/fb5f129e-7b15-49fa-9f02-4978dc5e60aa-config-volume\") pod \"alertmanager-main-0\" (UID: \"fb5f129e-7b15-49fa-9f02-4978dc5e60aa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 21:10:36.199564 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:36.199507 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb5f129e-7b15-49fa-9f02-4978dc5e60aa-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"fb5f129e-7b15-49fa-9f02-4978dc5e60aa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 21:10:36.199564 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:36.199528 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fb5f129e-7b15-49fa-9f02-4978dc5e60aa-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"fb5f129e-7b15-49fa-9f02-4978dc5e60aa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 21:10:36.199564 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:36.199544 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fb5f129e-7b15-49fa-9f02-4978dc5e60aa-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"fb5f129e-7b15-49fa-9f02-4978dc5e60aa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 21:10:36.199869 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:36.199567 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fb5f129e-7b15-49fa-9f02-4978dc5e60aa-config-out\") pod \"alertmanager-main-0\" (UID: \"fb5f129e-7b15-49fa-9f02-4978dc5e60aa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 21:10:36.199869 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:36.199605 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fb5f129e-7b15-49fa-9f02-4978dc5e60aa-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"fb5f129e-7b15-49fa-9f02-4978dc5e60aa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 21:10:36.199869 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:36.199631 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fb5f129e-7b15-49fa-9f02-4978dc5e60aa-tls-assets\") pod \"alertmanager-main-0\" (UID: \"fb5f129e-7b15-49fa-9f02-4978dc5e60aa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 21:10:36.273967 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:36.273933 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6q24d" event={"ID":"2e6c99e9-55f5-4cf5-b429-23c08bc76207","Type":"ContainerStarted","Data":"35b67b15743afa6ba6548732f7d6e939bb914f9cb7fbfc8514d5e0a2f4be187f"} Jul 22 21:10:36.275776 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:36.275748 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ckz8k" event={"ID":"0274094c-5b35-4e27-872c-f8a1df2d7f79","Type":"ContainerStarted","Data":"b89029ad08217c9ae4fe8469e3ac2cb42eca476741f59976e0db2bb1376891cf"} Jul 22 21:10:36.275891 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:36.275801 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ckz8k" event={"ID":"0274094c-5b35-4e27-872c-f8a1df2d7f79","Type":"ContainerStarted","Data":"a1c43a13f37311f1792789afb37038194439c703343a74191b5d97e5f70bf9ee"} Jul 22 21:10:36.275891 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:36.275822 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ckz8k" event={"ID":"0274094c-5b35-4e27-872c-f8a1df2d7f79","Type":"ContainerStarted","Data":"0cd49b9f825af1239464dbf76cbe714b199e8b08e811b619423cfc7710ff7b31"} Jul 22 21:10:36.300326 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:36.300287 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fb5f129e-7b15-49fa-9f02-4978dc5e60aa-web-config\") pod \"alertmanager-main-0\" (UID: \"fb5f129e-7b15-49fa-9f02-4978dc5e60aa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 21:10:36.300495 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:36.300341 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv4d2\" (UniqueName: \"kubernetes.io/projected/fb5f129e-7b15-49fa-9f02-4978dc5e60aa-kube-api-access-pv4d2\") pod \"alertmanager-main-0\" (UID: \"fb5f129e-7b15-49fa-9f02-4978dc5e60aa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 21:10:36.300495 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:36.300466 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fb5f129e-7b15-49fa-9f02-4978dc5e60aa-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"fb5f129e-7b15-49fa-9f02-4978dc5e60aa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 21:10:36.300613 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:36.300510 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/fb5f129e-7b15-49fa-9f02-4978dc5e60aa-config-volume\") pod \"alertmanager-main-0\" (UID: \"fb5f129e-7b15-49fa-9f02-4978dc5e60aa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 21:10:36.300613 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:36.300528 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb5f129e-7b15-49fa-9f02-4978dc5e60aa-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"fb5f129e-7b15-49fa-9f02-4978dc5e60aa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 21:10:36.300613 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:36.300563 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fb5f129e-7b15-49fa-9f02-4978dc5e60aa-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"fb5f129e-7b15-49fa-9f02-4978dc5e60aa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 21:10:36.300613 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:36.300603 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fb5f129e-7b15-49fa-9f02-4978dc5e60aa-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"fb5f129e-7b15-49fa-9f02-4978dc5e60aa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 21:10:36.300801 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:36.300636 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fb5f129e-7b15-49fa-9f02-4978dc5e60aa-config-out\") pod \"alertmanager-main-0\" (UID: \"fb5f129e-7b15-49fa-9f02-4978dc5e60aa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 21:10:36.300801 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:36.300665 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fb5f129e-7b15-49fa-9f02-4978dc5e60aa-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"fb5f129e-7b15-49fa-9f02-4978dc5e60aa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 21:10:36.300801 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:36.300696 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fb5f129e-7b15-49fa-9f02-4978dc5e60aa-tls-assets\") pod \"alertmanager-main-0\" (UID: \"fb5f129e-7b15-49fa-9f02-4978dc5e60aa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 21:10:36.300801 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:36.300739 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fb5f129e-7b15-49fa-9f02-4978dc5e60aa-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"fb5f129e-7b15-49fa-9f02-4978dc5e60aa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 21:10:36.300801 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:36.300787 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fb5f129e-7b15-49fa-9f02-4978dc5e60aa-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"fb5f129e-7b15-49fa-9f02-4978dc5e60aa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 21:10:36.301056 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:36.300816 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fb5f129e-7b15-49fa-9f02-4978dc5e60aa-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"fb5f129e-7b15-49fa-9f02-4978dc5e60aa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 21:10:36.301399 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:36.301148 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fb5f129e-7b15-49fa-9f02-4978dc5e60aa-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"fb5f129e-7b15-49fa-9f02-4978dc5e60aa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 21:10:36.302106 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:36.302077 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fb5f129e-7b15-49fa-9f02-4978dc5e60aa-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"fb5f129e-7b15-49fa-9f02-4978dc5e60aa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 21:10:36.303554 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:36.303528 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb5f129e-7b15-49fa-9f02-4978dc5e60aa-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"fb5f129e-7b15-49fa-9f02-4978dc5e60aa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 21:10:36.304180 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:36.304135 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/fb5f129e-7b15-49fa-9f02-4978dc5e60aa-config-volume\") pod \"alertmanager-main-0\" (UID: \"fb5f129e-7b15-49fa-9f02-4978dc5e60aa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 21:10:36.304521 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:36.304497 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fb5f129e-7b15-49fa-9f02-4978dc5e60aa-tls-assets\") pod \"alertmanager-main-0\" (UID: \"fb5f129e-7b15-49fa-9f02-4978dc5e60aa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 21:10:36.305156 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:36.305110 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fb5f129e-7b15-49fa-9f02-4978dc5e60aa-web-config\") pod \"alertmanager-main-0\" (UID: \"fb5f129e-7b15-49fa-9f02-4978dc5e60aa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 21:10:36.305156 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:36.305132 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fb5f129e-7b15-49fa-9f02-4978dc5e60aa-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"fb5f129e-7b15-49fa-9f02-4978dc5e60aa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 21:10:36.305325 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:36.305236 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fb5f129e-7b15-49fa-9f02-4978dc5e60aa-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"fb5f129e-7b15-49fa-9f02-4978dc5e60aa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 21:10:36.305620 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:36.305600 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fb5f129e-7b15-49fa-9f02-4978dc5e60aa-config-out\") pod \"alertmanager-main-0\" (UID: \"fb5f129e-7b15-49fa-9f02-4978dc5e60aa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 21:10:36.305702 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:36.305610 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fb5f129e-7b15-49fa-9f02-4978dc5e60aa-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"fb5f129e-7b15-49fa-9f02-4978dc5e60aa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 21:10:36.306236 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:36.306219 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fb5f129e-7b15-49fa-9f02-4978dc5e60aa-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"fb5f129e-7b15-49fa-9f02-4978dc5e60aa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 21:10:36.306609 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:36.306589 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fb5f129e-7b15-49fa-9f02-4978dc5e60aa-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"fb5f129e-7b15-49fa-9f02-4978dc5e60aa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 21:10:36.311237 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:36.311212 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv4d2\" (UniqueName: \"kubernetes.io/projected/fb5f129e-7b15-49fa-9f02-4978dc5e60aa-kube-api-access-pv4d2\") pod \"alertmanager-main-0\" (UID: \"fb5f129e-7b15-49fa-9f02-4978dc5e60aa\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 21:10:36.406614 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:36.406509 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 22 21:10:36.551240 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:36.551205 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 21:10:36.555107 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:10:36.555076 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb5f129e_7b15_49fa_9f02_4978dc5e60aa.slice/crio-e69e7bf7e6e2995ba0c53e56bee014e81d672ccbe820f5eca80764e44ff2c7e3 WatchSource:0}: Error finding container e69e7bf7e6e2995ba0c53e56bee014e81d672ccbe820f5eca80764e44ff2c7e3: Status 404 returned error can't find the container with id e69e7bf7e6e2995ba0c53e56bee014e81d672ccbe820f5eca80764e44ff2c7e3 Jul 22 21:10:36.864186 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:36.864129 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-f896b9d68-ndrf7" Jul 22 21:10:36.864375 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:36.864226 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-f896b9d68-ndrf7" Jul 22 21:10:36.870893 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:36.870866 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-f896b9d68-ndrf7" Jul 22 21:10:37.014746 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:37.014610 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-744cbd9999-72qk5"] Jul 22 21:10:37.018772 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:37.018751 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-744cbd9999-72qk5" Jul 22 21:10:37.021437 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:37.021412 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 22 21:10:37.021870 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:37.021844 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-5jclodm2jln3o\"" Jul 22 21:10:37.022015 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:37.021998 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-wtg2b\"" Jul 22 21:10:37.022092 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:37.022045 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 22 21:10:37.022519 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:37.022492 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 22 21:10:37.022519 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:37.022510 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 22 21:10:37.022670 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:37.022509 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 22 21:10:37.032275 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:37.032221 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-744cbd9999-72qk5"] Jul 22 21:10:37.108976 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:37.108940 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4ba222b4-9746-40b5-bb1b-a710d700b244-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-744cbd9999-72qk5\" (UID: \"4ba222b4-9746-40b5-bb1b-a710d700b244\") " pod="openshift-monitoring/thanos-querier-744cbd9999-72qk5" Jul 22 21:10:37.109474 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:37.109025 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4ba222b4-9746-40b5-bb1b-a710d700b244-metrics-client-ca\") pod \"thanos-querier-744cbd9999-72qk5\" (UID: \"4ba222b4-9746-40b5-bb1b-a710d700b244\") " pod="openshift-monitoring/thanos-querier-744cbd9999-72qk5" Jul 22 21:10:37.109474 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:37.109107 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/4ba222b4-9746-40b5-bb1b-a710d700b244-secret-thanos-querier-tls\") pod \"thanos-querier-744cbd9999-72qk5\" (UID: \"4ba222b4-9746-40b5-bb1b-a710d700b244\") " pod="openshift-monitoring/thanos-querier-744cbd9999-72qk5" Jul 22 21:10:37.109474 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:37.109180 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/4ba222b4-9746-40b5-bb1b-a710d700b244-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-744cbd9999-72qk5\" (UID: \"4ba222b4-9746-40b5-bb1b-a710d700b244\") " pod="openshift-monitoring/thanos-querier-744cbd9999-72qk5" Jul 22 21:10:37.109474 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:37.109203 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjw94\" (UniqueName: \"kubernetes.io/projected/4ba222b4-9746-40b5-bb1b-a710d700b244-kube-api-access-jjw94\") pod \"thanos-querier-744cbd9999-72qk5\" (UID: \"4ba222b4-9746-40b5-bb1b-a710d700b244\") " pod="openshift-monitoring/thanos-querier-744cbd9999-72qk5" Jul 22 21:10:37.109474 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:37.109226 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/4ba222b4-9746-40b5-bb1b-a710d700b244-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-744cbd9999-72qk5\" (UID: \"4ba222b4-9746-40b5-bb1b-a710d700b244\") " pod="openshift-monitoring/thanos-querier-744cbd9999-72qk5" Jul 22 21:10:37.109474 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:37.109272 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4ba222b4-9746-40b5-bb1b-a710d700b244-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-744cbd9999-72qk5\" (UID: \"4ba222b4-9746-40b5-bb1b-a710d700b244\") " pod="openshift-monitoring/thanos-querier-744cbd9999-72qk5" Jul 22 21:10:37.109474 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:37.109345 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4ba222b4-9746-40b5-bb1b-a710d700b244-secret-grpc-tls\") pod \"thanos-querier-744cbd9999-72qk5\" (UID: \"4ba222b4-9746-40b5-bb1b-a710d700b244\") " pod="openshift-monitoring/thanos-querier-744cbd9999-72qk5" Jul 22 21:10:37.210565 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:37.210486 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4ba222b4-9746-40b5-bb1b-a710d700b244-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-744cbd9999-72qk5\" (UID: \"4ba222b4-9746-40b5-bb1b-a710d700b244\") " pod="openshift-monitoring/thanos-querier-744cbd9999-72qk5" Jul 22 21:10:37.210565 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:37.210528 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4ba222b4-9746-40b5-bb1b-a710d700b244-secret-grpc-tls\") pod \"thanos-querier-744cbd9999-72qk5\" (UID: \"4ba222b4-9746-40b5-bb1b-a710d700b244\") " pod="openshift-monitoring/thanos-querier-744cbd9999-72qk5" Jul 22 21:10:37.210565 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:37.210557 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4ba222b4-9746-40b5-bb1b-a710d700b244-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-744cbd9999-72qk5\" (UID: \"4ba222b4-9746-40b5-bb1b-a710d700b244\") " pod="openshift-monitoring/thanos-querier-744cbd9999-72qk5" Jul 22 21:10:37.210821 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:37.210610 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4ba222b4-9746-40b5-bb1b-a710d700b244-metrics-client-ca\") pod \"thanos-querier-744cbd9999-72qk5\" (UID: \"4ba222b4-9746-40b5-bb1b-a710d700b244\") " pod="openshift-monitoring/thanos-querier-744cbd9999-72qk5" Jul 22 21:10:37.210821 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:37.210640 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/4ba222b4-9746-40b5-bb1b-a710d700b244-secret-thanos-querier-tls\") pod \"thanos-querier-744cbd9999-72qk5\" (UID: \"4ba222b4-9746-40b5-bb1b-a710d700b244\") " pod="openshift-monitoring/thanos-querier-744cbd9999-72qk5" Jul 22 21:10:37.210821 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:37.210677 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/4ba222b4-9746-40b5-bb1b-a710d700b244-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-744cbd9999-72qk5\" (UID: \"4ba222b4-9746-40b5-bb1b-a710d700b244\") " pod="openshift-monitoring/thanos-querier-744cbd9999-72qk5" Jul 22 21:10:37.210821 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:37.210700 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjw94\" (UniqueName: \"kubernetes.io/projected/4ba222b4-9746-40b5-bb1b-a710d700b244-kube-api-access-jjw94\") pod \"thanos-querier-744cbd9999-72qk5\" (UID: \"4ba222b4-9746-40b5-bb1b-a710d700b244\") " pod="openshift-monitoring/thanos-querier-744cbd9999-72qk5" Jul 22 21:10:37.210821 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:37.210733 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/4ba222b4-9746-40b5-bb1b-a710d700b244-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-744cbd9999-72qk5\" (UID: \"4ba222b4-9746-40b5-bb1b-a710d700b244\") " pod="openshift-monitoring/thanos-querier-744cbd9999-72qk5" Jul 22 21:10:37.212250 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:37.212048 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4ba222b4-9746-40b5-bb1b-a710d700b244-metrics-client-ca\") pod \"thanos-querier-744cbd9999-72qk5\" (UID: \"4ba222b4-9746-40b5-bb1b-a710d700b244\") " pod="openshift-monitoring/thanos-querier-744cbd9999-72qk5" Jul 22 21:10:37.213878 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:37.213842 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/4ba222b4-9746-40b5-bb1b-a710d700b244-secret-thanos-querier-tls\") pod \"thanos-querier-744cbd9999-72qk5\" (UID: \"4ba222b4-9746-40b5-bb1b-a710d700b244\") " pod="openshift-monitoring/thanos-querier-744cbd9999-72qk5" Jul 22 21:10:37.213995 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:37.213910 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4ba222b4-9746-40b5-bb1b-a710d700b244-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-744cbd9999-72qk5\" (UID: \"4ba222b4-9746-40b5-bb1b-a710d700b244\") " pod="openshift-monitoring/thanos-querier-744cbd9999-72qk5" Jul 22 21:10:37.214055 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:37.213993 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/4ba222b4-9746-40b5-bb1b-a710d700b244-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-744cbd9999-72qk5\" (UID: \"4ba222b4-9746-40b5-bb1b-a710d700b244\") " pod="openshift-monitoring/thanos-querier-744cbd9999-72qk5" Jul 22 21:10:37.214103 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:37.214088 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/4ba222b4-9746-40b5-bb1b-a710d700b244-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-744cbd9999-72qk5\" (UID: \"4ba222b4-9746-40b5-bb1b-a710d700b244\") " pod="openshift-monitoring/thanos-querier-744cbd9999-72qk5" Jul 22 21:10:37.214205 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:37.214156 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4ba222b4-9746-40b5-bb1b-a710d700b244-secret-grpc-tls\") pod \"thanos-querier-744cbd9999-72qk5\" (UID: \"4ba222b4-9746-40b5-bb1b-a710d700b244\") " pod="openshift-monitoring/thanos-querier-744cbd9999-72qk5" Jul 22 21:10:37.214457 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:37.214436 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4ba222b4-9746-40b5-bb1b-a710d700b244-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-744cbd9999-72qk5\" (UID: \"4ba222b4-9746-40b5-bb1b-a710d700b244\") " pod="openshift-monitoring/thanos-querier-744cbd9999-72qk5" Jul 22 21:10:37.218885 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:37.218861 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjw94\" (UniqueName: \"kubernetes.io/projected/4ba222b4-9746-40b5-bb1b-a710d700b244-kube-api-access-jjw94\") pod \"thanos-querier-744cbd9999-72qk5\" (UID: \"4ba222b4-9746-40b5-bb1b-a710d700b244\") " pod="openshift-monitoring/thanos-querier-744cbd9999-72qk5" Jul 22 21:10:37.280579 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:37.280552 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fb5f129e-7b15-49fa-9f02-4978dc5e60aa","Type":"ContainerStarted","Data":"e69e7bf7e6e2995ba0c53e56bee014e81d672ccbe820f5eca80764e44ff2c7e3"} Jul 22 21:10:37.285681 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:37.285655 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-f896b9d68-ndrf7" Jul 22 21:10:37.333665 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:37.333643 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-744cbd9999-72qk5" Jul 22 21:10:37.481514 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:37.481438 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-744cbd9999-72qk5"] Jul 22 21:10:37.491133 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:10:37.491102 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ba222b4_9746_40b5_bb1b_a710d700b244.slice/crio-ff217f948fce0e70a4c40b05a913a85c6566fc7f8f91b627a6ff489a15345096 WatchSource:0}: Error finding container ff217f948fce0e70a4c40b05a913a85c6566fc7f8f91b627a6ff489a15345096: Status 404 returned error can't find the container with id ff217f948fce0e70a4c40b05a913a85c6566fc7f8f91b627a6ff489a15345096 Jul 22 21:10:38.286082 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:38.286042 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-744cbd9999-72qk5" event={"ID":"4ba222b4-9746-40b5-bb1b-a710d700b244","Type":"ContainerStarted","Data":"ff217f948fce0e70a4c40b05a913a85c6566fc7f8f91b627a6ff489a15345096"} Jul 22 21:10:38.287857 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:38.287825 2572 generic.go:356] "Generic (PLEG): container finished" podID="2e6c99e9-55f5-4cf5-b429-23c08bc76207" containerID="b7fd62fbe7970d0bb351a4db3da8c7c24312cec6b13d9e5a1d3b4e39451ade3a" exitCode=0 Jul 22 21:10:38.288007 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:38.287917 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6q24d" event={"ID":"2e6c99e9-55f5-4cf5-b429-23c08bc76207","Type":"ContainerDied","Data":"b7fd62fbe7970d0bb351a4db3da8c7c24312cec6b13d9e5a1d3b4e39451ade3a"} Jul 22 21:10:38.290236 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:38.290198 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ckz8k" event={"ID":"0274094c-5b35-4e27-872c-f8a1df2d7f79","Type":"ContainerStarted","Data":"a8a583a471e5ed1b4a1145ccf9760a7dd5e1d9bdd2472737af93fa40be8eb30f"} Jul 22 21:10:38.329518 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:38.329453 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ckz8k" podStartSLOduration=1.655895519 podStartE2EDuration="3.329432793s" totalImagesPullingTime="1.673537274s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 21:10:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 21:10:38.32905754 +0000 UTC m=+293.443449885" watchObservedRunningTime="2026-07-22 21:10:38.329432793 +0000 UTC m=+293.443825139" Jul 22 21:10:39.295219 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:39.295150 2572 generic.go:356] "Generic (PLEG): container finished" podID="fb5f129e-7b15-49fa-9f02-4978dc5e60aa" containerID="b556b0d347e996528144d7ed431cdb2e701dd47724beebf5cf024a07fa8c8e24" exitCode=0 Jul 22 21:10:39.295219 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:39.295193 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fb5f129e-7b15-49fa-9f02-4978dc5e60aa","Type":"ContainerDied","Data":"b556b0d347e996528144d7ed431cdb2e701dd47724beebf5cf024a07fa8c8e24"} Jul 22 21:10:39.297564 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:39.297537 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6q24d" event={"ID":"2e6c99e9-55f5-4cf5-b429-23c08bc76207","Type":"ContainerStarted","Data":"b225af53da7ae7e62ab8bdea6d97ed0d8360bd900dd94b46ac40d7b461131e6a"} Jul 22 21:10:39.297669 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:39.297583 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6q24d" event={"ID":"2e6c99e9-55f5-4cf5-b429-23c08bc76207","Type":"ContainerStarted","Data":"d77701f6abe4f07a1f5f42347821d089090a42ad3fd80094a235f191cab05028"} Jul 22 21:10:39.341223 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:39.341153 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-6q24d" podStartSLOduration=2.994921256 podStartE2EDuration="4.341135751s" totalImagesPullingTime="1.2592257s" totalInitContainerRuntime="86.988795ms" isStatefulPod=true podCreationTimestamp="2026-07-22 21:10:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 21:10:39.339963581 +0000 UTC m=+294.454355930" watchObservedRunningTime="2026-07-22 21:10:39.341135751 +0000 UTC m=+294.455528094" Jul 22 21:10:39.432802 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:39.432772 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7c67d7698d-8srlj"] Jul 22 21:10:39.436016 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:39.435992 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7c67d7698d-8srlj" Jul 22 21:10:39.438465 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:39.438440 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 22 21:10:39.438760 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:39.438723 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-5fjtw\"" Jul 22 21:10:39.438873 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:39.438779 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 22 21:10:39.438873 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:39.438828 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-movs6m7pvqdq\"" Jul 22 21:10:39.438873 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:39.438783 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 22 21:10:39.438873 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:39.438834 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 22 21:10:39.445536 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:39.445513 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7c67d7698d-8srlj"] Jul 22 21:10:39.532783 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:39.532702 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e2c1979d-ec4c-4267-be6f-cb6aa56036ff-secret-metrics-server-tls\") pod \"metrics-server-7c67d7698d-8srlj\" (UID: \"e2c1979d-ec4c-4267-be6f-cb6aa56036ff\") " pod="openshift-monitoring/metrics-server-7c67d7698d-8srlj" Jul 22 21:10:39.532783 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:39.532773 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2c1979d-ec4c-4267-be6f-cb6aa56036ff-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7c67d7698d-8srlj\" (UID: \"e2c1979d-ec4c-4267-be6f-cb6aa56036ff\") " pod="openshift-monitoring/metrics-server-7c67d7698d-8srlj" Jul 22 21:10:39.533020 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:39.532815 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e2c1979d-ec4c-4267-be6f-cb6aa56036ff-metrics-server-audit-profiles\") pod \"metrics-server-7c67d7698d-8srlj\" (UID: \"e2c1979d-ec4c-4267-be6f-cb6aa56036ff\") " pod="openshift-monitoring/metrics-server-7c67d7698d-8srlj" Jul 22 21:10:39.533020 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:39.532835 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e2c1979d-ec4c-4267-be6f-cb6aa56036ff-client-ca-bundle\") pod \"metrics-server-7c67d7698d-8srlj\" (UID: \"e2c1979d-ec4c-4267-be6f-cb6aa56036ff\") " pod="openshift-monitoring/metrics-server-7c67d7698d-8srlj" Jul 22 21:10:39.533135 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:39.532904 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e2c1979d-ec4c-4267-be6f-cb6aa56036ff-audit-log\") pod \"metrics-server-7c67d7698d-8srlj\" (UID: \"e2c1979d-ec4c-4267-be6f-cb6aa56036ff\") " pod="openshift-monitoring/metrics-server-7c67d7698d-8srlj" Jul 22 21:10:39.537451 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:39.533337 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e2c1979d-ec4c-4267-be6f-cb6aa56036ff-secret-metrics-server-client-certs\") pod \"metrics-server-7c67d7698d-8srlj\" (UID: \"e2c1979d-ec4c-4267-be6f-cb6aa56036ff\") " pod="openshift-monitoring/metrics-server-7c67d7698d-8srlj" Jul 22 21:10:39.537451 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:39.533399 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xp2s9\" (UniqueName: \"kubernetes.io/projected/e2c1979d-ec4c-4267-be6f-cb6aa56036ff-kube-api-access-xp2s9\") pod \"metrics-server-7c67d7698d-8srlj\" (UID: \"e2c1979d-ec4c-4267-be6f-cb6aa56036ff\") " pod="openshift-monitoring/metrics-server-7c67d7698d-8srlj" Jul 22 21:10:39.634404 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:39.634351 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e2c1979d-ec4c-4267-be6f-cb6aa56036ff-secret-metrics-server-tls\") pod \"metrics-server-7c67d7698d-8srlj\" (UID: \"e2c1979d-ec4c-4267-be6f-cb6aa56036ff\") " pod="openshift-monitoring/metrics-server-7c67d7698d-8srlj" Jul 22 21:10:39.634594 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:39.634434 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2c1979d-ec4c-4267-be6f-cb6aa56036ff-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7c67d7698d-8srlj\" (UID: \"e2c1979d-ec4c-4267-be6f-cb6aa56036ff\") " pod="openshift-monitoring/metrics-server-7c67d7698d-8srlj" Jul 22 21:10:39.634594 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:39.634464 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e2c1979d-ec4c-4267-be6f-cb6aa56036ff-metrics-server-audit-profiles\") pod \"metrics-server-7c67d7698d-8srlj\" (UID: \"e2c1979d-ec4c-4267-be6f-cb6aa56036ff\") " pod="openshift-monitoring/metrics-server-7c67d7698d-8srlj" Jul 22 21:10:39.634594 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:39.634490 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e2c1979d-ec4c-4267-be6f-cb6aa56036ff-client-ca-bundle\") pod \"metrics-server-7c67d7698d-8srlj\" (UID: \"e2c1979d-ec4c-4267-be6f-cb6aa56036ff\") " pod="openshift-monitoring/metrics-server-7c67d7698d-8srlj" Jul 22 21:10:39.634594 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:39.634554 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e2c1979d-ec4c-4267-be6f-cb6aa56036ff-audit-log\") pod \"metrics-server-7c67d7698d-8srlj\" (UID: \"e2c1979d-ec4c-4267-be6f-cb6aa56036ff\") " pod="openshift-monitoring/metrics-server-7c67d7698d-8srlj" Jul 22 21:10:39.634808 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:39.634628 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e2c1979d-ec4c-4267-be6f-cb6aa56036ff-secret-metrics-server-client-certs\") pod \"metrics-server-7c67d7698d-8srlj\" (UID: \"e2c1979d-ec4c-4267-be6f-cb6aa56036ff\") " pod="openshift-monitoring/metrics-server-7c67d7698d-8srlj" Jul 22 21:10:39.634808 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:39.634661 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp2s9\" (UniqueName: \"kubernetes.io/projected/e2c1979d-ec4c-4267-be6f-cb6aa56036ff-kube-api-access-xp2s9\") pod \"metrics-server-7c67d7698d-8srlj\" (UID: \"e2c1979d-ec4c-4267-be6f-cb6aa56036ff\") " pod="openshift-monitoring/metrics-server-7c67d7698d-8srlj" Jul 22 21:10:39.635077 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:39.635039 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e2c1979d-ec4c-4267-be6f-cb6aa56036ff-audit-log\") pod \"metrics-server-7c67d7698d-8srlj\" (UID: \"e2c1979d-ec4c-4267-be6f-cb6aa56036ff\") " pod="openshift-monitoring/metrics-server-7c67d7698d-8srlj" Jul 22 21:10:39.635494 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:39.635475 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2c1979d-ec4c-4267-be6f-cb6aa56036ff-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7c67d7698d-8srlj\" (UID: \"e2c1979d-ec4c-4267-be6f-cb6aa56036ff\") " pod="openshift-monitoring/metrics-server-7c67d7698d-8srlj" Jul 22 21:10:39.635897 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:39.635869 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e2c1979d-ec4c-4267-be6f-cb6aa56036ff-metrics-server-audit-profiles\") pod \"metrics-server-7c67d7698d-8srlj\" (UID: \"e2c1979d-ec4c-4267-be6f-cb6aa56036ff\") " pod="openshift-monitoring/metrics-server-7c67d7698d-8srlj" Jul 22 21:10:39.637718 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:39.637694 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e2c1979d-ec4c-4267-be6f-cb6aa56036ff-secret-metrics-server-client-certs\") pod \"metrics-server-7c67d7698d-8srlj\" (UID: \"e2c1979d-ec4c-4267-be6f-cb6aa56036ff\") " pod="openshift-monitoring/metrics-server-7c67d7698d-8srlj" Jul 22 21:10:39.637815 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:39.637694 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e2c1979d-ec4c-4267-be6f-cb6aa56036ff-secret-metrics-server-tls\") pod \"metrics-server-7c67d7698d-8srlj\" (UID: \"e2c1979d-ec4c-4267-be6f-cb6aa56036ff\") " pod="openshift-monitoring/metrics-server-7c67d7698d-8srlj" Jul 22 21:10:39.637815 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:39.637762 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e2c1979d-ec4c-4267-be6f-cb6aa56036ff-client-ca-bundle\") pod \"metrics-server-7c67d7698d-8srlj\" (UID: \"e2c1979d-ec4c-4267-be6f-cb6aa56036ff\") " pod="openshift-monitoring/metrics-server-7c67d7698d-8srlj" Jul 22 21:10:39.643352 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:39.643331 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp2s9\" (UniqueName: \"kubernetes.io/projected/e2c1979d-ec4c-4267-be6f-cb6aa56036ff-kube-api-access-xp2s9\") pod \"metrics-server-7c67d7698d-8srlj\" (UID: \"e2c1979d-ec4c-4267-be6f-cb6aa56036ff\") " pod="openshift-monitoring/metrics-server-7c67d7698d-8srlj" Jul 22 21:10:39.748260 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:39.748224 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7c67d7698d-8srlj" Jul 22 21:10:39.826849 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:39.826813 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-qpftl"] Jul 22 21:10:39.830321 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:39.830293 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-qpftl" Jul 22 21:10:39.832835 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:39.832814 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 22 21:10:39.832992 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:39.832877 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-k8srg\"" Jul 22 21:10:39.838467 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:39.838076 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-qpftl"] Jul 22 21:10:39.936774 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:39.936740 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtcmq\" (UniqueName: \"kubernetes.io/projected/9a55d3ef-6b38-48ed-a6d5-51104b54eb5a-kube-api-access-dtcmq\") pod \"monitoring-plugin-7b7b7bc8bf-qpftl\" (UID: \"9a55d3ef-6b38-48ed-a6d5-51104b54eb5a\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-qpftl" Jul 22 21:10:39.936952 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:39.936808 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9a55d3ef-6b38-48ed-a6d5-51104b54eb5a-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-qpftl\" (UID: \"9a55d3ef-6b38-48ed-a6d5-51104b54eb5a\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-qpftl" Jul 22 21:10:40.037790 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:40.037757 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtcmq\" (UniqueName: \"kubernetes.io/projected/9a55d3ef-6b38-48ed-a6d5-51104b54eb5a-kube-api-access-dtcmq\") pod \"monitoring-plugin-7b7b7bc8bf-qpftl\" (UID: \"9a55d3ef-6b38-48ed-a6d5-51104b54eb5a\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-qpftl" Jul 22 21:10:40.037980 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:40.037816 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9a55d3ef-6b38-48ed-a6d5-51104b54eb5a-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-qpftl\" (UID: \"9a55d3ef-6b38-48ed-a6d5-51104b54eb5a\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-qpftl" Jul 22 21:10:40.040494 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:40.040468 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9a55d3ef-6b38-48ed-a6d5-51104b54eb5a-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-qpftl\" (UID: \"9a55d3ef-6b38-48ed-a6d5-51104b54eb5a\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-qpftl" Jul 22 21:10:40.048904 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:40.048878 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtcmq\" (UniqueName: \"kubernetes.io/projected/9a55d3ef-6b38-48ed-a6d5-51104b54eb5a-kube-api-access-dtcmq\") pod \"monitoring-plugin-7b7b7bc8bf-qpftl\" (UID: \"9a55d3ef-6b38-48ed-a6d5-51104b54eb5a\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-qpftl" Jul 22 21:10:40.142563 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:40.142533 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-qpftl" Jul 22 21:10:40.149494 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:40.149470 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7c67d7698d-8srlj"] Jul 22 21:10:40.151921 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:10:40.151895 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2c1979d_ec4c_4267_be6f_cb6aa56036ff.slice/crio-eecfac88d2b58c1f761fad4a15308a1f4681129a4945df9c4f672f530a57c96e WatchSource:0}: Error finding container eecfac88d2b58c1f761fad4a15308a1f4681129a4945df9c4f672f530a57c96e: Status 404 returned error can't find the container with id eecfac88d2b58c1f761fad4a15308a1f4681129a4945df9c4f672f530a57c96e Jul 22 21:10:40.298955 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:40.298929 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-qpftl"] Jul 22 21:10:40.301330 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:10:40.301301 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a55d3ef_6b38_48ed_a6d5_51104b54eb5a.slice/crio-e9e10c8389c278d80c29115a34f45f492d16c8fa5dc99abcb9ce481936cfbabb WatchSource:0}: Error finding container e9e10c8389c278d80c29115a34f45f492d16c8fa5dc99abcb9ce481936cfbabb: Status 404 returned error can't find the container with id e9e10c8389c278d80c29115a34f45f492d16c8fa5dc99abcb9ce481936cfbabb Jul 22 21:10:40.304137 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:40.304110 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-744cbd9999-72qk5" event={"ID":"4ba222b4-9746-40b5-bb1b-a710d700b244","Type":"ContainerStarted","Data":"bf3edaf5396c5cfc879c4276c273b88d5223b783246be500c4cfb311eb5f642d"} Jul 22 21:10:40.304256 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:40.304205 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-744cbd9999-72qk5" event={"ID":"4ba222b4-9746-40b5-bb1b-a710d700b244","Type":"ContainerStarted","Data":"ecdfcc1de43855a2f0f95f59044833058c70d9ba1b8719e19651ded2c9d535c6"} Jul 22 21:10:40.305569 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:40.305545 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7c67d7698d-8srlj" event={"ID":"e2c1979d-ec4c-4267-be6f-cb6aa56036ff","Type":"ContainerStarted","Data":"eecfac88d2b58c1f761fad4a15308a1f4681129a4945df9c4f672f530a57c96e"} Jul 22 21:10:41.311679 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:41.311638 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-qpftl" event={"ID":"9a55d3ef-6b38-48ed-a6d5-51104b54eb5a","Type":"ContainerStarted","Data":"e9e10c8389c278d80c29115a34f45f492d16c8fa5dc99abcb9ce481936cfbabb"} Jul 22 21:10:41.314374 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:41.314344 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-744cbd9999-72qk5" event={"ID":"4ba222b4-9746-40b5-bb1b-a710d700b244","Type":"ContainerStarted","Data":"9a7e13f36292b692bbf125b8753c93e2dfcd131daf5a1e60bf55dcad87c9a635"} Jul 22 21:10:41.378592 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:10:41.378552 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-lj48g" podUID="8a91c48d-374d-4fc3-9114-d5c88ea18f63" Jul 22 21:10:43.209082 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:43.208967 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-66564cbd6f-4djq8" podUID="bcf6675a-56d1-4606-9a1d-0330d7c6df44" containerName="registry" containerID="cri-o://e7cef73b0bb3c2025e82d6eea0de0fcd8990156596e84ca031f99ae5670e95ed" gracePeriod=30 Jul 22 21:10:43.226423 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:10:43.226391 2572 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcf6675a_56d1_4606_9a1d_0330d7c6df44.slice/crio-e7cef73b0bb3c2025e82d6eea0de0fcd8990156596e84ca031f99ae5670e95ed.scope\": RecentStats: unable to find data in memory cache]" Jul 22 21:10:43.268415 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:43.268385 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-7pnql" Jul 22 21:10:43.324752 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:43.324725 2572 generic.go:356] "Generic (PLEG): container finished" podID="bcf6675a-56d1-4606-9a1d-0330d7c6df44" containerID="e7cef73b0bb3c2025e82d6eea0de0fcd8990156596e84ca031f99ae5670e95ed" exitCode=0 Jul 22 21:10:43.324872 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:43.324804 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-66564cbd6f-4djq8" event={"ID":"bcf6675a-56d1-4606-9a1d-0330d7c6df44","Type":"ContainerDied","Data":"e7cef73b0bb3c2025e82d6eea0de0fcd8990156596e84ca031f99ae5670e95ed"} Jul 22 21:10:43.328779 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:43.328755 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fb5f129e-7b15-49fa-9f02-4978dc5e60aa","Type":"ContainerStarted","Data":"01c19e0f452dc6a90d7e42c79fe38fe6b585a9ea8c839b000fa2939b28763e73"} Jul 22 21:10:43.328896 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:43.328825 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fb5f129e-7b15-49fa-9f02-4978dc5e60aa","Type":"ContainerStarted","Data":"3c93264ff08292125c50471a1281e3b6bac25a76f59f8d5448e08a450c0bd35e"} Jul 22 21:10:43.328896 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:43.328857 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fb5f129e-7b15-49fa-9f02-4978dc5e60aa","Type":"ContainerStarted","Data":"c2f1f79d1b1a75b9a79a2ddc67123f2842b82864430d1922e6f0fc4e1f073d66"} Jul 22 21:10:43.328896 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:43.328883 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fb5f129e-7b15-49fa-9f02-4978dc5e60aa","Type":"ContainerStarted","Data":"9f3f0cb1e6d6358a070b364c742be14f7f760ef8b19053e03542fb44dc9957ce"} Jul 22 21:10:43.329042 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:43.328914 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fb5f129e-7b15-49fa-9f02-4978dc5e60aa","Type":"ContainerStarted","Data":"fba6a4055ada9a4b5007c1d701c405fa3c6071b5b38e1c256bada8111c8ac589"} Jul 22 21:10:43.329042 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:43.328943 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fb5f129e-7b15-49fa-9f02-4978dc5e60aa","Type":"ContainerStarted","Data":"7c4aecb3c740e48eb8f153301ced1b582b22114e1aa88e5894a1e76ce5fff3c2"} Jul 22 21:10:43.330459 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:43.330437 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-qpftl" event={"ID":"9a55d3ef-6b38-48ed-a6d5-51104b54eb5a","Type":"ContainerStarted","Data":"37aa2248d6a96ad1ef4212cd271065173ab1747a9ae720a3dca05ec1dafddd48"} Jul 22 21:10:43.330578 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:43.330554 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-qpftl" Jul 22 21:10:43.334577 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:43.334556 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-744cbd9999-72qk5" event={"ID":"4ba222b4-9746-40b5-bb1b-a710d700b244","Type":"ContainerStarted","Data":"c66cbcac2f56059ce35b7a809778b033c722c997aaee6117928756a11ac9bed9"} Jul 22 21:10:43.334694 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:43.334618 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-744cbd9999-72qk5" event={"ID":"4ba222b4-9746-40b5-bb1b-a710d700b244","Type":"ContainerStarted","Data":"ddc4e1131bad83875821abf3a351d2938a9fde26516cc35f5e18d841f22aa8fe"} Jul 22 21:10:43.334694 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:43.334658 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-744cbd9999-72qk5" event={"ID":"4ba222b4-9746-40b5-bb1b-a710d700b244","Type":"ContainerStarted","Data":"0043f7b0a5a2377065016d501ca4d65a40639fb3e85911ef19bb71758725e994"} Jul 22 21:10:43.334810 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:43.334796 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-744cbd9999-72qk5" Jul 22 21:10:43.336540 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:43.336518 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7c67d7698d-8srlj" event={"ID":"e2c1979d-ec4c-4267-be6f-cb6aa56036ff","Type":"ContainerStarted","Data":"bef6a8f4ca9a43ebd1183495b516d54d8fd69c7688874b39ba358fa87454aac8"} Jul 22 21:10:43.337334 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:43.337315 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-qpftl" Jul 22 21:10:43.354011 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:43.353955 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.014004437 podStartE2EDuration="7.353931437s" totalImagesPullingTime="5.322120913s" totalInitContainerRuntime="17.806087ms" isStatefulPod=false podCreationTimestamp="2026-07-22 21:10:36 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 21:10:43.353239035 +0000 UTC m=+298.467631405" watchObservedRunningTime="2026-07-22 21:10:43.353931437 +0000 UTC m=+298.468323781" Jul 22 21:10:43.379316 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:43.378937 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-744cbd9999-72qk5" podStartSLOduration=2.458499712 podStartE2EDuration="7.378917365s" totalImagesPullingTime="4.920417653s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 21:10:36 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 21:10:43.377350749 +0000 UTC m=+298.491743121" watchObservedRunningTime="2026-07-22 21:10:43.378917365 +0000 UTC m=+298.493309711" Jul 22 21:10:43.394840 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:43.394785 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7c67d7698d-8srlj" podStartSLOduration=1.90179587 podStartE2EDuration="4.39476794s" totalImagesPullingTime="2.49297207s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 21:10:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 21:10:43.393549192 +0000 UTC m=+298.507941551" watchObservedRunningTime="2026-07-22 21:10:43.39476794 +0000 UTC m=+298.509160328" Jul 22 21:10:43.409433 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:43.409383 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-qpftl" podStartSLOduration=2.064334107 podStartE2EDuration="4.409367374s" totalImagesPullingTime="2.345033267s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 21:10:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 21:10:43.407593709 +0000 UTC m=+298.521986065" watchObservedRunningTime="2026-07-22 21:10:43.409367374 +0000 UTC m=+298.523759695" Jul 22 21:10:43.454898 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:43.454875 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66564cbd6f-4djq8" Jul 22 21:10:43.573883 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:43.573853 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t9k25\" (UniqueName: \"kubernetes.io/projected/bcf6675a-56d1-4606-9a1d-0330d7c6df44-kube-api-access-t9k25\") pod \"bcf6675a-56d1-4606-9a1d-0330d7c6df44\" (UID: \"bcf6675a-56d1-4606-9a1d-0330d7c6df44\") " Jul 22 21:10:43.573883 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:43.573888 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bcf6675a-56d1-4606-9a1d-0330d7c6df44-ca-trust-extracted\") pod \"bcf6675a-56d1-4606-9a1d-0330d7c6df44\" (UID: \"bcf6675a-56d1-4606-9a1d-0330d7c6df44\") " Jul 22 21:10:43.574057 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:43.573934 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bcf6675a-56d1-4606-9a1d-0330d7c6df44-trusted-ca\") pod \"bcf6675a-56d1-4606-9a1d-0330d7c6df44\" (UID: \"bcf6675a-56d1-4606-9a1d-0330d7c6df44\") " Jul 22 21:10:43.574057 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:43.573963 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bcf6675a-56d1-4606-9a1d-0330d7c6df44-registry-certificates\") pod \"bcf6675a-56d1-4606-9a1d-0330d7c6df44\" (UID: \"bcf6675a-56d1-4606-9a1d-0330d7c6df44\") " Jul 22 21:10:43.574057 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:43.573984 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bcf6675a-56d1-4606-9a1d-0330d7c6df44-installation-pull-secrets\") pod \"bcf6675a-56d1-4606-9a1d-0330d7c6df44\" (UID: \"bcf6675a-56d1-4606-9a1d-0330d7c6df44\") " Jul 22 21:10:43.574057 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:43.574011 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bcf6675a-56d1-4606-9a1d-0330d7c6df44-registry-tls\") pod \"bcf6675a-56d1-4606-9a1d-0330d7c6df44\" (UID: \"bcf6675a-56d1-4606-9a1d-0330d7c6df44\") " Jul 22 21:10:43.574057 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:43.574041 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bcf6675a-56d1-4606-9a1d-0330d7c6df44-bound-sa-token\") pod \"bcf6675a-56d1-4606-9a1d-0330d7c6df44\" (UID: \"bcf6675a-56d1-4606-9a1d-0330d7c6df44\") " Jul 22 21:10:43.574309 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:43.574095 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bcf6675a-56d1-4606-9a1d-0330d7c6df44-image-registry-private-configuration\") pod \"bcf6675a-56d1-4606-9a1d-0330d7c6df44\" (UID: \"bcf6675a-56d1-4606-9a1d-0330d7c6df44\") " Jul 22 21:10:43.574503 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:43.574472 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcf6675a-56d1-4606-9a1d-0330d7c6df44-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "bcf6675a-56d1-4606-9a1d-0330d7c6df44" (UID: "bcf6675a-56d1-4606-9a1d-0330d7c6df44"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 21:10:43.574503 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:43.574494 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcf6675a-56d1-4606-9a1d-0330d7c6df44-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "bcf6675a-56d1-4606-9a1d-0330d7c6df44" (UID: "bcf6675a-56d1-4606-9a1d-0330d7c6df44"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 21:10:43.576472 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:43.576444 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcf6675a-56d1-4606-9a1d-0330d7c6df44-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "bcf6675a-56d1-4606-9a1d-0330d7c6df44" (UID: "bcf6675a-56d1-4606-9a1d-0330d7c6df44"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 21:10:43.576592 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:43.576569 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcf6675a-56d1-4606-9a1d-0330d7c6df44-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "bcf6675a-56d1-4606-9a1d-0330d7c6df44" (UID: "bcf6675a-56d1-4606-9a1d-0330d7c6df44"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 21:10:43.576656 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:43.576639 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcf6675a-56d1-4606-9a1d-0330d7c6df44-kube-api-access-t9k25" (OuterVolumeSpecName: "kube-api-access-t9k25") pod "bcf6675a-56d1-4606-9a1d-0330d7c6df44" (UID: "bcf6675a-56d1-4606-9a1d-0330d7c6df44"). InnerVolumeSpecName "kube-api-access-t9k25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 21:10:43.576883 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:43.576860 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcf6675a-56d1-4606-9a1d-0330d7c6df44-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "bcf6675a-56d1-4606-9a1d-0330d7c6df44" (UID: "bcf6675a-56d1-4606-9a1d-0330d7c6df44"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 21:10:43.576883 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:43.576875 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcf6675a-56d1-4606-9a1d-0330d7c6df44-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "bcf6675a-56d1-4606-9a1d-0330d7c6df44" (UID: "bcf6675a-56d1-4606-9a1d-0330d7c6df44"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 21:10:43.582418 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:43.582396 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcf6675a-56d1-4606-9a1d-0330d7c6df44-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "bcf6675a-56d1-4606-9a1d-0330d7c6df44" (UID: "bcf6675a-56d1-4606-9a1d-0330d7c6df44"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 21:10:43.675510 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:43.675481 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9k25\" (UniqueName: \"kubernetes.io/projected/bcf6675a-56d1-4606-9a1d-0330d7c6df44-kube-api-access-t9k25\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:10:43.675510 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:43.675508 2572 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bcf6675a-56d1-4606-9a1d-0330d7c6df44-ca-trust-extracted\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:10:43.675650 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:43.675517 2572 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bcf6675a-56d1-4606-9a1d-0330d7c6df44-trusted-ca\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:10:43.675650 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:43.675528 2572 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bcf6675a-56d1-4606-9a1d-0330d7c6df44-registry-certificates\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:10:43.675650 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:43.675538 2572 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bcf6675a-56d1-4606-9a1d-0330d7c6df44-installation-pull-secrets\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:10:43.675650 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:43.675547 2572 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bcf6675a-56d1-4606-9a1d-0330d7c6df44-registry-tls\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:10:43.675650 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:43.675556 2572 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bcf6675a-56d1-4606-9a1d-0330d7c6df44-bound-sa-token\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:10:43.675650 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:43.675565 2572 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bcf6675a-56d1-4606-9a1d-0330d7c6df44-image-registry-private-configuration\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:10:44.342017 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:44.341987 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66564cbd6f-4djq8" Jul 22 21:10:44.342478 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:44.342054 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-66564cbd6f-4djq8" event={"ID":"bcf6675a-56d1-4606-9a1d-0330d7c6df44","Type":"ContainerDied","Data":"f03f8c784d7650458cc86aaf2114a4187c32567ba13addf38e7205d1904110aa"} Jul 22 21:10:44.342478 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:44.342113 2572 scope.go:117] "RemoveContainer" containerID="e7cef73b0bb3c2025e82d6eea0de0fcd8990156596e84ca031f99ae5670e95ed" Jul 22 21:10:44.362126 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:44.362102 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-66564cbd6f-4djq8"] Jul 22 21:10:44.365705 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:44.365684 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-66564cbd6f-4djq8"] Jul 22 21:10:44.807007 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:44.806929 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-746784ffcc-88dk7"] Jul 22 21:10:44.807330 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:44.807313 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcf6675a-56d1-4606-9a1d-0330d7c6df44" containerName="registry" Jul 22 21:10:44.807398 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:44.807335 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcf6675a-56d1-4606-9a1d-0330d7c6df44" containerName="registry" Jul 22 21:10:44.807439 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:44.807405 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="bcf6675a-56d1-4606-9a1d-0330d7c6df44" containerName="registry" Jul 22 21:10:44.812434 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:44.812414 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-746784ffcc-88dk7" Jul 22 21:10:44.819553 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:44.819529 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 22 21:10:44.821668 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:44.821649 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-746784ffcc-88dk7"] Jul 22 21:10:44.988467 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:44.988429 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v944q\" (UniqueName: \"kubernetes.io/projected/57159d41-02c2-4d57-bbdc-67b79f18766d-kube-api-access-v944q\") pod \"console-746784ffcc-88dk7\" (UID: \"57159d41-02c2-4d57-bbdc-67b79f18766d\") " pod="openshift-console/console-746784ffcc-88dk7" Jul 22 21:10:44.988467 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:44.988467 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57159d41-02c2-4d57-bbdc-67b79f18766d-trusted-ca-bundle\") pod \"console-746784ffcc-88dk7\" (UID: \"57159d41-02c2-4d57-bbdc-67b79f18766d\") " pod="openshift-console/console-746784ffcc-88dk7" Jul 22 21:10:44.988663 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:44.988486 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/57159d41-02c2-4d57-bbdc-67b79f18766d-console-config\") pod \"console-746784ffcc-88dk7\" (UID: \"57159d41-02c2-4d57-bbdc-67b79f18766d\") " pod="openshift-console/console-746784ffcc-88dk7" Jul 22 21:10:44.988663 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:44.988553 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/57159d41-02c2-4d57-bbdc-67b79f18766d-console-serving-cert\") pod \"console-746784ffcc-88dk7\" (UID: \"57159d41-02c2-4d57-bbdc-67b79f18766d\") " pod="openshift-console/console-746784ffcc-88dk7" Jul 22 21:10:44.988663 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:44.988593 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/57159d41-02c2-4d57-bbdc-67b79f18766d-console-oauth-config\") pod \"console-746784ffcc-88dk7\" (UID: \"57159d41-02c2-4d57-bbdc-67b79f18766d\") " pod="openshift-console/console-746784ffcc-88dk7" Jul 22 21:10:44.988663 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:44.988658 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/57159d41-02c2-4d57-bbdc-67b79f18766d-service-ca\") pod \"console-746784ffcc-88dk7\" (UID: \"57159d41-02c2-4d57-bbdc-67b79f18766d\") " pod="openshift-console/console-746784ffcc-88dk7" Jul 22 21:10:44.988782 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:44.988690 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/57159d41-02c2-4d57-bbdc-67b79f18766d-oauth-serving-cert\") pod \"console-746784ffcc-88dk7\" (UID: \"57159d41-02c2-4d57-bbdc-67b79f18766d\") " pod="openshift-console/console-746784ffcc-88dk7" Jul 22 21:10:45.089433 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:45.089399 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/57159d41-02c2-4d57-bbdc-67b79f18766d-service-ca\") pod \"console-746784ffcc-88dk7\" (UID: \"57159d41-02c2-4d57-bbdc-67b79f18766d\") " pod="openshift-console/console-746784ffcc-88dk7" Jul 22 21:10:45.089562 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:45.089441 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/57159d41-02c2-4d57-bbdc-67b79f18766d-oauth-serving-cert\") pod \"console-746784ffcc-88dk7\" (UID: \"57159d41-02c2-4d57-bbdc-67b79f18766d\") " pod="openshift-console/console-746784ffcc-88dk7" Jul 22 21:10:45.089562 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:45.089480 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v944q\" (UniqueName: \"kubernetes.io/projected/57159d41-02c2-4d57-bbdc-67b79f18766d-kube-api-access-v944q\") pod \"console-746784ffcc-88dk7\" (UID: \"57159d41-02c2-4d57-bbdc-67b79f18766d\") " pod="openshift-console/console-746784ffcc-88dk7" Jul 22 21:10:45.089562 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:45.089499 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57159d41-02c2-4d57-bbdc-67b79f18766d-trusted-ca-bundle\") pod \"console-746784ffcc-88dk7\" (UID: \"57159d41-02c2-4d57-bbdc-67b79f18766d\") " pod="openshift-console/console-746784ffcc-88dk7" Jul 22 21:10:45.089562 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:45.089515 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/57159d41-02c2-4d57-bbdc-67b79f18766d-console-config\") pod \"console-746784ffcc-88dk7\" (UID: \"57159d41-02c2-4d57-bbdc-67b79f18766d\") " pod="openshift-console/console-746784ffcc-88dk7" Jul 22 21:10:45.089562 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:45.089531 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/57159d41-02c2-4d57-bbdc-67b79f18766d-console-serving-cert\") pod \"console-746784ffcc-88dk7\" (UID: \"57159d41-02c2-4d57-bbdc-67b79f18766d\") " pod="openshift-console/console-746784ffcc-88dk7" Jul 22 21:10:45.089562 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:45.089545 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/57159d41-02c2-4d57-bbdc-67b79f18766d-console-oauth-config\") pod \"console-746784ffcc-88dk7\" (UID: \"57159d41-02c2-4d57-bbdc-67b79f18766d\") " pod="openshift-console/console-746784ffcc-88dk7" Jul 22 21:10:45.090319 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:45.090231 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/57159d41-02c2-4d57-bbdc-67b79f18766d-service-ca\") pod \"console-746784ffcc-88dk7\" (UID: \"57159d41-02c2-4d57-bbdc-67b79f18766d\") " pod="openshift-console/console-746784ffcc-88dk7" Jul 22 21:10:45.090319 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:45.090284 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/57159d41-02c2-4d57-bbdc-67b79f18766d-console-config\") pod \"console-746784ffcc-88dk7\" (UID: \"57159d41-02c2-4d57-bbdc-67b79f18766d\") " pod="openshift-console/console-746784ffcc-88dk7" Jul 22 21:10:45.090319 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:45.090402 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/57159d41-02c2-4d57-bbdc-67b79f18766d-oauth-serving-cert\") pod \"console-746784ffcc-88dk7\" (UID: \"57159d41-02c2-4d57-bbdc-67b79f18766d\") " pod="openshift-console/console-746784ffcc-88dk7" Jul 22 21:10:45.090638 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:45.090461 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57159d41-02c2-4d57-bbdc-67b79f18766d-trusted-ca-bundle\") pod \"console-746784ffcc-88dk7\" (UID: \"57159d41-02c2-4d57-bbdc-67b79f18766d\") " pod="openshift-console/console-746784ffcc-88dk7" Jul 22 21:10:45.092147 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:45.092122 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/57159d41-02c2-4d57-bbdc-67b79f18766d-console-serving-cert\") pod \"console-746784ffcc-88dk7\" (UID: \"57159d41-02c2-4d57-bbdc-67b79f18766d\") " pod="openshift-console/console-746784ffcc-88dk7" Jul 22 21:10:45.092247 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:45.092211 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/57159d41-02c2-4d57-bbdc-67b79f18766d-console-oauth-config\") pod \"console-746784ffcc-88dk7\" (UID: \"57159d41-02c2-4d57-bbdc-67b79f18766d\") " pod="openshift-console/console-746784ffcc-88dk7" Jul 22 21:10:45.098356 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:45.098334 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v944q\" (UniqueName: \"kubernetes.io/projected/57159d41-02c2-4d57-bbdc-67b79f18766d-kube-api-access-v944q\") pod \"console-746784ffcc-88dk7\" (UID: \"57159d41-02c2-4d57-bbdc-67b79f18766d\") " pod="openshift-console/console-746784ffcc-88dk7" Jul 22 21:10:45.122363 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:45.122323 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-746784ffcc-88dk7" Jul 22 21:10:45.267743 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:45.267718 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-746784ffcc-88dk7"] Jul 22 21:10:45.270364 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:10:45.270326 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57159d41_02c2_4d57_bbdc_67b79f18766d.slice/crio-a9bf993104f1fbe599fbf3cee849ff8e21f6e8e1a9f1185b459421459a47b804 WatchSource:0}: Error finding container a9bf993104f1fbe599fbf3cee849ff8e21f6e8e1a9f1185b459421459a47b804: Status 404 returned error can't find the container with id a9bf993104f1fbe599fbf3cee849ff8e21f6e8e1a9f1185b459421459a47b804 Jul 22 21:10:45.277864 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:45.277802 2572 kubelet.go:1648] "Image garbage collection succeeded" Jul 22 21:10:45.348651 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:45.348555 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-746784ffcc-88dk7" event={"ID":"57159d41-02c2-4d57-bbdc-67b79f18766d","Type":"ContainerStarted","Data":"2492e89c444f8aa0c7c48293fb67a285ecf6daa840a0d65ded48791a25a1cd26"} Jul 22 21:10:45.348651 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:45.348617 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-746784ffcc-88dk7" event={"ID":"57159d41-02c2-4d57-bbdc-67b79f18766d","Type":"ContainerStarted","Data":"a9bf993104f1fbe599fbf3cee849ff8e21f6e8e1a9f1185b459421459a47b804"} Jul 22 21:10:45.366650 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:45.366600 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-746784ffcc-88dk7" podStartSLOduration=1.366583788 podStartE2EDuration="1.366583788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 21:10:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 21:10:45.365089968 +0000 UTC m=+300.479482336" watchObservedRunningTime="2026-07-22 21:10:45.366583788 +0000 UTC m=+300.480976132" Jul 22 21:10:45.387400 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:45.385096 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bcf6675a-56d1-4606-9a1d-0330d7c6df44" path="/var/lib/kubelet/pods/bcf6675a-56d1-4606-9a1d-0330d7c6df44/volumes" Jul 22 21:10:49.349793 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:49.349767 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-744cbd9999-72qk5" Jul 22 21:10:54.376954 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:54.376914 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lj48g" Jul 22 21:10:55.123034 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:55.122993 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-746784ffcc-88dk7" Jul 22 21:10:55.123260 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:55.123080 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-746784ffcc-88dk7" Jul 22 21:10:55.128586 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:55.128564 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-746784ffcc-88dk7" Jul 22 21:10:55.391984 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:55.391910 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-746784ffcc-88dk7" Jul 22 21:10:55.437790 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:55.437753 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-f896b9d68-ndrf7"] Jul 22 21:10:59.749003 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:59.748957 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7c67d7698d-8srlj" Jul 22 21:10:59.749418 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:10:59.749022 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7c67d7698d-8srlj" Jul 22 21:11:19.754648 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:19.754614 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7c67d7698d-8srlj" Jul 22 21:11:19.758569 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:19.758541 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7c67d7698d-8srlj" Jul 22 21:11:20.463239 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:20.463182 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-f896b9d68-ndrf7" podUID="e591af84-a844-4ba8-8855-3dc41a45b70b" containerName="console" containerID="cri-o://fd6fbbf02fd7b3c0035a449c244bcdd2cda1283c819c3b5b3aa6ec488834b13a" gracePeriod=15 Jul 22 21:11:20.705559 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:20.705537 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f896b9d68-ndrf7" Jul 22 21:11:20.808083 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:20.808060 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x5l5v\" (UniqueName: \"kubernetes.io/projected/e591af84-a844-4ba8-8855-3dc41a45b70b-kube-api-access-x5l5v\") pod \"e591af84-a844-4ba8-8855-3dc41a45b70b\" (UID: \"e591af84-a844-4ba8-8855-3dc41a45b70b\") " Jul 22 21:11:20.808531 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:20.808133 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e591af84-a844-4ba8-8855-3dc41a45b70b-oauth-serving-cert\") pod \"e591af84-a844-4ba8-8855-3dc41a45b70b\" (UID: \"e591af84-a844-4ba8-8855-3dc41a45b70b\") " Jul 22 21:11:20.808531 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:20.808193 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e591af84-a844-4ba8-8855-3dc41a45b70b-console-config\") pod \"e591af84-a844-4ba8-8855-3dc41a45b70b\" (UID: \"e591af84-a844-4ba8-8855-3dc41a45b70b\") " Jul 22 21:11:20.808531 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:20.808216 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e591af84-a844-4ba8-8855-3dc41a45b70b-service-ca\") pod \"e591af84-a844-4ba8-8855-3dc41a45b70b\" (UID: \"e591af84-a844-4ba8-8855-3dc41a45b70b\") " Jul 22 21:11:20.808531 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:20.808244 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e591af84-a844-4ba8-8855-3dc41a45b70b-console-serving-cert\") pod \"e591af84-a844-4ba8-8855-3dc41a45b70b\" (UID: \"e591af84-a844-4ba8-8855-3dc41a45b70b\") " Jul 22 21:11:20.808531 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:20.808285 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e591af84-a844-4ba8-8855-3dc41a45b70b-console-oauth-config\") pod \"e591af84-a844-4ba8-8855-3dc41a45b70b\" (UID: \"e591af84-a844-4ba8-8855-3dc41a45b70b\") " Jul 22 21:11:20.808750 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:20.808563 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e591af84-a844-4ba8-8855-3dc41a45b70b-console-config" (OuterVolumeSpecName: "console-config") pod "e591af84-a844-4ba8-8855-3dc41a45b70b" (UID: "e591af84-a844-4ba8-8855-3dc41a45b70b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 21:11:20.808750 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:20.808559 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e591af84-a844-4ba8-8855-3dc41a45b70b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "e591af84-a844-4ba8-8855-3dc41a45b70b" (UID: "e591af84-a844-4ba8-8855-3dc41a45b70b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 21:11:20.808750 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:20.808598 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e591af84-a844-4ba8-8855-3dc41a45b70b-service-ca" (OuterVolumeSpecName: "service-ca") pod "e591af84-a844-4ba8-8855-3dc41a45b70b" (UID: "e591af84-a844-4ba8-8855-3dc41a45b70b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 21:11:20.810631 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:20.810597 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e591af84-a844-4ba8-8855-3dc41a45b70b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "e591af84-a844-4ba8-8855-3dc41a45b70b" (UID: "e591af84-a844-4ba8-8855-3dc41a45b70b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 21:11:20.810727 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:20.810627 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e591af84-a844-4ba8-8855-3dc41a45b70b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "e591af84-a844-4ba8-8855-3dc41a45b70b" (UID: "e591af84-a844-4ba8-8855-3dc41a45b70b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 21:11:20.810727 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:20.810687 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e591af84-a844-4ba8-8855-3dc41a45b70b-kube-api-access-x5l5v" (OuterVolumeSpecName: "kube-api-access-x5l5v") pod "e591af84-a844-4ba8-8855-3dc41a45b70b" (UID: "e591af84-a844-4ba8-8855-3dc41a45b70b"). InnerVolumeSpecName "kube-api-access-x5l5v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 21:11:20.909406 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:20.909371 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5l5v\" (UniqueName: \"kubernetes.io/projected/e591af84-a844-4ba8-8855-3dc41a45b70b-kube-api-access-x5l5v\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:11:20.909406 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:20.909399 2572 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e591af84-a844-4ba8-8855-3dc41a45b70b-oauth-serving-cert\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:11:20.909406 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:20.909410 2572 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e591af84-a844-4ba8-8855-3dc41a45b70b-console-config\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:11:20.909633 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:20.909419 2572 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e591af84-a844-4ba8-8855-3dc41a45b70b-service-ca\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:11:20.909633 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:20.909427 2572 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e591af84-a844-4ba8-8855-3dc41a45b70b-console-serving-cert\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:11:20.909633 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:20.909435 2572 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e591af84-a844-4ba8-8855-3dc41a45b70b-console-oauth-config\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:11:21.484014 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:21.483979 2572 generic.go:356] "Generic (PLEG): container finished" podID="e591af84-a844-4ba8-8855-3dc41a45b70b" containerID="fd6fbbf02fd7b3c0035a449c244bcdd2cda1283c819c3b5b3aa6ec488834b13a" exitCode=0 Jul 22 21:11:21.484151 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:21.484012 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f896b9d68-ndrf7" event={"ID":"e591af84-a844-4ba8-8855-3dc41a45b70b","Type":"ContainerDied","Data":"fd6fbbf02fd7b3c0035a449c244bcdd2cda1283c819c3b5b3aa6ec488834b13a"} Jul 22 21:11:21.484151 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:21.484056 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f896b9d68-ndrf7" event={"ID":"e591af84-a844-4ba8-8855-3dc41a45b70b","Type":"ContainerDied","Data":"e7fe55b014d8883155596f442a6b27c40de87cb41bcacc820219d759fab2b603"} Jul 22 21:11:21.484151 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:21.484082 2572 scope.go:117] "RemoveContainer" containerID="fd6fbbf02fd7b3c0035a449c244bcdd2cda1283c819c3b5b3aa6ec488834b13a" Jul 22 21:11:21.484151 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:21.484023 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f896b9d68-ndrf7" Jul 22 21:11:21.492108 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:21.492092 2572 scope.go:117] "RemoveContainer" containerID="fd6fbbf02fd7b3c0035a449c244bcdd2cda1283c819c3b5b3aa6ec488834b13a" Jul 22 21:11:21.492371 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:11:21.492349 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd6fbbf02fd7b3c0035a449c244bcdd2cda1283c819c3b5b3aa6ec488834b13a\": container with ID starting with fd6fbbf02fd7b3c0035a449c244bcdd2cda1283c819c3b5b3aa6ec488834b13a not found: ID does not exist" containerID="fd6fbbf02fd7b3c0035a449c244bcdd2cda1283c819c3b5b3aa6ec488834b13a" Jul 22 21:11:21.492458 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:21.492378 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd6fbbf02fd7b3c0035a449c244bcdd2cda1283c819c3b5b3aa6ec488834b13a"} err="failed to get container status \"fd6fbbf02fd7b3c0035a449c244bcdd2cda1283c819c3b5b3aa6ec488834b13a\": rpc error: code = NotFound desc = could not find container \"fd6fbbf02fd7b3c0035a449c244bcdd2cda1283c819c3b5b3aa6ec488834b13a\": container with ID starting with fd6fbbf02fd7b3c0035a449c244bcdd2cda1283c819c3b5b3aa6ec488834b13a not found: ID does not exist" Jul 22 21:11:21.504135 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:21.504112 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-f896b9d68-ndrf7"] Jul 22 21:11:21.509950 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:21.509931 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-f896b9d68-ndrf7"] Jul 22 21:11:23.382182 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:23.382131 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e591af84-a844-4ba8-8855-3dc41a45b70b" path="/var/lib/kubelet/pods/e591af84-a844-4ba8-8855-3dc41a45b70b/volumes" Jul 22 21:11:55.381922 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:55.381868 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 21:11:55.382506 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:55.382309 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="fb5f129e-7b15-49fa-9f02-4978dc5e60aa" containerName="alertmanager" containerID="cri-o://7c4aecb3c740e48eb8f153301ced1b582b22114e1aa88e5894a1e76ce5fff3c2" gracePeriod=120 Jul 22 21:11:55.382506 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:55.382388 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="fb5f129e-7b15-49fa-9f02-4978dc5e60aa" containerName="kube-rbac-proxy-metric" containerID="cri-o://3c93264ff08292125c50471a1281e3b6bac25a76f59f8d5448e08a450c0bd35e" gracePeriod=120 Jul 22 21:11:55.382506 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:55.382444 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="fb5f129e-7b15-49fa-9f02-4978dc5e60aa" containerName="kube-rbac-proxy" containerID="cri-o://c2f1f79d1b1a75b9a79a2ddc67123f2842b82864430d1922e6f0fc4e1f073d66" gracePeriod=120 Jul 22 21:11:55.382506 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:55.382459 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="fb5f129e-7b15-49fa-9f02-4978dc5e60aa" containerName="prom-label-proxy" containerID="cri-o://01c19e0f452dc6a90d7e42c79fe38fe6b585a9ea8c839b000fa2939b28763e73" gracePeriod=120 Jul 22 21:11:55.382738 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:55.382511 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="fb5f129e-7b15-49fa-9f02-4978dc5e60aa" containerName="kube-rbac-proxy-web" containerID="cri-o://9f3f0cb1e6d6358a070b364c742be14f7f760ef8b19053e03542fb44dc9957ce" gracePeriod=120 Jul 22 21:11:55.382738 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:55.382476 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="fb5f129e-7b15-49fa-9f02-4978dc5e60aa" containerName="config-reloader" containerID="cri-o://fba6a4055ada9a4b5007c1d701c405fa3c6071b5b38e1c256bada8111c8ac589" gracePeriod=120 Jul 22 21:11:55.617494 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:55.617464 2572 generic.go:356] "Generic (PLEG): container finished" podID="fb5f129e-7b15-49fa-9f02-4978dc5e60aa" containerID="01c19e0f452dc6a90d7e42c79fe38fe6b585a9ea8c839b000fa2939b28763e73" exitCode=0 Jul 22 21:11:55.617494 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:55.617487 2572 generic.go:356] "Generic (PLEG): container finished" podID="fb5f129e-7b15-49fa-9f02-4978dc5e60aa" containerID="c2f1f79d1b1a75b9a79a2ddc67123f2842b82864430d1922e6f0fc4e1f073d66" exitCode=0 Jul 22 21:11:55.617494 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:55.617493 2572 generic.go:356] "Generic (PLEG): container finished" podID="fb5f129e-7b15-49fa-9f02-4978dc5e60aa" containerID="fba6a4055ada9a4b5007c1d701c405fa3c6071b5b38e1c256bada8111c8ac589" exitCode=0 Jul 22 21:11:55.617494 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:55.617498 2572 generic.go:356] "Generic (PLEG): container finished" podID="fb5f129e-7b15-49fa-9f02-4978dc5e60aa" containerID="7c4aecb3c740e48eb8f153301ced1b582b22114e1aa88e5894a1e76ce5fff3c2" exitCode=0 Jul 22 21:11:55.617815 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:55.617549 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fb5f129e-7b15-49fa-9f02-4978dc5e60aa","Type":"ContainerDied","Data":"01c19e0f452dc6a90d7e42c79fe38fe6b585a9ea8c839b000fa2939b28763e73"} Jul 22 21:11:55.617815 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:55.617628 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fb5f129e-7b15-49fa-9f02-4978dc5e60aa","Type":"ContainerDied","Data":"c2f1f79d1b1a75b9a79a2ddc67123f2842b82864430d1922e6f0fc4e1f073d66"} Jul 22 21:11:55.617815 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:55.617669 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fb5f129e-7b15-49fa-9f02-4978dc5e60aa","Type":"ContainerDied","Data":"fba6a4055ada9a4b5007c1d701c405fa3c6071b5b38e1c256bada8111c8ac589"} Jul 22 21:11:55.617815 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:55.617706 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fb5f129e-7b15-49fa-9f02-4978dc5e60aa","Type":"ContainerDied","Data":"7c4aecb3c740e48eb8f153301ced1b582b22114e1aa88e5894a1e76ce5fff3c2"} Jul 22 21:11:56.626958 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:56.626932 2572 generic.go:356] "Generic (PLEG): container finished" podID="fb5f129e-7b15-49fa-9f02-4978dc5e60aa" containerID="3c93264ff08292125c50471a1281e3b6bac25a76f59f8d5448e08a450c0bd35e" exitCode=0 Jul 22 21:11:56.626958 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:56.626955 2572 generic.go:356] "Generic (PLEG): container finished" podID="fb5f129e-7b15-49fa-9f02-4978dc5e60aa" containerID="9f3f0cb1e6d6358a070b364c742be14f7f760ef8b19053e03542fb44dc9957ce" exitCode=0 Jul 22 21:11:56.627332 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:56.626998 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fb5f129e-7b15-49fa-9f02-4978dc5e60aa","Type":"ContainerDied","Data":"3c93264ff08292125c50471a1281e3b6bac25a76f59f8d5448e08a450c0bd35e"} Jul 22 21:11:56.627332 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:56.627053 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fb5f129e-7b15-49fa-9f02-4978dc5e60aa","Type":"ContainerDied","Data":"9f3f0cb1e6d6358a070b364c742be14f7f760ef8b19053e03542fb44dc9957ce"} Jul 22 21:11:56.627332 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:56.627077 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fb5f129e-7b15-49fa-9f02-4978dc5e60aa","Type":"ContainerDied","Data":"e69e7bf7e6e2995ba0c53e56bee014e81d672ccbe820f5eca80764e44ff2c7e3"} Jul 22 21:11:56.627332 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:56.627101 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e69e7bf7e6e2995ba0c53e56bee014e81d672ccbe820f5eca80764e44ff2c7e3" Jul 22 21:11:56.628900 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:56.628880 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 22 21:11:56.695890 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:56.695805 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fb5f129e-7b15-49fa-9f02-4978dc5e60aa-cluster-tls-config\") pod \"fb5f129e-7b15-49fa-9f02-4978dc5e60aa\" (UID: \"fb5f129e-7b15-49fa-9f02-4978dc5e60aa\") " Jul 22 21:11:56.695890 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:56.695848 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fb5f129e-7b15-49fa-9f02-4978dc5e60aa-web-config\") pod \"fb5f129e-7b15-49fa-9f02-4978dc5e60aa\" (UID: \"fb5f129e-7b15-49fa-9f02-4978dc5e60aa\") " Jul 22 21:11:56.695890 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:56.695886 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fb5f129e-7b15-49fa-9f02-4978dc5e60aa-tls-assets\") pod \"fb5f129e-7b15-49fa-9f02-4978dc5e60aa\" (UID: \"fb5f129e-7b15-49fa-9f02-4978dc5e60aa\") " Jul 22 21:11:56.696152 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:56.695914 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fb5f129e-7b15-49fa-9f02-4978dc5e60aa-alertmanager-main-db\") pod \"fb5f129e-7b15-49fa-9f02-4978dc5e60aa\" (UID: \"fb5f129e-7b15-49fa-9f02-4978dc5e60aa\") " Jul 22 21:11:56.696152 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:56.695943 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fb5f129e-7b15-49fa-9f02-4978dc5e60aa-secret-alertmanager-kube-rbac-proxy-web\") pod \"fb5f129e-7b15-49fa-9f02-4978dc5e60aa\" (UID: \"fb5f129e-7b15-49fa-9f02-4978dc5e60aa\") " Jul 22 21:11:56.696152 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:56.695974 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fb5f129e-7b15-49fa-9f02-4978dc5e60aa-config-out\") pod \"fb5f129e-7b15-49fa-9f02-4978dc5e60aa\" (UID: \"fb5f129e-7b15-49fa-9f02-4978dc5e60aa\") " Jul 22 21:11:56.696152 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:56.696001 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fb5f129e-7b15-49fa-9f02-4978dc5e60aa-secret-alertmanager-kube-rbac-proxy-metric\") pod \"fb5f129e-7b15-49fa-9f02-4978dc5e60aa\" (UID: \"fb5f129e-7b15-49fa-9f02-4978dc5e60aa\") " Jul 22 21:11:56.696394 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:56.696284 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fb5f129e-7b15-49fa-9f02-4978dc5e60aa-secret-alertmanager-main-tls\") pod \"fb5f129e-7b15-49fa-9f02-4978dc5e60aa\" (UID: \"fb5f129e-7b15-49fa-9f02-4978dc5e60aa\") " Jul 22 21:11:56.696394 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:56.696356 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb5f129e-7b15-49fa-9f02-4978dc5e60aa-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "fb5f129e-7b15-49fa-9f02-4978dc5e60aa" (UID: "fb5f129e-7b15-49fa-9f02-4978dc5e60aa"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 21:11:56.697765 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:56.696332 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pv4d2\" (UniqueName: \"kubernetes.io/projected/fb5f129e-7b15-49fa-9f02-4978dc5e60aa-kube-api-access-pv4d2\") pod \"fb5f129e-7b15-49fa-9f02-4978dc5e60aa\" (UID: \"fb5f129e-7b15-49fa-9f02-4978dc5e60aa\") " Jul 22 21:11:56.697765 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:56.696802 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fb5f129e-7b15-49fa-9f02-4978dc5e60aa-secret-alertmanager-kube-rbac-proxy\") pod \"fb5f129e-7b15-49fa-9f02-4978dc5e60aa\" (UID: \"fb5f129e-7b15-49fa-9f02-4978dc5e60aa\") " Jul 22 21:11:56.697765 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:56.696845 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb5f129e-7b15-49fa-9f02-4978dc5e60aa-alertmanager-trusted-ca-bundle\") pod \"fb5f129e-7b15-49fa-9f02-4978dc5e60aa\" (UID: \"fb5f129e-7b15-49fa-9f02-4978dc5e60aa\") " Jul 22 21:11:56.697765 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:56.696907 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fb5f129e-7b15-49fa-9f02-4978dc5e60aa-metrics-client-ca\") pod \"fb5f129e-7b15-49fa-9f02-4978dc5e60aa\" (UID: \"fb5f129e-7b15-49fa-9f02-4978dc5e60aa\") " Jul 22 21:11:56.697765 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:56.696948 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/fb5f129e-7b15-49fa-9f02-4978dc5e60aa-config-volume\") pod \"fb5f129e-7b15-49fa-9f02-4978dc5e60aa\" (UID: \"fb5f129e-7b15-49fa-9f02-4978dc5e60aa\") " Jul 22 21:11:56.697765 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:56.697258 2572 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fb5f129e-7b15-49fa-9f02-4978dc5e60aa-alertmanager-main-db\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:11:56.698815 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:56.698765 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb5f129e-7b15-49fa-9f02-4978dc5e60aa-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "fb5f129e-7b15-49fa-9f02-4978dc5e60aa" (UID: "fb5f129e-7b15-49fa-9f02-4978dc5e60aa"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 21:11:56.699315 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:56.699284 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb5f129e-7b15-49fa-9f02-4978dc5e60aa-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "fb5f129e-7b15-49fa-9f02-4978dc5e60aa" (UID: "fb5f129e-7b15-49fa-9f02-4978dc5e60aa"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 21:11:56.699525 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:56.699442 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb5f129e-7b15-49fa-9f02-4978dc5e60aa-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "fb5f129e-7b15-49fa-9f02-4978dc5e60aa" (UID: "fb5f129e-7b15-49fa-9f02-4978dc5e60aa"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 21:11:56.700038 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:56.700017 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb5f129e-7b15-49fa-9f02-4978dc5e60aa-config-out" (OuterVolumeSpecName: "config-out") pod "fb5f129e-7b15-49fa-9f02-4978dc5e60aa" (UID: "fb5f129e-7b15-49fa-9f02-4978dc5e60aa"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 21:11:56.700532 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:56.700480 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb5f129e-7b15-49fa-9f02-4978dc5e60aa-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "fb5f129e-7b15-49fa-9f02-4978dc5e60aa" (UID: "fb5f129e-7b15-49fa-9f02-4978dc5e60aa"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 21:11:56.700753 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:56.700717 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb5f129e-7b15-49fa-9f02-4978dc5e60aa-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "fb5f129e-7b15-49fa-9f02-4978dc5e60aa" (UID: "fb5f129e-7b15-49fa-9f02-4978dc5e60aa"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 21:11:56.700901 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:56.700874 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb5f129e-7b15-49fa-9f02-4978dc5e60aa-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "fb5f129e-7b15-49fa-9f02-4978dc5e60aa" (UID: "fb5f129e-7b15-49fa-9f02-4978dc5e60aa"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 21:11:56.701380 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:56.701349 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb5f129e-7b15-49fa-9f02-4978dc5e60aa-kube-api-access-pv4d2" (OuterVolumeSpecName: "kube-api-access-pv4d2") pod "fb5f129e-7b15-49fa-9f02-4978dc5e60aa" (UID: "fb5f129e-7b15-49fa-9f02-4978dc5e60aa"). InnerVolumeSpecName "kube-api-access-pv4d2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 21:11:56.702273 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:56.702249 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb5f129e-7b15-49fa-9f02-4978dc5e60aa-config-volume" (OuterVolumeSpecName: "config-volume") pod "fb5f129e-7b15-49fa-9f02-4978dc5e60aa" (UID: "fb5f129e-7b15-49fa-9f02-4978dc5e60aa"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 21:11:56.702413 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:56.702389 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb5f129e-7b15-49fa-9f02-4978dc5e60aa-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "fb5f129e-7b15-49fa-9f02-4978dc5e60aa" (UID: "fb5f129e-7b15-49fa-9f02-4978dc5e60aa"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 21:11:56.704215 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:56.704139 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb5f129e-7b15-49fa-9f02-4978dc5e60aa-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "fb5f129e-7b15-49fa-9f02-4978dc5e60aa" (UID: "fb5f129e-7b15-49fa-9f02-4978dc5e60aa"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 21:11:56.711520 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:56.711498 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb5f129e-7b15-49fa-9f02-4978dc5e60aa-web-config" (OuterVolumeSpecName: "web-config") pod "fb5f129e-7b15-49fa-9f02-4978dc5e60aa" (UID: "fb5f129e-7b15-49fa-9f02-4978dc5e60aa"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 21:11:56.797849 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:56.797816 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pv4d2\" (UniqueName: \"kubernetes.io/projected/fb5f129e-7b15-49fa-9f02-4978dc5e60aa-kube-api-access-pv4d2\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:11:56.797849 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:56.797843 2572 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fb5f129e-7b15-49fa-9f02-4978dc5e60aa-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:11:56.797849 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:56.797853 2572 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb5f129e-7b15-49fa-9f02-4978dc5e60aa-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:11:56.798051 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:56.797861 2572 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fb5f129e-7b15-49fa-9f02-4978dc5e60aa-metrics-client-ca\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:11:56.798051 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:56.797871 2572 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/fb5f129e-7b15-49fa-9f02-4978dc5e60aa-config-volume\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:11:56.798051 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:56.797879 2572 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fb5f129e-7b15-49fa-9f02-4978dc5e60aa-cluster-tls-config\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:11:56.798051 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:56.797887 2572 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fb5f129e-7b15-49fa-9f02-4978dc5e60aa-web-config\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:11:56.798051 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:56.797895 2572 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fb5f129e-7b15-49fa-9f02-4978dc5e60aa-tls-assets\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:11:56.798051 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:56.797903 2572 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fb5f129e-7b15-49fa-9f02-4978dc5e60aa-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:11:56.798051 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:56.797910 2572 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fb5f129e-7b15-49fa-9f02-4978dc5e60aa-config-out\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:11:56.798051 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:56.797918 2572 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fb5f129e-7b15-49fa-9f02-4978dc5e60aa-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:11:56.798051 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:56.797926 2572 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fb5f129e-7b15-49fa-9f02-4978dc5e60aa-secret-alertmanager-main-tls\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:11:57.630985 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:57.630954 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 22 21:11:57.654291 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:57.654224 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 21:11:57.658737 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:57.658709 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 21:11:57.686451 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:57.686428 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 21:11:57.686729 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:57.686714 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb5f129e-7b15-49fa-9f02-4978dc5e60aa" containerName="alertmanager" Jul 22 21:11:57.686778 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:57.686733 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb5f129e-7b15-49fa-9f02-4978dc5e60aa" containerName="alertmanager" Jul 22 21:11:57.686778 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:57.686741 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb5f129e-7b15-49fa-9f02-4978dc5e60aa" containerName="kube-rbac-proxy" Jul 22 21:11:57.686778 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:57.686747 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb5f129e-7b15-49fa-9f02-4978dc5e60aa" containerName="kube-rbac-proxy" Jul 22 21:11:57.686778 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:57.686757 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb5f129e-7b15-49fa-9f02-4978dc5e60aa" containerName="init-config-reloader" Jul 22 21:11:57.686778 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:57.686762 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb5f129e-7b15-49fa-9f02-4978dc5e60aa" containerName="init-config-reloader" Jul 22 21:11:57.686778 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:57.686770 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb5f129e-7b15-49fa-9f02-4978dc5e60aa" containerName="prom-label-proxy" Jul 22 21:11:57.686778 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:57.686774 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb5f129e-7b15-49fa-9f02-4978dc5e60aa" containerName="prom-label-proxy" Jul 22 21:11:57.686967 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:57.686784 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb5f129e-7b15-49fa-9f02-4978dc5e60aa" containerName="kube-rbac-proxy-web" Jul 22 21:11:57.686967 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:57.686790 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb5f129e-7b15-49fa-9f02-4978dc5e60aa" containerName="kube-rbac-proxy-web" Jul 22 21:11:57.686967 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:57.686803 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb5f129e-7b15-49fa-9f02-4978dc5e60aa" containerName="kube-rbac-proxy-metric" Jul 22 21:11:57.686967 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:57.686811 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb5f129e-7b15-49fa-9f02-4978dc5e60aa" containerName="kube-rbac-proxy-metric" Jul 22 21:11:57.686967 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:57.686818 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb5f129e-7b15-49fa-9f02-4978dc5e60aa" containerName="config-reloader" Jul 22 21:11:57.686967 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:57.686823 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb5f129e-7b15-49fa-9f02-4978dc5e60aa" containerName="config-reloader" Jul 22 21:11:57.686967 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:57.686835 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e591af84-a844-4ba8-8855-3dc41a45b70b" containerName="console" Jul 22 21:11:57.686967 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:57.686840 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="e591af84-a844-4ba8-8855-3dc41a45b70b" containerName="console" Jul 22 21:11:57.686967 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:57.686895 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb5f129e-7b15-49fa-9f02-4978dc5e60aa" containerName="config-reloader" Jul 22 21:11:57.686967 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:57.686903 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb5f129e-7b15-49fa-9f02-4978dc5e60aa" containerName="prom-label-proxy" Jul 22 21:11:57.686967 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:57.686909 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb5f129e-7b15-49fa-9f02-4978dc5e60aa" containerName="kube-rbac-proxy-metric" Jul 22 21:11:57.686967 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:57.686914 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb5f129e-7b15-49fa-9f02-4978dc5e60aa" containerName="kube-rbac-proxy-web" Jul 22 21:11:57.686967 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:57.686918 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb5f129e-7b15-49fa-9f02-4978dc5e60aa" containerName="kube-rbac-proxy" Jul 22 21:11:57.686967 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:57.686924 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="e591af84-a844-4ba8-8855-3dc41a45b70b" containerName="console" Jul 22 21:11:57.686967 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:57.686930 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb5f129e-7b15-49fa-9f02-4978dc5e60aa" containerName="alertmanager" Jul 22 21:11:57.691459 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:57.691444 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 22 21:11:57.693935 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:57.693914 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 22 21:11:57.694032 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:57.693959 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 22 21:11:57.694233 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:57.694216 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 22 21:11:57.694567 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:57.694432 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 22 21:11:57.694567 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:57.694439 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 22 21:11:57.694567 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:57.694463 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 22 21:11:57.694567 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:57.694474 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-8p5gr\"" Jul 22 21:11:57.694567 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:57.694466 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 22 21:11:57.694567 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:57.694466 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 22 21:11:57.700050 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:57.700030 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 22 21:11:57.707873 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:57.707852 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 21:11:57.807465 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:57.807437 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/dec07965-3435-4118-b8bd-f6b20ca3db58-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"dec07965-3435-4118-b8bd-f6b20ca3db58\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 21:11:57.807599 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:57.807476 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/dec07965-3435-4118-b8bd-f6b20ca3db58-config-volume\") pod \"alertmanager-main-0\" (UID: \"dec07965-3435-4118-b8bd-f6b20ca3db58\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 21:11:57.807599 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:57.807498 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dec07965-3435-4118-b8bd-f6b20ca3db58-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"dec07965-3435-4118-b8bd-f6b20ca3db58\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 21:11:57.807599 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:57.807516 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dec07965-3435-4118-b8bd-f6b20ca3db58-tls-assets\") pod \"alertmanager-main-0\" (UID: \"dec07965-3435-4118-b8bd-f6b20ca3db58\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 21:11:57.807719 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:57.807611 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/dec07965-3435-4118-b8bd-f6b20ca3db58-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"dec07965-3435-4118-b8bd-f6b20ca3db58\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 21:11:57.807719 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:57.807644 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/dec07965-3435-4118-b8bd-f6b20ca3db58-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"dec07965-3435-4118-b8bd-f6b20ca3db58\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 21:11:57.807719 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:57.807669 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dec07965-3435-4118-b8bd-f6b20ca3db58-config-out\") pod \"alertmanager-main-0\" (UID: \"dec07965-3435-4118-b8bd-f6b20ca3db58\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 21:11:57.807719 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:57.807693 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/dec07965-3435-4118-b8bd-f6b20ca3db58-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"dec07965-3435-4118-b8bd-f6b20ca3db58\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 21:11:57.807719 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:57.807716 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dec07965-3435-4118-b8bd-f6b20ca3db58-web-config\") pod \"alertmanager-main-0\" (UID: \"dec07965-3435-4118-b8bd-f6b20ca3db58\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 21:11:57.807893 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:57.807740 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dec07965-3435-4118-b8bd-f6b20ca3db58-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"dec07965-3435-4118-b8bd-f6b20ca3db58\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 21:11:57.807893 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:57.807788 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lmqw\" (UniqueName: \"kubernetes.io/projected/dec07965-3435-4118-b8bd-f6b20ca3db58-kube-api-access-5lmqw\") pod \"alertmanager-main-0\" (UID: \"dec07965-3435-4118-b8bd-f6b20ca3db58\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 21:11:57.807893 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:57.807817 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dec07965-3435-4118-b8bd-f6b20ca3db58-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"dec07965-3435-4118-b8bd-f6b20ca3db58\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 21:11:57.807893 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:57.807836 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dec07965-3435-4118-b8bd-f6b20ca3db58-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"dec07965-3435-4118-b8bd-f6b20ca3db58\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 21:11:57.908772 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:57.908708 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/dec07965-3435-4118-b8bd-f6b20ca3db58-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"dec07965-3435-4118-b8bd-f6b20ca3db58\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 21:11:57.908772 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:57.908737 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/dec07965-3435-4118-b8bd-f6b20ca3db58-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"dec07965-3435-4118-b8bd-f6b20ca3db58\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 21:11:57.908772 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:57.908753 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dec07965-3435-4118-b8bd-f6b20ca3db58-config-out\") pod \"alertmanager-main-0\" (UID: \"dec07965-3435-4118-b8bd-f6b20ca3db58\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 21:11:57.908772 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:57.908769 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/dec07965-3435-4118-b8bd-f6b20ca3db58-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"dec07965-3435-4118-b8bd-f6b20ca3db58\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 21:11:57.908995 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:57.908788 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dec07965-3435-4118-b8bd-f6b20ca3db58-web-config\") pod \"alertmanager-main-0\" (UID: \"dec07965-3435-4118-b8bd-f6b20ca3db58\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 21:11:57.908995 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:57.908814 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dec07965-3435-4118-b8bd-f6b20ca3db58-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"dec07965-3435-4118-b8bd-f6b20ca3db58\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 21:11:57.908995 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:57.908845 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lmqw\" (UniqueName: \"kubernetes.io/projected/dec07965-3435-4118-b8bd-f6b20ca3db58-kube-api-access-5lmqw\") pod \"alertmanager-main-0\" (UID: \"dec07965-3435-4118-b8bd-f6b20ca3db58\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 21:11:57.908995 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:57.908981 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dec07965-3435-4118-b8bd-f6b20ca3db58-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"dec07965-3435-4118-b8bd-f6b20ca3db58\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 21:11:57.909908 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:57.909024 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dec07965-3435-4118-b8bd-f6b20ca3db58-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"dec07965-3435-4118-b8bd-f6b20ca3db58\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 21:11:57.909908 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:57.909068 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/dec07965-3435-4118-b8bd-f6b20ca3db58-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"dec07965-3435-4118-b8bd-f6b20ca3db58\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 21:11:57.909908 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:57.909108 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/dec07965-3435-4118-b8bd-f6b20ca3db58-config-volume\") pod \"alertmanager-main-0\" (UID: \"dec07965-3435-4118-b8bd-f6b20ca3db58\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 21:11:57.909908 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:57.909138 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dec07965-3435-4118-b8bd-f6b20ca3db58-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"dec07965-3435-4118-b8bd-f6b20ca3db58\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 21:11:57.909908 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:57.909192 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dec07965-3435-4118-b8bd-f6b20ca3db58-tls-assets\") pod \"alertmanager-main-0\" (UID: \"dec07965-3435-4118-b8bd-f6b20ca3db58\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 21:11:57.909908 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:57.909487 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/dec07965-3435-4118-b8bd-f6b20ca3db58-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"dec07965-3435-4118-b8bd-f6b20ca3db58\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 21:11:57.910464 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:57.910442 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dec07965-3435-4118-b8bd-f6b20ca3db58-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"dec07965-3435-4118-b8bd-f6b20ca3db58\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 21:11:57.910823 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:57.910692 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dec07965-3435-4118-b8bd-f6b20ca3db58-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"dec07965-3435-4118-b8bd-f6b20ca3db58\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 21:11:57.912103 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:57.912060 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dec07965-3435-4118-b8bd-f6b20ca3db58-config-out\") pod \"alertmanager-main-0\" (UID: \"dec07965-3435-4118-b8bd-f6b20ca3db58\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 21:11:57.912243 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:57.912222 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dec07965-3435-4118-b8bd-f6b20ca3db58-tls-assets\") pod \"alertmanager-main-0\" (UID: \"dec07965-3435-4118-b8bd-f6b20ca3db58\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 21:11:57.912458 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:57.912438 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/dec07965-3435-4118-b8bd-f6b20ca3db58-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"dec07965-3435-4118-b8bd-f6b20ca3db58\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 21:11:57.912536 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:57.912438 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/dec07965-3435-4118-b8bd-f6b20ca3db58-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"dec07965-3435-4118-b8bd-f6b20ca3db58\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 21:11:57.912968 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:57.912949 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dec07965-3435-4118-b8bd-f6b20ca3db58-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"dec07965-3435-4118-b8bd-f6b20ca3db58\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 21:11:57.913388 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:57.913367 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/dec07965-3435-4118-b8bd-f6b20ca3db58-config-volume\") pod \"alertmanager-main-0\" (UID: \"dec07965-3435-4118-b8bd-f6b20ca3db58\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 21:11:57.913459 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:57.913432 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/dec07965-3435-4118-b8bd-f6b20ca3db58-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"dec07965-3435-4118-b8bd-f6b20ca3db58\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 21:11:57.913716 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:57.913694 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dec07965-3435-4118-b8bd-f6b20ca3db58-web-config\") pod \"alertmanager-main-0\" (UID: \"dec07965-3435-4118-b8bd-f6b20ca3db58\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 21:11:57.913772 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:57.913735 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dec07965-3435-4118-b8bd-f6b20ca3db58-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"dec07965-3435-4118-b8bd-f6b20ca3db58\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 21:11:57.917920 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:57.917898 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lmqw\" (UniqueName: \"kubernetes.io/projected/dec07965-3435-4118-b8bd-f6b20ca3db58-kube-api-access-5lmqw\") pod \"alertmanager-main-0\" (UID: \"dec07965-3435-4118-b8bd-f6b20ca3db58\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 21:11:58.000240 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:58.000202 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 22 21:11:58.135240 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:58.135134 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 21:11:58.137475 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:11:58.137446 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddec07965_3435_4118_b8bd_f6b20ca3db58.slice/crio-a94dedd5b23a2b9144a5f3cf4a315c634cc269bef0ad49cf833e2264af2425c4 WatchSource:0}: Error finding container a94dedd5b23a2b9144a5f3cf4a315c634cc269bef0ad49cf833e2264af2425c4: Status 404 returned error can't find the container with id a94dedd5b23a2b9144a5f3cf4a315c634cc269bef0ad49cf833e2264af2425c4 Jul 22 21:11:58.636038 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:58.636003 2572 generic.go:356] "Generic (PLEG): container finished" podID="dec07965-3435-4118-b8bd-f6b20ca3db58" containerID="beeac001d0c5dd371d94960cbbf97904ac4c69bf0ca6b35665d229d5f464e715" exitCode=0 Jul 22 21:11:58.636447 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:58.636068 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dec07965-3435-4118-b8bd-f6b20ca3db58","Type":"ContainerDied","Data":"beeac001d0c5dd371d94960cbbf97904ac4c69bf0ca6b35665d229d5f464e715"} Jul 22 21:11:58.636447 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:58.636145 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dec07965-3435-4118-b8bd-f6b20ca3db58","Type":"ContainerStarted","Data":"a94dedd5b23a2b9144a5f3cf4a315c634cc269bef0ad49cf833e2264af2425c4"} Jul 22 21:11:59.119386 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:59.119351 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a91c48d-374d-4fc3-9114-d5c88ea18f63-metrics-certs\") pod \"network-metrics-daemon-lj48g\" (UID: \"8a91c48d-374d-4fc3-9114-d5c88ea18f63\") " pod="openshift-multus/network-metrics-daemon-lj48g" Jul 22 21:11:59.121574 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:59.121544 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a91c48d-374d-4fc3-9114-d5c88ea18f63-metrics-certs\") pod \"network-metrics-daemon-lj48g\" (UID: \"8a91c48d-374d-4fc3-9114-d5c88ea18f63\") " pod="openshift-multus/network-metrics-daemon-lj48g" Jul 22 21:11:59.180406 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:59.180342 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8629h\"" Jul 22 21:11:59.188438 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:59.188416 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lj48g" Jul 22 21:11:59.313805 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:59.313774 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lj48g"] Jul 22 21:11:59.316877 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:11:59.316856 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a91c48d_374d_4fc3_9114_d5c88ea18f63.slice/crio-47f4cc8c1eb7e8aff2d0be57ab3fac779c7d3c42f4689f46b170512085041146 WatchSource:0}: Error finding container 47f4cc8c1eb7e8aff2d0be57ab3fac779c7d3c42f4689f46b170512085041146: Status 404 returned error can't find the container with id 47f4cc8c1eb7e8aff2d0be57ab3fac779c7d3c42f4689f46b170512085041146 Jul 22 21:11:59.318796 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:59.318778 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 21:11:59.383429 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:59.383403 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb5f129e-7b15-49fa-9f02-4978dc5e60aa" path="/var/lib/kubelet/pods/fb5f129e-7b15-49fa-9f02-4978dc5e60aa/volumes" Jul 22 21:11:59.641858 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:59.641822 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lj48g" event={"ID":"8a91c48d-374d-4fc3-9114-d5c88ea18f63","Type":"ContainerStarted","Data":"47f4cc8c1eb7e8aff2d0be57ab3fac779c7d3c42f4689f46b170512085041146"} Jul 22 21:11:59.644697 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:59.644668 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dec07965-3435-4118-b8bd-f6b20ca3db58","Type":"ContainerStarted","Data":"24e79fd87c0bd4659af149cef19a228fabbaed46a76cafec25754b60ee65f51b"} Jul 22 21:11:59.644791 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:59.644738 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dec07965-3435-4118-b8bd-f6b20ca3db58","Type":"ContainerStarted","Data":"8558592be84d4147c44136430eca312366b89d7580d7cb8927cea887b2c09e97"} Jul 22 21:11:59.644842 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:59.644822 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dec07965-3435-4118-b8bd-f6b20ca3db58","Type":"ContainerStarted","Data":"9df3a86ef67119bb2afde3afbb0eb2e416fae2453a201b9c22ec3243b08b226e"} Jul 22 21:11:59.644876 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:59.644858 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dec07965-3435-4118-b8bd-f6b20ca3db58","Type":"ContainerStarted","Data":"e14e35d69b949817c9ef00a5f60df5076cbe693bd332ea0393137692ea491c15"} Jul 22 21:11:59.644911 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:59.644895 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dec07965-3435-4118-b8bd-f6b20ca3db58","Type":"ContainerStarted","Data":"a0266b99a79e02a3199c962a6efd7378dfe6d92fe586bc9cf8896fb316cf1041"} Jul 22 21:11:59.644940 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:59.644930 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dec07965-3435-4118-b8bd-f6b20ca3db58","Type":"ContainerStarted","Data":"6435e248437aa2db71f32882b7a1b051f05e086c647bf50d2a28363aeb7c4e7e"} Jul 22 21:11:59.675219 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:11:59.675153 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.658617884 podStartE2EDuration="2.675139854s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.52197ms" isStatefulPod=false podCreationTimestamp="2026-07-22 21:11:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 21:11:59.673110029 +0000 UTC m=+374.787502374" watchObservedRunningTime="2026-07-22 21:11:59.675139854 +0000 UTC m=+374.789532197" Jul 22 21:12:00.651556 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:00.651516 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lj48g" event={"ID":"8a91c48d-374d-4fc3-9114-d5c88ea18f63","Type":"ContainerStarted","Data":"663123149bff689b41a0b3a45105d95fcac80f0bf59765596935378609086fa4"} Jul 22 21:12:00.651954 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:00.651583 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lj48g" event={"ID":"8a91c48d-374d-4fc3-9114-d5c88ea18f63","Type":"ContainerStarted","Data":"3570ef7eef0cf7e7cf6690ce939b57dc736c61ddd0cf080bc1fe9979e93faf7c"} Jul 22 21:12:00.667637 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:00.667596 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-lj48g" podStartSLOduration=374.730967428 podStartE2EDuration="6m15.667581701s" totalImagesPullingTime="936.614273ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 21:05:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 21:12:00.666209508 +0000 UTC m=+375.780601875" watchObservedRunningTime="2026-07-22 21:12:00.667581701 +0000 UTC m=+375.781974044" Jul 22 21:12:01.149223 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:01.149190 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-84856587d6-k8ntg"] Jul 22 21:12:01.152583 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:01.152567 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84856587d6-k8ntg" Jul 22 21:12:01.162909 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:01.162882 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84856587d6-k8ntg"] Jul 22 21:12:01.237954 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:01.237929 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gblgv\" (UniqueName: \"kubernetes.io/projected/2631535b-cad2-48f9-b739-f96f22a0707a-kube-api-access-gblgv\") pod \"console-84856587d6-k8ntg\" (UID: \"2631535b-cad2-48f9-b739-f96f22a0707a\") " pod="openshift-console/console-84856587d6-k8ntg" Jul 22 21:12:01.238060 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:01.237958 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2631535b-cad2-48f9-b739-f96f22a0707a-service-ca\") pod \"console-84856587d6-k8ntg\" (UID: \"2631535b-cad2-48f9-b739-f96f22a0707a\") " pod="openshift-console/console-84856587d6-k8ntg" Jul 22 21:12:01.238060 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:01.237977 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2631535b-cad2-48f9-b739-f96f22a0707a-trusted-ca-bundle\") pod \"console-84856587d6-k8ntg\" (UID: \"2631535b-cad2-48f9-b739-f96f22a0707a\") " pod="openshift-console/console-84856587d6-k8ntg" Jul 22 21:12:01.238145 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:01.238067 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2631535b-cad2-48f9-b739-f96f22a0707a-console-config\") pod \"console-84856587d6-k8ntg\" (UID: \"2631535b-cad2-48f9-b739-f96f22a0707a\") " pod="openshift-console/console-84856587d6-k8ntg" Jul 22 21:12:01.238145 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:01.238131 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2631535b-cad2-48f9-b739-f96f22a0707a-oauth-serving-cert\") pod \"console-84856587d6-k8ntg\" (UID: \"2631535b-cad2-48f9-b739-f96f22a0707a\") " pod="openshift-console/console-84856587d6-k8ntg" Jul 22 21:12:01.238239 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:01.238195 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2631535b-cad2-48f9-b739-f96f22a0707a-console-serving-cert\") pod \"console-84856587d6-k8ntg\" (UID: \"2631535b-cad2-48f9-b739-f96f22a0707a\") " pod="openshift-console/console-84856587d6-k8ntg" Jul 22 21:12:01.238239 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:01.238214 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2631535b-cad2-48f9-b739-f96f22a0707a-console-oauth-config\") pod \"console-84856587d6-k8ntg\" (UID: \"2631535b-cad2-48f9-b739-f96f22a0707a\") " pod="openshift-console/console-84856587d6-k8ntg" Jul 22 21:12:01.338865 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:01.338841 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2631535b-cad2-48f9-b739-f96f22a0707a-console-config\") pod \"console-84856587d6-k8ntg\" (UID: \"2631535b-cad2-48f9-b739-f96f22a0707a\") " pod="openshift-console/console-84856587d6-k8ntg" Jul 22 21:12:01.339026 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:01.338879 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2631535b-cad2-48f9-b739-f96f22a0707a-oauth-serving-cert\") pod \"console-84856587d6-k8ntg\" (UID: \"2631535b-cad2-48f9-b739-f96f22a0707a\") " pod="openshift-console/console-84856587d6-k8ntg" Jul 22 21:12:01.339026 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:01.338900 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2631535b-cad2-48f9-b739-f96f22a0707a-console-serving-cert\") pod \"console-84856587d6-k8ntg\" (UID: \"2631535b-cad2-48f9-b739-f96f22a0707a\") " pod="openshift-console/console-84856587d6-k8ntg" Jul 22 21:12:01.339026 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:01.338915 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2631535b-cad2-48f9-b739-f96f22a0707a-console-oauth-config\") pod \"console-84856587d6-k8ntg\" (UID: \"2631535b-cad2-48f9-b739-f96f22a0707a\") " pod="openshift-console/console-84856587d6-k8ntg" Jul 22 21:12:01.339026 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:01.338931 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gblgv\" (UniqueName: \"kubernetes.io/projected/2631535b-cad2-48f9-b739-f96f22a0707a-kube-api-access-gblgv\") pod \"console-84856587d6-k8ntg\" (UID: \"2631535b-cad2-48f9-b739-f96f22a0707a\") " pod="openshift-console/console-84856587d6-k8ntg" Jul 22 21:12:01.339026 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:01.338947 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2631535b-cad2-48f9-b739-f96f22a0707a-service-ca\") pod \"console-84856587d6-k8ntg\" (UID: \"2631535b-cad2-48f9-b739-f96f22a0707a\") " pod="openshift-console/console-84856587d6-k8ntg" Jul 22 21:12:01.339026 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:01.338963 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2631535b-cad2-48f9-b739-f96f22a0707a-trusted-ca-bundle\") pod \"console-84856587d6-k8ntg\" (UID: \"2631535b-cad2-48f9-b739-f96f22a0707a\") " pod="openshift-console/console-84856587d6-k8ntg" Jul 22 21:12:01.339719 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:01.339695 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2631535b-cad2-48f9-b739-f96f22a0707a-oauth-serving-cert\") pod \"console-84856587d6-k8ntg\" (UID: \"2631535b-cad2-48f9-b739-f96f22a0707a\") " pod="openshift-console/console-84856587d6-k8ntg" Jul 22 21:12:01.339806 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:01.339790 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2631535b-cad2-48f9-b739-f96f22a0707a-service-ca\") pod \"console-84856587d6-k8ntg\" (UID: \"2631535b-cad2-48f9-b739-f96f22a0707a\") " pod="openshift-console/console-84856587d6-k8ntg" Jul 22 21:12:01.339918 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:01.339902 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2631535b-cad2-48f9-b739-f96f22a0707a-trusted-ca-bundle\") pod \"console-84856587d6-k8ntg\" (UID: \"2631535b-cad2-48f9-b739-f96f22a0707a\") " pod="openshift-console/console-84856587d6-k8ntg" Jul 22 21:12:01.339958 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:01.339907 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2631535b-cad2-48f9-b739-f96f22a0707a-console-config\") pod \"console-84856587d6-k8ntg\" (UID: \"2631535b-cad2-48f9-b739-f96f22a0707a\") " pod="openshift-console/console-84856587d6-k8ntg" Jul 22 21:12:01.341526 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:01.341504 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2631535b-cad2-48f9-b739-f96f22a0707a-console-serving-cert\") pod \"console-84856587d6-k8ntg\" (UID: \"2631535b-cad2-48f9-b739-f96f22a0707a\") " pod="openshift-console/console-84856587d6-k8ntg" Jul 22 21:12:01.341989 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:01.341971 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2631535b-cad2-48f9-b739-f96f22a0707a-console-oauth-config\") pod \"console-84856587d6-k8ntg\" (UID: \"2631535b-cad2-48f9-b739-f96f22a0707a\") " pod="openshift-console/console-84856587d6-k8ntg" Jul 22 21:12:01.346884 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:01.346867 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gblgv\" (UniqueName: \"kubernetes.io/projected/2631535b-cad2-48f9-b739-f96f22a0707a-kube-api-access-gblgv\") pod \"console-84856587d6-k8ntg\" (UID: \"2631535b-cad2-48f9-b739-f96f22a0707a\") " pod="openshift-console/console-84856587d6-k8ntg" Jul 22 21:12:01.462092 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:01.462032 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84856587d6-k8ntg" Jul 22 21:12:01.588471 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:01.588448 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84856587d6-k8ntg"] Jul 22 21:12:01.590748 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:12:01.590721 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2631535b_cad2_48f9_b739_f96f22a0707a.slice/crio-7dc480be8487172b42f6801f1cda9478a88cbb5973313b58876243eda0e85414 WatchSource:0}: Error finding container 7dc480be8487172b42f6801f1cda9478a88cbb5973313b58876243eda0e85414: Status 404 returned error can't find the container with id 7dc480be8487172b42f6801f1cda9478a88cbb5973313b58876243eda0e85414 Jul 22 21:12:01.659471 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:01.659446 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84856587d6-k8ntg" event={"ID":"2631535b-cad2-48f9-b739-f96f22a0707a","Type":"ContainerStarted","Data":"7dc480be8487172b42f6801f1cda9478a88cbb5973313b58876243eda0e85414"} Jul 22 21:12:02.665860 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:02.665829 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84856587d6-k8ntg" event={"ID":"2631535b-cad2-48f9-b739-f96f22a0707a","Type":"ContainerStarted","Data":"e14020f083b8ae5d56ca9a871cf253cf33e6986cf9b657036811d832666bc913"} Jul 22 21:12:02.684796 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:02.684745 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-84856587d6-k8ntg" podStartSLOduration=1.684729471 podStartE2EDuration="1.684729471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 21:12:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 21:12:02.683233918 +0000 UTC m=+377.797626286" watchObservedRunningTime="2026-07-22 21:12:02.684729471 +0000 UTC m=+377.799121816" Jul 22 21:12:11.462127 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:11.462091 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-84856587d6-k8ntg" Jul 22 21:12:11.462615 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:11.462147 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-84856587d6-k8ntg" Jul 22 21:12:11.466706 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:11.466680 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-84856587d6-k8ntg" Jul 22 21:12:11.707090 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:11.707058 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-84856587d6-k8ntg" Jul 22 21:12:11.759716 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:11.759649 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-746784ffcc-88dk7"] Jul 22 21:12:36.785333 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:36.785293 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-746784ffcc-88dk7" podUID="57159d41-02c2-4d57-bbdc-67b79f18766d" containerName="console" containerID="cri-o://2492e89c444f8aa0c7c48293fb67a285ecf6daa840a0d65ded48791a25a1cd26" gracePeriod=15 Jul 22 21:12:37.035521 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:37.035464 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-746784ffcc-88dk7" Jul 22 21:12:37.216721 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:37.216687 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/57159d41-02c2-4d57-bbdc-67b79f18766d-console-oauth-config\") pod \"57159d41-02c2-4d57-bbdc-67b79f18766d\" (UID: \"57159d41-02c2-4d57-bbdc-67b79f18766d\") " Jul 22 21:12:37.216872 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:37.216732 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/57159d41-02c2-4d57-bbdc-67b79f18766d-console-config\") pod \"57159d41-02c2-4d57-bbdc-67b79f18766d\" (UID: \"57159d41-02c2-4d57-bbdc-67b79f18766d\") " Jul 22 21:12:37.216872 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:37.216772 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57159d41-02c2-4d57-bbdc-67b79f18766d-trusted-ca-bundle\") pod \"57159d41-02c2-4d57-bbdc-67b79f18766d\" (UID: \"57159d41-02c2-4d57-bbdc-67b79f18766d\") " Jul 22 21:12:37.216872 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:37.216798 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v944q\" (UniqueName: \"kubernetes.io/projected/57159d41-02c2-4d57-bbdc-67b79f18766d-kube-api-access-v944q\") pod \"57159d41-02c2-4d57-bbdc-67b79f18766d\" (UID: \"57159d41-02c2-4d57-bbdc-67b79f18766d\") " Jul 22 21:12:37.216872 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:37.216842 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/57159d41-02c2-4d57-bbdc-67b79f18766d-console-serving-cert\") pod \"57159d41-02c2-4d57-bbdc-67b79f18766d\" (UID: \"57159d41-02c2-4d57-bbdc-67b79f18766d\") " Jul 22 21:12:37.216872 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:37.216868 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/57159d41-02c2-4d57-bbdc-67b79f18766d-oauth-serving-cert\") pod \"57159d41-02c2-4d57-bbdc-67b79f18766d\" (UID: \"57159d41-02c2-4d57-bbdc-67b79f18766d\") " Jul 22 21:12:37.217198 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:37.216884 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/57159d41-02c2-4d57-bbdc-67b79f18766d-service-ca\") pod \"57159d41-02c2-4d57-bbdc-67b79f18766d\" (UID: \"57159d41-02c2-4d57-bbdc-67b79f18766d\") " Jul 22 21:12:37.217353 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:37.217325 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57159d41-02c2-4d57-bbdc-67b79f18766d-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "57159d41-02c2-4d57-bbdc-67b79f18766d" (UID: "57159d41-02c2-4d57-bbdc-67b79f18766d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 21:12:37.217353 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:37.217332 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57159d41-02c2-4d57-bbdc-67b79f18766d-console-config" (OuterVolumeSpecName: "console-config") pod "57159d41-02c2-4d57-bbdc-67b79f18766d" (UID: "57159d41-02c2-4d57-bbdc-67b79f18766d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 21:12:37.217471 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:37.217370 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57159d41-02c2-4d57-bbdc-67b79f18766d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "57159d41-02c2-4d57-bbdc-67b79f18766d" (UID: "57159d41-02c2-4d57-bbdc-67b79f18766d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 21:12:37.217471 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:37.217380 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57159d41-02c2-4d57-bbdc-67b79f18766d-service-ca" (OuterVolumeSpecName: "service-ca") pod "57159d41-02c2-4d57-bbdc-67b79f18766d" (UID: "57159d41-02c2-4d57-bbdc-67b79f18766d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 21:12:37.219042 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:37.219016 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57159d41-02c2-4d57-bbdc-67b79f18766d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "57159d41-02c2-4d57-bbdc-67b79f18766d" (UID: "57159d41-02c2-4d57-bbdc-67b79f18766d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 21:12:37.219145 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:37.219067 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57159d41-02c2-4d57-bbdc-67b79f18766d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "57159d41-02c2-4d57-bbdc-67b79f18766d" (UID: "57159d41-02c2-4d57-bbdc-67b79f18766d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 21:12:37.219145 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:37.219069 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57159d41-02c2-4d57-bbdc-67b79f18766d-kube-api-access-v944q" (OuterVolumeSpecName: "kube-api-access-v944q") pod "57159d41-02c2-4d57-bbdc-67b79f18766d" (UID: "57159d41-02c2-4d57-bbdc-67b79f18766d"). InnerVolumeSpecName "kube-api-access-v944q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 21:12:37.317781 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:37.317751 2572 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57159d41-02c2-4d57-bbdc-67b79f18766d-trusted-ca-bundle\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:12:37.317781 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:37.317777 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v944q\" (UniqueName: \"kubernetes.io/projected/57159d41-02c2-4d57-bbdc-67b79f18766d-kube-api-access-v944q\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:12:37.317988 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:37.317791 2572 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/57159d41-02c2-4d57-bbdc-67b79f18766d-console-serving-cert\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:12:37.317988 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:37.317803 2572 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/57159d41-02c2-4d57-bbdc-67b79f18766d-oauth-serving-cert\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:12:37.317988 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:37.317815 2572 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/57159d41-02c2-4d57-bbdc-67b79f18766d-service-ca\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:12:37.317988 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:37.317827 2572 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/57159d41-02c2-4d57-bbdc-67b79f18766d-console-oauth-config\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:12:37.317988 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:37.317838 2572 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/57159d41-02c2-4d57-bbdc-67b79f18766d-console-config\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:12:37.808681 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:37.808651 2572 generic.go:356] "Generic (PLEG): container finished" podID="57159d41-02c2-4d57-bbdc-67b79f18766d" containerID="2492e89c444f8aa0c7c48293fb67a285ecf6daa840a0d65ded48791a25a1cd26" exitCode=0 Jul 22 21:12:37.809038 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:37.808690 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-746784ffcc-88dk7" Jul 22 21:12:37.809038 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:37.808738 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-746784ffcc-88dk7" event={"ID":"57159d41-02c2-4d57-bbdc-67b79f18766d","Type":"ContainerDied","Data":"2492e89c444f8aa0c7c48293fb67a285ecf6daa840a0d65ded48791a25a1cd26"} Jul 22 21:12:37.809038 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:37.808787 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-746784ffcc-88dk7" event={"ID":"57159d41-02c2-4d57-bbdc-67b79f18766d","Type":"ContainerDied","Data":"a9bf993104f1fbe599fbf3cee849ff8e21f6e8e1a9f1185b459421459a47b804"} Jul 22 21:12:37.809038 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:37.808810 2572 scope.go:117] "RemoveContainer" containerID="2492e89c444f8aa0c7c48293fb67a285ecf6daa840a0d65ded48791a25a1cd26" Jul 22 21:12:37.816957 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:37.816942 2572 scope.go:117] "RemoveContainer" containerID="2492e89c444f8aa0c7c48293fb67a285ecf6daa840a0d65ded48791a25a1cd26" Jul 22 21:12:37.817238 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:12:37.817220 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2492e89c444f8aa0c7c48293fb67a285ecf6daa840a0d65ded48791a25a1cd26\": container with ID starting with 2492e89c444f8aa0c7c48293fb67a285ecf6daa840a0d65ded48791a25a1cd26 not found: ID does not exist" containerID="2492e89c444f8aa0c7c48293fb67a285ecf6daa840a0d65ded48791a25a1cd26" Jul 22 21:12:37.817306 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:37.817246 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2492e89c444f8aa0c7c48293fb67a285ecf6daa840a0d65ded48791a25a1cd26"} err="failed to get container status \"2492e89c444f8aa0c7c48293fb67a285ecf6daa840a0d65ded48791a25a1cd26\": rpc error: code = NotFound desc = could not find container \"2492e89c444f8aa0c7c48293fb67a285ecf6daa840a0d65ded48791a25a1cd26\": container with ID starting with 2492e89c444f8aa0c7c48293fb67a285ecf6daa840a0d65ded48791a25a1cd26 not found: ID does not exist" Jul 22 21:12:37.827287 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:37.827268 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-746784ffcc-88dk7"] Jul 22 21:12:37.831201 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:37.831156 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-746784ffcc-88dk7"] Jul 22 21:12:39.385349 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:39.385314 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57159d41-02c2-4d57-bbdc-67b79f18766d" path="/var/lib/kubelet/pods/57159d41-02c2-4d57-bbdc-67b79f18766d/volumes" Jul 22 21:12:49.722701 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:49.722667 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540htrrj"] Jul 22 21:12:49.723061 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:49.722971 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57159d41-02c2-4d57-bbdc-67b79f18766d" containerName="console" Jul 22 21:12:49.723061 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:49.722981 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="57159d41-02c2-4d57-bbdc-67b79f18766d" containerName="console" Jul 22 21:12:49.723061 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:49.723038 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="57159d41-02c2-4d57-bbdc-67b79f18766d" containerName="console" Jul 22 21:12:49.727624 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:49.727607 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540htrrj" Jul 22 21:12:49.730462 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:49.730441 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 22 21:12:49.731226 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:49.731209 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 22 21:12:49.731327 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:49.731214 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-fqwnq\"" Jul 22 21:12:49.736466 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:49.736443 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540htrrj"] Jul 22 21:12:49.807928 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:49.807895 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gss5v\" (UniqueName: \"kubernetes.io/projected/fd0c7ad5-77a4-4d49-b884-7d9eca964947-kube-api-access-gss5v\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540htrrj\" (UID: \"fd0c7ad5-77a4-4d49-b884-7d9eca964947\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540htrrj" Jul 22 21:12:49.807928 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:49.807931 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fd0c7ad5-77a4-4d49-b884-7d9eca964947-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540htrrj\" (UID: \"fd0c7ad5-77a4-4d49-b884-7d9eca964947\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540htrrj" Jul 22 21:12:49.808102 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:49.807950 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fd0c7ad5-77a4-4d49-b884-7d9eca964947-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540htrrj\" (UID: \"fd0c7ad5-77a4-4d49-b884-7d9eca964947\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540htrrj" Jul 22 21:12:49.909308 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:49.909277 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gss5v\" (UniqueName: \"kubernetes.io/projected/fd0c7ad5-77a4-4d49-b884-7d9eca964947-kube-api-access-gss5v\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540htrrj\" (UID: \"fd0c7ad5-77a4-4d49-b884-7d9eca964947\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540htrrj" Jul 22 21:12:49.909308 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:49.909313 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fd0c7ad5-77a4-4d49-b884-7d9eca964947-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540htrrj\" (UID: \"fd0c7ad5-77a4-4d49-b884-7d9eca964947\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540htrrj" Jul 22 21:12:49.909557 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:49.909332 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fd0c7ad5-77a4-4d49-b884-7d9eca964947-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540htrrj\" (UID: \"fd0c7ad5-77a4-4d49-b884-7d9eca964947\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540htrrj" Jul 22 21:12:49.909657 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:49.909641 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fd0c7ad5-77a4-4d49-b884-7d9eca964947-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540htrrj\" (UID: \"fd0c7ad5-77a4-4d49-b884-7d9eca964947\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540htrrj" Jul 22 21:12:49.909753 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:49.909730 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fd0c7ad5-77a4-4d49-b884-7d9eca964947-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540htrrj\" (UID: \"fd0c7ad5-77a4-4d49-b884-7d9eca964947\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540htrrj" Jul 22 21:12:49.917953 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:49.917933 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gss5v\" (UniqueName: \"kubernetes.io/projected/fd0c7ad5-77a4-4d49-b884-7d9eca964947-kube-api-access-gss5v\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540htrrj\" (UID: \"fd0c7ad5-77a4-4d49-b884-7d9eca964947\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540htrrj" Jul 22 21:12:50.037739 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:50.037645 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540htrrj" Jul 22 21:12:50.169668 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:50.169644 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540htrrj"] Jul 22 21:12:50.172037 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:12:50.172008 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd0c7ad5_77a4_4d49_b884_7d9eca964947.slice/crio-2ffb0f8f89721d6e85643c4d13505c42587ffa2707a927e8ed26dc6847b5c804 WatchSource:0}: Error finding container 2ffb0f8f89721d6e85643c4d13505c42587ffa2707a927e8ed26dc6847b5c804: Status 404 returned error can't find the container with id 2ffb0f8f89721d6e85643c4d13505c42587ffa2707a927e8ed26dc6847b5c804 Jul 22 21:12:50.863981 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:50.863944 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540htrrj" event={"ID":"fd0c7ad5-77a4-4d49-b884-7d9eca964947","Type":"ContainerStarted","Data":"2ffb0f8f89721d6e85643c4d13505c42587ffa2707a927e8ed26dc6847b5c804"} Jul 22 21:12:57.897882 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:57.897851 2572 generic.go:356] "Generic (PLEG): container finished" podID="fd0c7ad5-77a4-4d49-b884-7d9eca964947" containerID="289b0854d1dca3a464e1494cf17b1b4e38ef454cda2a989558c8c2cde531c67a" exitCode=0 Jul 22 21:12:57.898284 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:57.897889 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540htrrj" event={"ID":"fd0c7ad5-77a4-4d49-b884-7d9eca964947","Type":"ContainerDied","Data":"289b0854d1dca3a464e1494cf17b1b4e38ef454cda2a989558c8c2cde531c67a"} Jul 22 21:12:59.908944 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:12:59.908909 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540htrrj" event={"ID":"fd0c7ad5-77a4-4d49-b884-7d9eca964947","Type":"ContainerStarted","Data":"ced1c7b9afa909e880a78f6d7b3b3c421cb91fd8ddbf4639d5dcd2c0c0e020dc"} Jul 22 21:13:00.914376 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:00.914333 2572 generic.go:356] "Generic (PLEG): container finished" podID="fd0c7ad5-77a4-4d49-b884-7d9eca964947" containerID="ced1c7b9afa909e880a78f6d7b3b3c421cb91fd8ddbf4639d5dcd2c0c0e020dc" exitCode=0 Jul 22 21:13:00.914806 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:00.914423 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540htrrj" event={"ID":"fd0c7ad5-77a4-4d49-b884-7d9eca964947","Type":"ContainerDied","Data":"ced1c7b9afa909e880a78f6d7b3b3c421cb91fd8ddbf4639d5dcd2c0c0e020dc"} Jul 22 21:13:08.952912 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:08.952812 2572 generic.go:356] "Generic (PLEG): container finished" podID="fd0c7ad5-77a4-4d49-b884-7d9eca964947" containerID="e35f955ebc6a058e470c7591968bd0640a0681bff7aa2b61191170797c454515" exitCode=0 Jul 22 21:13:08.952912 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:08.952897 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540htrrj" event={"ID":"fd0c7ad5-77a4-4d49-b884-7d9eca964947","Type":"ContainerDied","Data":"e35f955ebc6a058e470c7591968bd0640a0681bff7aa2b61191170797c454515"} Jul 22 21:13:10.082037 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:10.082013 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540htrrj" Jul 22 21:13:10.191022 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:10.190988 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fd0c7ad5-77a4-4d49-b884-7d9eca964947-bundle\") pod \"fd0c7ad5-77a4-4d49-b884-7d9eca964947\" (UID: \"fd0c7ad5-77a4-4d49-b884-7d9eca964947\") " Jul 22 21:13:10.191187 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:10.191066 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gss5v\" (UniqueName: \"kubernetes.io/projected/fd0c7ad5-77a4-4d49-b884-7d9eca964947-kube-api-access-gss5v\") pod \"fd0c7ad5-77a4-4d49-b884-7d9eca964947\" (UID: \"fd0c7ad5-77a4-4d49-b884-7d9eca964947\") " Jul 22 21:13:10.191187 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:10.191101 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fd0c7ad5-77a4-4d49-b884-7d9eca964947-util\") pod \"fd0c7ad5-77a4-4d49-b884-7d9eca964947\" (UID: \"fd0c7ad5-77a4-4d49-b884-7d9eca964947\") " Jul 22 21:13:10.191805 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:10.191777 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd0c7ad5-77a4-4d49-b884-7d9eca964947-bundle" (OuterVolumeSpecName: "bundle") pod "fd0c7ad5-77a4-4d49-b884-7d9eca964947" (UID: "fd0c7ad5-77a4-4d49-b884-7d9eca964947"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 21:13:10.193408 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:10.193386 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd0c7ad5-77a4-4d49-b884-7d9eca964947-kube-api-access-gss5v" (OuterVolumeSpecName: "kube-api-access-gss5v") pod "fd0c7ad5-77a4-4d49-b884-7d9eca964947" (UID: "fd0c7ad5-77a4-4d49-b884-7d9eca964947"). InnerVolumeSpecName "kube-api-access-gss5v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 21:13:10.195434 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:10.195406 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd0c7ad5-77a4-4d49-b884-7d9eca964947-util" (OuterVolumeSpecName: "util") pod "fd0c7ad5-77a4-4d49-b884-7d9eca964947" (UID: "fd0c7ad5-77a4-4d49-b884-7d9eca964947"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 21:13:10.291997 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:10.291906 2572 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fd0c7ad5-77a4-4d49-b884-7d9eca964947-bundle\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:13:10.291997 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:10.291942 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gss5v\" (UniqueName: \"kubernetes.io/projected/fd0c7ad5-77a4-4d49-b884-7d9eca964947-kube-api-access-gss5v\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:13:10.291997 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:10.291952 2572 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fd0c7ad5-77a4-4d49-b884-7d9eca964947-util\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:13:10.967728 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:10.967700 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540htrrj" Jul 22 21:13:10.967868 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:10.967701 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540htrrj" event={"ID":"fd0c7ad5-77a4-4d49-b884-7d9eca964947","Type":"ContainerDied","Data":"2ffb0f8f89721d6e85643c4d13505c42587ffa2707a927e8ed26dc6847b5c804"} Jul 22 21:13:10.967868 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:10.967838 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ffb0f8f89721d6e85643c4d13505c42587ffa2707a927e8ed26dc6847b5c804" Jul 22 21:13:17.914303 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:17.914273 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7jqdf"] Jul 22 21:13:17.914765 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:17.914718 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd0c7ad5-77a4-4d49-b884-7d9eca964947" containerName="pull" Jul 22 21:13:17.914765 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:17.914735 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd0c7ad5-77a4-4d49-b884-7d9eca964947" containerName="pull" Jul 22 21:13:17.914765 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:17.914746 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd0c7ad5-77a4-4d49-b884-7d9eca964947" containerName="extract" Jul 22 21:13:17.914765 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:17.914753 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd0c7ad5-77a4-4d49-b884-7d9eca964947" containerName="extract" Jul 22 21:13:17.914965 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:17.914796 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd0c7ad5-77a4-4d49-b884-7d9eca964947" containerName="util" Jul 22 21:13:17.914965 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:17.914806 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd0c7ad5-77a4-4d49-b884-7d9eca964947" containerName="util" Jul 22 21:13:17.914965 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:17.914879 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd0c7ad5-77a4-4d49-b884-7d9eca964947" containerName="extract" Jul 22 21:13:17.945508 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:17.945483 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7jqdf"] Jul 22 21:13:17.945678 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:17.945595 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7jqdf" Jul 22 21:13:17.950634 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:17.950610 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 22 21:13:17.950831 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:17.950814 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 22 21:13:17.950908 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:17.950876 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-g7xgg\"" Jul 22 21:13:18.055797 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:18.055761 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b98r2\" (UniqueName: \"kubernetes.io/projected/a77296b4-ac5f-4a9e-a645-ca1891ae1241-kube-api-access-b98r2\") pod \"cert-manager-operator-controller-manager-55c444bf86-7jqdf\" (UID: \"a77296b4-ac5f-4a9e-a645-ca1891ae1241\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7jqdf" Jul 22 21:13:18.055969 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:18.055815 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a77296b4-ac5f-4a9e-a645-ca1891ae1241-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-7jqdf\" (UID: \"a77296b4-ac5f-4a9e-a645-ca1891ae1241\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7jqdf" Jul 22 21:13:18.156695 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:18.156659 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b98r2\" (UniqueName: \"kubernetes.io/projected/a77296b4-ac5f-4a9e-a645-ca1891ae1241-kube-api-access-b98r2\") pod \"cert-manager-operator-controller-manager-55c444bf86-7jqdf\" (UID: \"a77296b4-ac5f-4a9e-a645-ca1891ae1241\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7jqdf" Jul 22 21:13:18.156854 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:18.156714 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a77296b4-ac5f-4a9e-a645-ca1891ae1241-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-7jqdf\" (UID: \"a77296b4-ac5f-4a9e-a645-ca1891ae1241\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7jqdf" Jul 22 21:13:18.157103 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:18.157084 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a77296b4-ac5f-4a9e-a645-ca1891ae1241-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-7jqdf\" (UID: \"a77296b4-ac5f-4a9e-a645-ca1891ae1241\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7jqdf" Jul 22 21:13:18.164839 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:18.164792 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b98r2\" (UniqueName: \"kubernetes.io/projected/a77296b4-ac5f-4a9e-a645-ca1891ae1241-kube-api-access-b98r2\") pod \"cert-manager-operator-controller-manager-55c444bf86-7jqdf\" (UID: \"a77296b4-ac5f-4a9e-a645-ca1891ae1241\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7jqdf" Jul 22 21:13:18.255728 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:18.255698 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7jqdf" Jul 22 21:13:18.413764 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:18.413740 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7jqdf"] Jul 22 21:13:18.416666 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:13:18.416606 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda77296b4_ac5f_4a9e_a645_ca1891ae1241.slice/crio-84775d6685988e821445d3a8b3358a5960aeb1e0d0bfe0b000da8bb74af18e86 WatchSource:0}: Error finding container 84775d6685988e821445d3a8b3358a5960aeb1e0d0bfe0b000da8bb74af18e86: Status 404 returned error can't find the container with id 84775d6685988e821445d3a8b3358a5960aeb1e0d0bfe0b000da8bb74af18e86 Jul 22 21:13:19.002220 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:19.002181 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7jqdf" event={"ID":"a77296b4-ac5f-4a9e-a645-ca1891ae1241","Type":"ContainerStarted","Data":"84775d6685988e821445d3a8b3358a5960aeb1e0d0bfe0b000da8bb74af18e86"} Jul 22 21:13:21.014563 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:21.014528 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7jqdf" event={"ID":"a77296b4-ac5f-4a9e-a645-ca1891ae1241","Type":"ContainerStarted","Data":"2c26e03f6a8c3b664bcfee05a3b6e0633a5d01dcbd7bb759ca4c1214bebf9f3f"} Jul 22 21:13:21.036892 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:21.036839 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7jqdf" podStartSLOduration=2.399705423 podStartE2EDuration="4.036822646s" totalImagesPullingTime="1.637117223s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 21:13:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 21:13:21.034506153 +0000 UTC m=+456.148898520" watchObservedRunningTime="2026-07-22 21:13:21.036822646 +0000 UTC m=+456.151214990" Jul 22 21:13:22.265887 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:22.265851 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-ps2tk"] Jul 22 21:13:22.268504 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:22.268483 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-ps2tk" Jul 22 21:13:22.271062 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:22.271041 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 22 21:13:22.271915 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:22.271896 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 22 21:13:22.271994 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:22.271951 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-779jj\"" Jul 22 21:13:22.277118 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:22.277087 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-ps2tk"] Jul 22 21:13:22.392897 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:22.392862 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr9h6\" (UniqueName: \"kubernetes.io/projected/be55a0b7-5c73-4209-a106-fb56cc96f536-kube-api-access-dr9h6\") pod \"cert-manager-webhook-6486ffd997-ps2tk\" (UID: \"be55a0b7-5c73-4209-a106-fb56cc96f536\") " pod="cert-manager/cert-manager-webhook-6486ffd997-ps2tk" Jul 22 21:13:22.393075 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:22.392905 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/be55a0b7-5c73-4209-a106-fb56cc96f536-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-ps2tk\" (UID: \"be55a0b7-5c73-4209-a106-fb56cc96f536\") " pod="cert-manager/cert-manager-webhook-6486ffd997-ps2tk" Jul 22 21:13:22.494385 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:22.494343 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr9h6\" (UniqueName: \"kubernetes.io/projected/be55a0b7-5c73-4209-a106-fb56cc96f536-kube-api-access-dr9h6\") pod \"cert-manager-webhook-6486ffd997-ps2tk\" (UID: \"be55a0b7-5c73-4209-a106-fb56cc96f536\") " pod="cert-manager/cert-manager-webhook-6486ffd997-ps2tk" Jul 22 21:13:22.494539 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:22.494393 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/be55a0b7-5c73-4209-a106-fb56cc96f536-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-ps2tk\" (UID: \"be55a0b7-5c73-4209-a106-fb56cc96f536\") " pod="cert-manager/cert-manager-webhook-6486ffd997-ps2tk" Jul 22 21:13:22.503023 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:22.502986 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr9h6\" (UniqueName: \"kubernetes.io/projected/be55a0b7-5c73-4209-a106-fb56cc96f536-kube-api-access-dr9h6\") pod \"cert-manager-webhook-6486ffd997-ps2tk\" (UID: \"be55a0b7-5c73-4209-a106-fb56cc96f536\") " pod="cert-manager/cert-manager-webhook-6486ffd997-ps2tk" Jul 22 21:13:22.503139 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:22.503066 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/be55a0b7-5c73-4209-a106-fb56cc96f536-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-ps2tk\" (UID: \"be55a0b7-5c73-4209-a106-fb56cc96f536\") " pod="cert-manager/cert-manager-webhook-6486ffd997-ps2tk" Jul 22 21:13:22.578481 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:22.578454 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-ps2tk" Jul 22 21:13:22.712657 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:22.712634 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-ps2tk"] Jul 22 21:13:22.715083 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:13:22.715062 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbe55a0b7_5c73_4209_a106_fb56cc96f536.slice/crio-72422fdf6de3882f74182c45082100429fb292cc8f096fcefb8126c05c78abf4 WatchSource:0}: Error finding container 72422fdf6de3882f74182c45082100429fb292cc8f096fcefb8126c05c78abf4: Status 404 returned error can't find the container with id 72422fdf6de3882f74182c45082100429fb292cc8f096fcefb8126c05c78abf4 Jul 22 21:13:23.024181 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:23.024086 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-ps2tk" event={"ID":"be55a0b7-5c73-4209-a106-fb56cc96f536","Type":"ContainerStarted","Data":"72422fdf6de3882f74182c45082100429fb292cc8f096fcefb8126c05c78abf4"} Jul 22 21:13:23.697583 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:23.697543 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-p7t5j"] Jul 22 21:13:23.700561 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:23.700544 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-p7t5j" Jul 22 21:13:23.702860 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:23.702839 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-j5m8g\"" Jul 22 21:13:23.710519 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:23.710495 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-p7t5j"] Jul 22 21:13:23.807465 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:23.807429 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b6a46bca-b8a2-41a4-9cb7-a2f6c4091168-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-p7t5j\" (UID: \"b6a46bca-b8a2-41a4-9cb7-a2f6c4091168\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-p7t5j" Jul 22 21:13:23.807465 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:23.807462 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw8ss\" (UniqueName: \"kubernetes.io/projected/b6a46bca-b8a2-41a4-9cb7-a2f6c4091168-kube-api-access-fw8ss\") pod \"cert-manager-cainjector-559cdd8c87-p7t5j\" (UID: \"b6a46bca-b8a2-41a4-9cb7-a2f6c4091168\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-p7t5j" Jul 22 21:13:23.908549 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:23.908518 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b6a46bca-b8a2-41a4-9cb7-a2f6c4091168-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-p7t5j\" (UID: \"b6a46bca-b8a2-41a4-9cb7-a2f6c4091168\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-p7t5j" Jul 22 21:13:23.908549 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:23.908552 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw8ss\" (UniqueName: \"kubernetes.io/projected/b6a46bca-b8a2-41a4-9cb7-a2f6c4091168-kube-api-access-fw8ss\") pod \"cert-manager-cainjector-559cdd8c87-p7t5j\" (UID: \"b6a46bca-b8a2-41a4-9cb7-a2f6c4091168\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-p7t5j" Jul 22 21:13:23.916508 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:23.916480 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b6a46bca-b8a2-41a4-9cb7-a2f6c4091168-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-p7t5j\" (UID: \"b6a46bca-b8a2-41a4-9cb7-a2f6c4091168\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-p7t5j" Jul 22 21:13:23.916612 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:23.916544 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw8ss\" (UniqueName: \"kubernetes.io/projected/b6a46bca-b8a2-41a4-9cb7-a2f6c4091168-kube-api-access-fw8ss\") pod \"cert-manager-cainjector-559cdd8c87-p7t5j\" (UID: \"b6a46bca-b8a2-41a4-9cb7-a2f6c4091168\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-p7t5j" Jul 22 21:13:24.010004 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:24.009934 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-p7t5j" Jul 22 21:13:24.136365 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:24.136234 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-p7t5j"] Jul 22 21:13:24.138837 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:13:24.138807 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb6a46bca_b8a2_41a4_9cb7_a2f6c4091168.slice/crio-0023e17451d13e804514744b09d6f9e6e6da24997fb60e4f26a81379bef36fd2 WatchSource:0}: Error finding container 0023e17451d13e804514744b09d6f9e6e6da24997fb60e4f26a81379bef36fd2: Status 404 returned error can't find the container with id 0023e17451d13e804514744b09d6f9e6e6da24997fb60e4f26a81379bef36fd2 Jul 22 21:13:25.035453 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:25.035416 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-p7t5j" event={"ID":"b6a46bca-b8a2-41a4-9cb7-a2f6c4091168","Type":"ContainerStarted","Data":"0023e17451d13e804514744b09d6f9e6e6da24997fb60e4f26a81379bef36fd2"} Jul 22 21:13:29.056114 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:29.056081 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-p7t5j" event={"ID":"b6a46bca-b8a2-41a4-9cb7-a2f6c4091168","Type":"ContainerStarted","Data":"22351170d2154f8963ee67f3828a2fef5001c179471d18aa63d485f9c3b67e3a"} Jul 22 21:13:29.072687 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:29.072645 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-p7t5j" podStartSLOduration=2.159734967 podStartE2EDuration="6.072630493s" totalImagesPullingTime="3.912895526s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 21:13:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 21:13:29.072370436 +0000 UTC m=+464.186762780" watchObservedRunningTime="2026-07-22 21:13:29.072630493 +0000 UTC m=+464.187022848" Jul 22 21:13:44.128060 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:44.128021 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-ps2tk" event={"ID":"be55a0b7-5c73-4209-a106-fb56cc96f536","Type":"ContainerStarted","Data":"e9b5dbdcb4c8956ba5fd5795c6c5a394bc8cc993f7b603027a65f12dd5db1f53"} Jul 22 21:13:44.128540 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:44.128093 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-ps2tk" Jul 22 21:13:44.144773 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:44.144729 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-ps2tk" podStartSLOduration=1.685899706 podStartE2EDuration="22.14471393s" totalImagesPullingTime="20.458814224s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 21:13:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 21:13:44.143223393 +0000 UTC m=+479.257615740" watchObservedRunningTime="2026-07-22 21:13:44.14471393 +0000 UTC m=+479.259106274" Jul 22 21:13:50.135870 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:50.135837 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-ps2tk" Jul 22 21:13:51.312780 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:51.312745 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvmgpl"] Jul 22 21:13:51.317799 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:51.317780 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvmgpl" Jul 22 21:13:51.320226 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:51.320196 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-fqwnq\"" Jul 22 21:13:51.320321 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:51.320251 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 22 21:13:51.320321 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:51.320260 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 22 21:13:51.324810 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:51.324771 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvmgpl"] Jul 22 21:13:51.433323 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:51.433298 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ea19c76e-9f96-44fc-ad75-39a44e147503-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvmgpl\" (UID: \"ea19c76e-9f96-44fc-ad75-39a44e147503\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvmgpl" Jul 22 21:13:51.433323 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:51.433327 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfnd4\" (UniqueName: \"kubernetes.io/projected/ea19c76e-9f96-44fc-ad75-39a44e147503-kube-api-access-sfnd4\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvmgpl\" (UID: \"ea19c76e-9f96-44fc-ad75-39a44e147503\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvmgpl" Jul 22 21:13:51.433531 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:51.433371 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ea19c76e-9f96-44fc-ad75-39a44e147503-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvmgpl\" (UID: \"ea19c76e-9f96-44fc-ad75-39a44e147503\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvmgpl" Jul 22 21:13:51.534400 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:51.534365 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ea19c76e-9f96-44fc-ad75-39a44e147503-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvmgpl\" (UID: \"ea19c76e-9f96-44fc-ad75-39a44e147503\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvmgpl" Jul 22 21:13:51.534400 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:51.534402 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfnd4\" (UniqueName: \"kubernetes.io/projected/ea19c76e-9f96-44fc-ad75-39a44e147503-kube-api-access-sfnd4\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvmgpl\" (UID: \"ea19c76e-9f96-44fc-ad75-39a44e147503\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvmgpl" Jul 22 21:13:51.534583 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:51.534469 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ea19c76e-9f96-44fc-ad75-39a44e147503-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvmgpl\" (UID: \"ea19c76e-9f96-44fc-ad75-39a44e147503\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvmgpl" Jul 22 21:13:51.534915 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:51.534894 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ea19c76e-9f96-44fc-ad75-39a44e147503-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvmgpl\" (UID: \"ea19c76e-9f96-44fc-ad75-39a44e147503\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvmgpl" Jul 22 21:13:51.534954 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:51.534905 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ea19c76e-9f96-44fc-ad75-39a44e147503-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvmgpl\" (UID: \"ea19c76e-9f96-44fc-ad75-39a44e147503\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvmgpl" Jul 22 21:13:51.543405 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:51.543380 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfnd4\" (UniqueName: \"kubernetes.io/projected/ea19c76e-9f96-44fc-ad75-39a44e147503-kube-api-access-sfnd4\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvmgpl\" (UID: \"ea19c76e-9f96-44fc-ad75-39a44e147503\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvmgpl" Jul 22 21:13:51.627700 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:51.627626 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvmgpl" Jul 22 21:13:51.760483 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:51.760452 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvmgpl"] Jul 22 21:13:51.764154 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:13:51.764123 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea19c76e_9f96_44fc_ad75_39a44e147503.slice/crio-6e2456ce1b90dd9a1a559ecaf06483f50e73c769fe265c44954083f6dbeeb728 WatchSource:0}: Error finding container 6e2456ce1b90dd9a1a559ecaf06483f50e73c769fe265c44954083f6dbeeb728: Status 404 returned error can't find the container with id 6e2456ce1b90dd9a1a559ecaf06483f50e73c769fe265c44954083f6dbeeb728 Jul 22 21:13:52.167778 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:52.167747 2572 generic.go:356] "Generic (PLEG): container finished" podID="ea19c76e-9f96-44fc-ad75-39a44e147503" containerID="036c2da1a7076cc365cab8acd0d303d52541e84ab54ea6df0e58aafaca8d4e4b" exitCode=0 Jul 22 21:13:52.167925 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:52.167840 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvmgpl" event={"ID":"ea19c76e-9f96-44fc-ad75-39a44e147503","Type":"ContainerDied","Data":"036c2da1a7076cc365cab8acd0d303d52541e84ab54ea6df0e58aafaca8d4e4b"} Jul 22 21:13:52.167925 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:52.167901 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvmgpl" event={"ID":"ea19c76e-9f96-44fc-ad75-39a44e147503","Type":"ContainerStarted","Data":"6e2456ce1b90dd9a1a559ecaf06483f50e73c769fe265c44954083f6dbeeb728"} Jul 22 21:13:53.174013 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:53.173928 2572 generic.go:356] "Generic (PLEG): container finished" podID="ea19c76e-9f96-44fc-ad75-39a44e147503" containerID="239b22774592d9f5e36156413ec7d2a01ede26378676285df91e9fc954614f27" exitCode=0 Jul 22 21:13:53.174013 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:53.173964 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvmgpl" event={"ID":"ea19c76e-9f96-44fc-ad75-39a44e147503","Type":"ContainerDied","Data":"239b22774592d9f5e36156413ec7d2a01ede26378676285df91e9fc954614f27"} Jul 22 21:13:54.180500 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:54.180463 2572 generic.go:356] "Generic (PLEG): container finished" podID="ea19c76e-9f96-44fc-ad75-39a44e147503" containerID="56cd787cd73d570abfde1ed5cfa444c9d130f0b131c0bdb1b7ce4e7ad9e2ca96" exitCode=0 Jul 22 21:13:54.180882 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:54.180546 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvmgpl" event={"ID":"ea19c76e-9f96-44fc-ad75-39a44e147503","Type":"ContainerDied","Data":"56cd787cd73d570abfde1ed5cfa444c9d130f0b131c0bdb1b7ce4e7ad9e2ca96"} Jul 22 21:13:55.314036 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:55.314011 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvmgpl" Jul 22 21:13:55.368936 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:55.368899 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sfnd4\" (UniqueName: \"kubernetes.io/projected/ea19c76e-9f96-44fc-ad75-39a44e147503-kube-api-access-sfnd4\") pod \"ea19c76e-9f96-44fc-ad75-39a44e147503\" (UID: \"ea19c76e-9f96-44fc-ad75-39a44e147503\") " Jul 22 21:13:55.369120 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:55.368949 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ea19c76e-9f96-44fc-ad75-39a44e147503-bundle\") pod \"ea19c76e-9f96-44fc-ad75-39a44e147503\" (UID: \"ea19c76e-9f96-44fc-ad75-39a44e147503\") " Jul 22 21:13:55.369120 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:55.368998 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ea19c76e-9f96-44fc-ad75-39a44e147503-util\") pod \"ea19c76e-9f96-44fc-ad75-39a44e147503\" (UID: \"ea19c76e-9f96-44fc-ad75-39a44e147503\") " Jul 22 21:13:55.369742 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:55.369715 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea19c76e-9f96-44fc-ad75-39a44e147503-bundle" (OuterVolumeSpecName: "bundle") pod "ea19c76e-9f96-44fc-ad75-39a44e147503" (UID: "ea19c76e-9f96-44fc-ad75-39a44e147503"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 21:13:55.372069 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:55.372036 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea19c76e-9f96-44fc-ad75-39a44e147503-kube-api-access-sfnd4" (OuterVolumeSpecName: "kube-api-access-sfnd4") pod "ea19c76e-9f96-44fc-ad75-39a44e147503" (UID: "ea19c76e-9f96-44fc-ad75-39a44e147503"). InnerVolumeSpecName "kube-api-access-sfnd4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 21:13:55.375012 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:55.374990 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea19c76e-9f96-44fc-ad75-39a44e147503-util" (OuterVolumeSpecName: "util") pod "ea19c76e-9f96-44fc-ad75-39a44e147503" (UID: "ea19c76e-9f96-44fc-ad75-39a44e147503"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 21:13:55.470363 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:55.470284 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfnd4\" (UniqueName: \"kubernetes.io/projected/ea19c76e-9f96-44fc-ad75-39a44e147503-kube-api-access-sfnd4\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:13:55.470363 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:55.470312 2572 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ea19c76e-9f96-44fc-ad75-39a44e147503-bundle\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:13:55.470363 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:55.470320 2572 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ea19c76e-9f96-44fc-ad75-39a44e147503-util\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:13:55.480290 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:55.480265 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-795d667875-4klxv"] Jul 22 21:13:55.480574 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:55.480563 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea19c76e-9f96-44fc-ad75-39a44e147503" containerName="pull" Jul 22 21:13:55.480617 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:55.480576 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea19c76e-9f96-44fc-ad75-39a44e147503" containerName="pull" Jul 22 21:13:55.480617 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:55.480596 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea19c76e-9f96-44fc-ad75-39a44e147503" containerName="extract" Jul 22 21:13:55.480617 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:55.480604 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea19c76e-9f96-44fc-ad75-39a44e147503" containerName="extract" Jul 22 21:13:55.480617 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:55.480612 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea19c76e-9f96-44fc-ad75-39a44e147503" containerName="util" Jul 22 21:13:55.480617 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:55.480617 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea19c76e-9f96-44fc-ad75-39a44e147503" containerName="util" Jul 22 21:13:55.480760 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:55.480664 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea19c76e-9f96-44fc-ad75-39a44e147503" containerName="extract" Jul 22 21:13:55.483653 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:55.483639 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-795d667875-4klxv" Jul 22 21:13:55.486607 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:55.486579 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-hrbzb\"" Jul 22 21:13:55.486753 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:55.486730 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 22 21:13:55.486851 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:55.486782 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 22 21:13:55.486851 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:55.486786 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 22 21:13:55.486851 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:55.486796 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 22 21:13:55.486851 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:55.486783 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 22 21:13:55.496351 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:55.496323 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-795d667875-4klxv"] Jul 22 21:13:55.571380 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:55.571343 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwfsk\" (UniqueName: \"kubernetes.io/projected/0e92181f-ccd3-476e-91bd-fb2bdc76b00a-kube-api-access-pwfsk\") pod \"lws-controller-manager-795d667875-4klxv\" (UID: \"0e92181f-ccd3-476e-91bd-fb2bdc76b00a\") " pod="openshift-lws-operator/lws-controller-manager-795d667875-4klxv" Jul 22 21:13:55.571554 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:55.571396 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0e92181f-ccd3-476e-91bd-fb2bdc76b00a-cert\") pod \"lws-controller-manager-795d667875-4klxv\" (UID: \"0e92181f-ccd3-476e-91bd-fb2bdc76b00a\") " pod="openshift-lws-operator/lws-controller-manager-795d667875-4klxv" Jul 22 21:13:55.571554 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:55.571421 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0e92181f-ccd3-476e-91bd-fb2bdc76b00a-manager-config\") pod \"lws-controller-manager-795d667875-4klxv\" (UID: \"0e92181f-ccd3-476e-91bd-fb2bdc76b00a\") " pod="openshift-lws-operator/lws-controller-manager-795d667875-4klxv" Jul 22 21:13:55.571554 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:55.571437 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0e92181f-ccd3-476e-91bd-fb2bdc76b00a-metrics-cert\") pod \"lws-controller-manager-795d667875-4klxv\" (UID: \"0e92181f-ccd3-476e-91bd-fb2bdc76b00a\") " pod="openshift-lws-operator/lws-controller-manager-795d667875-4klxv" Jul 22 21:13:55.672280 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:55.672240 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwfsk\" (UniqueName: \"kubernetes.io/projected/0e92181f-ccd3-476e-91bd-fb2bdc76b00a-kube-api-access-pwfsk\") pod \"lws-controller-manager-795d667875-4klxv\" (UID: \"0e92181f-ccd3-476e-91bd-fb2bdc76b00a\") " pod="openshift-lws-operator/lws-controller-manager-795d667875-4klxv" Jul 22 21:13:55.672454 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:55.672293 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0e92181f-ccd3-476e-91bd-fb2bdc76b00a-cert\") pod \"lws-controller-manager-795d667875-4klxv\" (UID: \"0e92181f-ccd3-476e-91bd-fb2bdc76b00a\") " pod="openshift-lws-operator/lws-controller-manager-795d667875-4klxv" Jul 22 21:13:55.672454 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:55.672331 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0e92181f-ccd3-476e-91bd-fb2bdc76b00a-manager-config\") pod \"lws-controller-manager-795d667875-4klxv\" (UID: \"0e92181f-ccd3-476e-91bd-fb2bdc76b00a\") " pod="openshift-lws-operator/lws-controller-manager-795d667875-4klxv" Jul 22 21:13:55.672454 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:55.672347 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0e92181f-ccd3-476e-91bd-fb2bdc76b00a-metrics-cert\") pod \"lws-controller-manager-795d667875-4klxv\" (UID: \"0e92181f-ccd3-476e-91bd-fb2bdc76b00a\") " pod="openshift-lws-operator/lws-controller-manager-795d667875-4klxv" Jul 22 21:13:55.672955 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:55.672935 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0e92181f-ccd3-476e-91bd-fb2bdc76b00a-manager-config\") pod \"lws-controller-manager-795d667875-4klxv\" (UID: \"0e92181f-ccd3-476e-91bd-fb2bdc76b00a\") " pod="openshift-lws-operator/lws-controller-manager-795d667875-4klxv" Jul 22 21:13:55.675063 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:55.675038 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0e92181f-ccd3-476e-91bd-fb2bdc76b00a-cert\") pod \"lws-controller-manager-795d667875-4klxv\" (UID: \"0e92181f-ccd3-476e-91bd-fb2bdc76b00a\") " pod="openshift-lws-operator/lws-controller-manager-795d667875-4klxv" Jul 22 21:13:55.675201 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:55.675092 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0e92181f-ccd3-476e-91bd-fb2bdc76b00a-metrics-cert\") pod \"lws-controller-manager-795d667875-4klxv\" (UID: \"0e92181f-ccd3-476e-91bd-fb2bdc76b00a\") " pod="openshift-lws-operator/lws-controller-manager-795d667875-4klxv" Jul 22 21:13:55.690470 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:55.690444 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwfsk\" (UniqueName: \"kubernetes.io/projected/0e92181f-ccd3-476e-91bd-fb2bdc76b00a-kube-api-access-pwfsk\") pod \"lws-controller-manager-795d667875-4klxv\" (UID: \"0e92181f-ccd3-476e-91bd-fb2bdc76b00a\") " pod="openshift-lws-operator/lws-controller-manager-795d667875-4klxv" Jul 22 21:13:55.792621 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:55.792517 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-795d667875-4klxv" Jul 22 21:13:55.947183 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:55.947134 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-795d667875-4klxv"] Jul 22 21:13:55.950200 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:13:55.950157 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e92181f_ccd3_476e_91bd_fb2bdc76b00a.slice/crio-0db1f2c9f44c385a2308e6e4db0f3ae79dbee1407a71f02c7b244be62ec514fd WatchSource:0}: Error finding container 0db1f2c9f44c385a2308e6e4db0f3ae79dbee1407a71f02c7b244be62ec514fd: Status 404 returned error can't find the container with id 0db1f2c9f44c385a2308e6e4db0f3ae79dbee1407a71f02c7b244be62ec514fd Jul 22 21:13:56.192752 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:56.192725 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvmgpl" Jul 22 21:13:56.192947 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:56.192762 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvmgpl" event={"ID":"ea19c76e-9f96-44fc-ad75-39a44e147503","Type":"ContainerDied","Data":"6e2456ce1b90dd9a1a559ecaf06483f50e73c769fe265c44954083f6dbeeb728"} Jul 22 21:13:56.192947 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:56.192813 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e2456ce1b90dd9a1a559ecaf06483f50e73c769fe265c44954083f6dbeeb728" Jul 22 21:13:56.194068 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:56.194046 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-795d667875-4klxv" event={"ID":"0e92181f-ccd3-476e-91bd-fb2bdc76b00a","Type":"ContainerStarted","Data":"0db1f2c9f44c385a2308e6e4db0f3ae79dbee1407a71f02c7b244be62ec514fd"} Jul 22 21:13:59.210720 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:59.210631 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-795d667875-4klxv" event={"ID":"0e92181f-ccd3-476e-91bd-fb2bdc76b00a","Type":"ContainerStarted","Data":"35ef5efed909ea6cd9f85714f1b6386acffc6aac4d0771ce8912d2cb4f5d76e2"} Jul 22 21:13:59.211052 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:59.210727 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-795d667875-4klxv" Jul 22 21:13:59.245452 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:13:59.240067 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-795d667875-4klxv" podStartSLOduration=1.8160128709999999 podStartE2EDuration="4.24005243s" totalImagesPullingTime="2.424039559s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 21:13:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 21:13:59.239604254 +0000 UTC m=+494.353996611" watchObservedRunningTime="2026-07-22 21:13:59.24005243 +0000 UTC m=+494.354444773" Jul 22 21:14:06.093478 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:06.093441 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5b88fc4594-pc9qd"] Jul 22 21:14:06.096213 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:06.096197 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5b88fc4594-pc9qd" Jul 22 21:14:06.099012 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:06.098991 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 22 21:14:06.099128 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:06.098994 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 22 21:14:06.099128 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:06.099028 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 22 21:14:06.099256 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:06.099157 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 22 21:14:06.099256 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:06.099214 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-q65kp\"" Jul 22 21:14:06.110883 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:06.110860 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5b88fc4594-pc9qd"] Jul 22 21:14:06.162078 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:06.162044 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b016459c-2f29-4fa9-9681-3e320ad43e7a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5b88fc4594-pc9qd\" (UID: \"b016459c-2f29-4fa9-9681-3e320ad43e7a\") " pod="opendatahub/opendatahub-operator-controller-manager-5b88fc4594-pc9qd" Jul 22 21:14:06.162260 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:06.162088 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvrw4\" (UniqueName: \"kubernetes.io/projected/b016459c-2f29-4fa9-9681-3e320ad43e7a-kube-api-access-vvrw4\") pod \"opendatahub-operator-controller-manager-5b88fc4594-pc9qd\" (UID: \"b016459c-2f29-4fa9-9681-3e320ad43e7a\") " pod="opendatahub/opendatahub-operator-controller-manager-5b88fc4594-pc9qd" Jul 22 21:14:06.162260 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:06.162120 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b016459c-2f29-4fa9-9681-3e320ad43e7a-manifests\") pod \"opendatahub-operator-controller-manager-5b88fc4594-pc9qd\" (UID: \"b016459c-2f29-4fa9-9681-3e320ad43e7a\") " pod="opendatahub/opendatahub-operator-controller-manager-5b88fc4594-pc9qd" Jul 22 21:14:06.162260 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:06.162147 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b016459c-2f29-4fa9-9681-3e320ad43e7a-tmp\") pod \"opendatahub-operator-controller-manager-5b88fc4594-pc9qd\" (UID: \"b016459c-2f29-4fa9-9681-3e320ad43e7a\") " pod="opendatahub/opendatahub-operator-controller-manager-5b88fc4594-pc9qd" Jul 22 21:14:06.162260 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:06.162213 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b016459c-2f29-4fa9-9681-3e320ad43e7a-webhook-cert\") pod \"opendatahub-operator-controller-manager-5b88fc4594-pc9qd\" (UID: \"b016459c-2f29-4fa9-9681-3e320ad43e7a\") " pod="opendatahub/opendatahub-operator-controller-manager-5b88fc4594-pc9qd" Jul 22 21:14:06.263491 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:06.263459 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b016459c-2f29-4fa9-9681-3e320ad43e7a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5b88fc4594-pc9qd\" (UID: \"b016459c-2f29-4fa9-9681-3e320ad43e7a\") " pod="opendatahub/opendatahub-operator-controller-manager-5b88fc4594-pc9qd" Jul 22 21:14:06.263644 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:06.263501 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvrw4\" (UniqueName: \"kubernetes.io/projected/b016459c-2f29-4fa9-9681-3e320ad43e7a-kube-api-access-vvrw4\") pod \"opendatahub-operator-controller-manager-5b88fc4594-pc9qd\" (UID: \"b016459c-2f29-4fa9-9681-3e320ad43e7a\") " pod="opendatahub/opendatahub-operator-controller-manager-5b88fc4594-pc9qd" Jul 22 21:14:06.263644 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:06.263569 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b016459c-2f29-4fa9-9681-3e320ad43e7a-manifests\") pod \"opendatahub-operator-controller-manager-5b88fc4594-pc9qd\" (UID: \"b016459c-2f29-4fa9-9681-3e320ad43e7a\") " pod="opendatahub/opendatahub-operator-controller-manager-5b88fc4594-pc9qd" Jul 22 21:14:06.263644 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:06.263615 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b016459c-2f29-4fa9-9681-3e320ad43e7a-tmp\") pod \"opendatahub-operator-controller-manager-5b88fc4594-pc9qd\" (UID: \"b016459c-2f29-4fa9-9681-3e320ad43e7a\") " pod="opendatahub/opendatahub-operator-controller-manager-5b88fc4594-pc9qd" Jul 22 21:14:06.263644 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:06.263634 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b016459c-2f29-4fa9-9681-3e320ad43e7a-webhook-cert\") pod \"opendatahub-operator-controller-manager-5b88fc4594-pc9qd\" (UID: \"b016459c-2f29-4fa9-9681-3e320ad43e7a\") " pod="opendatahub/opendatahub-operator-controller-manager-5b88fc4594-pc9qd" Jul 22 21:14:06.264031 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:06.264009 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b016459c-2f29-4fa9-9681-3e320ad43e7a-manifests\") pod \"opendatahub-operator-controller-manager-5b88fc4594-pc9qd\" (UID: \"b016459c-2f29-4fa9-9681-3e320ad43e7a\") " pod="opendatahub/opendatahub-operator-controller-manager-5b88fc4594-pc9qd" Jul 22 21:14:06.266101 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:06.266077 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b016459c-2f29-4fa9-9681-3e320ad43e7a-tmp\") pod \"opendatahub-operator-controller-manager-5b88fc4594-pc9qd\" (UID: \"b016459c-2f29-4fa9-9681-3e320ad43e7a\") " pod="opendatahub/opendatahub-operator-controller-manager-5b88fc4594-pc9qd" Jul 22 21:14:06.266229 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:06.266212 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b016459c-2f29-4fa9-9681-3e320ad43e7a-webhook-cert\") pod \"opendatahub-operator-controller-manager-5b88fc4594-pc9qd\" (UID: \"b016459c-2f29-4fa9-9681-3e320ad43e7a\") " pod="opendatahub/opendatahub-operator-controller-manager-5b88fc4594-pc9qd" Jul 22 21:14:06.266277 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:06.266232 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b016459c-2f29-4fa9-9681-3e320ad43e7a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5b88fc4594-pc9qd\" (UID: \"b016459c-2f29-4fa9-9681-3e320ad43e7a\") " pod="opendatahub/opendatahub-operator-controller-manager-5b88fc4594-pc9qd" Jul 22 21:14:06.272235 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:06.272213 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvrw4\" (UniqueName: \"kubernetes.io/projected/b016459c-2f29-4fa9-9681-3e320ad43e7a-kube-api-access-vvrw4\") pod \"opendatahub-operator-controller-manager-5b88fc4594-pc9qd\" (UID: \"b016459c-2f29-4fa9-9681-3e320ad43e7a\") " pod="opendatahub/opendatahub-operator-controller-manager-5b88fc4594-pc9qd" Jul 22 21:14:06.406708 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:06.406623 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5b88fc4594-pc9qd" Jul 22 21:14:06.543660 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:06.543628 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5b88fc4594-pc9qd"] Jul 22 21:14:06.547113 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:14:06.547085 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb016459c_2f29_4fa9_9681_3e320ad43e7a.slice/crio-789d3d5e9cc7ae1240d581e64f62c2abce3ead9d5143e9e04dca2d6c05e6f881 WatchSource:0}: Error finding container 789d3d5e9cc7ae1240d581e64f62c2abce3ead9d5143e9e04dca2d6c05e6f881: Status 404 returned error can't find the container with id 789d3d5e9cc7ae1240d581e64f62c2abce3ead9d5143e9e04dca2d6c05e6f881 Jul 22 21:14:07.254147 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:07.254078 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5b88fc4594-pc9qd" event={"ID":"b016459c-2f29-4fa9-9681-3e320ad43e7a","Type":"ContainerStarted","Data":"789d3d5e9cc7ae1240d581e64f62c2abce3ead9d5143e9e04dca2d6c05e6f881"} Jul 22 21:14:10.217123 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:10.217084 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-795d667875-4klxv" Jul 22 21:14:10.271440 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:10.271407 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5b88fc4594-pc9qd" event={"ID":"b016459c-2f29-4fa9-9681-3e320ad43e7a","Type":"ContainerStarted","Data":"a48f160f8200a09066691a6d86efd29d099caf72fb35759bb43bbdd9d57a8d5b"} Jul 22 21:14:11.278014 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:11.277978 2572 generic.go:356] "Generic (PLEG): container finished" podID="b016459c-2f29-4fa9-9681-3e320ad43e7a" containerID="a48f160f8200a09066691a6d86efd29d099caf72fb35759bb43bbdd9d57a8d5b" exitCode=0 Jul 22 21:14:11.278406 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:11.278028 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5b88fc4594-pc9qd" event={"ID":"b016459c-2f29-4fa9-9681-3e320ad43e7a","Type":"ContainerDied","Data":"a48f160f8200a09066691a6d86efd29d099caf72fb35759bb43bbdd9d57a8d5b"} Jul 22 21:14:12.284793 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:12.284711 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5b88fc4594-pc9qd" event={"ID":"b016459c-2f29-4fa9-9681-3e320ad43e7a","Type":"ContainerStarted","Data":"d2fe9b406ad52080e831186a4bcf8f68394077d53678bb5189d63709cf27a00a"} Jul 22 21:14:12.284793 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:12.284789 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5b88fc4594-pc9qd" Jul 22 21:14:12.308698 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:12.308660 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5b88fc4594-pc9qd" podStartSLOduration=1.607184584 podStartE2EDuration="6.308645303s" totalImagesPullingTime="3.616594849s" totalInitContainerRuntime="1.08486587s" isStatefulPod=false podCreationTimestamp="2026-07-22 21:14:06 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 21:14:12.307129915 +0000 UTC m=+507.421522290" watchObservedRunningTime="2026-07-22 21:14:12.308645303 +0000 UTC m=+507.423037647" Jul 22 21:14:23.291864 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:23.291821 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5b88fc4594-pc9qd" Jul 22 21:14:33.928286 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:33.928256 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gcskt"] Jul 22 21:14:33.938448 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:33.938426 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gcskt" Jul 22 21:14:33.941039 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:33.941018 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 22 21:14:33.941195 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:33.941072 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-g9shs\"" Jul 22 21:14:33.951533 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:33.951510 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gcskt"] Jul 22 21:14:33.991379 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:33.991347 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0096832-cef8-4070-9566-49b7553d4607-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4gcskt\" (UID: \"a0096832-cef8-4070-9566-49b7553d4607\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gcskt" Jul 22 21:14:33.991379 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:33.991379 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0096832-cef8-4070-9566-49b7553d4607-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4gcskt\" (UID: \"a0096832-cef8-4070-9566-49b7553d4607\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gcskt" Jul 22 21:14:33.991547 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:33.991411 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0096832-cef8-4070-9566-49b7553d4607-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4gcskt\" (UID: \"a0096832-cef8-4070-9566-49b7553d4607\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gcskt" Jul 22 21:14:33.991547 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:33.991448 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcpbr\" (UniqueName: \"kubernetes.io/projected/a0096832-cef8-4070-9566-49b7553d4607-kube-api-access-wcpbr\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4gcskt\" (UID: \"a0096832-cef8-4070-9566-49b7553d4607\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gcskt" Jul 22 21:14:33.991547 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:33.991479 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0096832-cef8-4070-9566-49b7553d4607-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4gcskt\" (UID: \"a0096832-cef8-4070-9566-49b7553d4607\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gcskt" Jul 22 21:14:33.991547 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:33.991525 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0096832-cef8-4070-9566-49b7553d4607-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4gcskt\" (UID: \"a0096832-cef8-4070-9566-49b7553d4607\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gcskt" Jul 22 21:14:33.991671 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:33.991561 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0096832-cef8-4070-9566-49b7553d4607-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4gcskt\" (UID: \"a0096832-cef8-4070-9566-49b7553d4607\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gcskt" Jul 22 21:14:33.991671 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:33.991607 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0096832-cef8-4070-9566-49b7553d4607-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4gcskt\" (UID: \"a0096832-cef8-4070-9566-49b7553d4607\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gcskt" Jul 22 21:14:33.991671 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:33.991621 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0096832-cef8-4070-9566-49b7553d4607-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4gcskt\" (UID: \"a0096832-cef8-4070-9566-49b7553d4607\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gcskt" Jul 22 21:14:34.092378 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:34.092338 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0096832-cef8-4070-9566-49b7553d4607-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4gcskt\" (UID: \"a0096832-cef8-4070-9566-49b7553d4607\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gcskt" Jul 22 21:14:34.092651 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:34.092633 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0096832-cef8-4070-9566-49b7553d4607-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4gcskt\" (UID: \"a0096832-cef8-4070-9566-49b7553d4607\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gcskt" Jul 22 21:14:34.092792 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:34.092779 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0096832-cef8-4070-9566-49b7553d4607-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4gcskt\" (UID: \"a0096832-cef8-4070-9566-49b7553d4607\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gcskt" Jul 22 21:14:34.093316 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:34.093032 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0096832-cef8-4070-9566-49b7553d4607-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4gcskt\" (UID: \"a0096832-cef8-4070-9566-49b7553d4607\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gcskt" Jul 22 21:14:34.093460 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:34.093286 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0096832-cef8-4070-9566-49b7553d4607-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4gcskt\" (UID: \"a0096832-cef8-4070-9566-49b7553d4607\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gcskt" Jul 22 21:14:34.093555 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:34.093284 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0096832-cef8-4070-9566-49b7553d4607-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4gcskt\" (UID: \"a0096832-cef8-4070-9566-49b7553d4607\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gcskt" Jul 22 21:14:34.093656 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:34.093640 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0096832-cef8-4070-9566-49b7553d4607-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4gcskt\" (UID: \"a0096832-cef8-4070-9566-49b7553d4607\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gcskt" Jul 22 21:14:34.093797 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:34.093784 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0096832-cef8-4070-9566-49b7553d4607-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4gcskt\" (UID: \"a0096832-cef8-4070-9566-49b7553d4607\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gcskt" Jul 22 21:14:34.093887 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:34.093874 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0096832-cef8-4070-9566-49b7553d4607-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4gcskt\" (UID: \"a0096832-cef8-4070-9566-49b7553d4607\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gcskt" Jul 22 21:14:34.093986 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:34.093974 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0096832-cef8-4070-9566-49b7553d4607-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4gcskt\" (UID: \"a0096832-cef8-4070-9566-49b7553d4607\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gcskt" Jul 22 21:14:34.094078 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:34.094066 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcpbr\" (UniqueName: \"kubernetes.io/projected/a0096832-cef8-4070-9566-49b7553d4607-kube-api-access-wcpbr\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4gcskt\" (UID: \"a0096832-cef8-4070-9566-49b7553d4607\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gcskt" Jul 22 21:14:34.094309 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:34.094285 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0096832-cef8-4070-9566-49b7553d4607-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4gcskt\" (UID: \"a0096832-cef8-4070-9566-49b7553d4607\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gcskt" Jul 22 21:14:34.094716 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:34.094693 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0096832-cef8-4070-9566-49b7553d4607-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4gcskt\" (UID: \"a0096832-cef8-4070-9566-49b7553d4607\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gcskt" Jul 22 21:14:34.094985 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:34.094968 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0096832-cef8-4070-9566-49b7553d4607-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4gcskt\" (UID: \"a0096832-cef8-4070-9566-49b7553d4607\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gcskt" Jul 22 21:14:34.097484 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:34.097465 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0096832-cef8-4070-9566-49b7553d4607-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4gcskt\" (UID: \"a0096832-cef8-4070-9566-49b7553d4607\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gcskt" Jul 22 21:14:34.097923 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:34.097897 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0096832-cef8-4070-9566-49b7553d4607-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4gcskt\" (UID: \"a0096832-cef8-4070-9566-49b7553d4607\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gcskt" Jul 22 21:14:34.117979 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:34.117951 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcpbr\" (UniqueName: \"kubernetes.io/projected/a0096832-cef8-4070-9566-49b7553d4607-kube-api-access-wcpbr\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4gcskt\" (UID: \"a0096832-cef8-4070-9566-49b7553d4607\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gcskt" Jul 22 21:14:34.120144 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:34.120119 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0096832-cef8-4070-9566-49b7553d4607-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4gcskt\" (UID: \"a0096832-cef8-4070-9566-49b7553d4607\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gcskt" Jul 22 21:14:34.255407 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:34.255327 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gcskt" Jul 22 21:14:34.405140 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:34.405095 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gcskt"] Jul 22 21:14:34.410609 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:14:34.410586 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0096832_cef8_4070_9566_49b7553d4607.slice/crio-f9d8238da8f39b1d5d0b66d620869bf56135652e05601731b580cd667c8c3fdb WatchSource:0}: Error finding container f9d8238da8f39b1d5d0b66d620869bf56135652e05601731b580cd667c8c3fdb: Status 404 returned error can't find the container with id f9d8238da8f39b1d5d0b66d620869bf56135652e05601731b580cd667c8c3fdb Jul 22 21:14:35.397343 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:35.397304 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gcskt" event={"ID":"a0096832-cef8-4070-9566-49b7553d4607","Type":"ContainerStarted","Data":"f9d8238da8f39b1d5d0b66d620869bf56135652e05601731b580cd667c8c3fdb"} Jul 22 21:14:37.079553 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:37.079515 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 22 21:14:37.079817 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:37.079597 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 22 21:14:37.079817 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:37.079628 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 22 21:14:37.234914 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:37.234885 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-2q2tj"] Jul 22 21:14:37.238304 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:37.238274 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2q2tj" Jul 22 21:14:37.241280 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:37.241229 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-qww25\"" Jul 22 21:14:37.242726 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:37.242706 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 22 21:14:37.251556 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:37.251534 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-2q2tj"] Jul 22 21:14:37.322792 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:37.322758 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aadfbe03-82fd-48a8-840d-53c85d26653c-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-2q2tj\" (UID: \"aadfbe03-82fd-48a8-840d-53c85d26653c\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2q2tj" Jul 22 21:14:37.322958 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:37.322820 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg8hc\" (UniqueName: \"kubernetes.io/projected/aadfbe03-82fd-48a8-840d-53c85d26653c-kube-api-access-kg8hc\") pod \"llmisvc-controller-manager-777c6bd7d5-2q2tj\" (UID: \"aadfbe03-82fd-48a8-840d-53c85d26653c\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2q2tj" Jul 22 21:14:37.408380 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:37.408293 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gcskt" event={"ID":"a0096832-cef8-4070-9566-49b7553d4607","Type":"ContainerStarted","Data":"bdd7ce6033649d90f79f2cfaba88def267b767a98b57f86dab8744c2e3670d7f"} Jul 22 21:14:37.423342 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:37.423302 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aadfbe03-82fd-48a8-840d-53c85d26653c-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-2q2tj\" (UID: \"aadfbe03-82fd-48a8-840d-53c85d26653c\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2q2tj" Jul 22 21:14:37.423342 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:37.423343 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg8hc\" (UniqueName: \"kubernetes.io/projected/aadfbe03-82fd-48a8-840d-53c85d26653c-kube-api-access-kg8hc\") pod \"llmisvc-controller-manager-777c6bd7d5-2q2tj\" (UID: \"aadfbe03-82fd-48a8-840d-53c85d26653c\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2q2tj" Jul 22 21:14:37.426001 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:37.425971 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aadfbe03-82fd-48a8-840d-53c85d26653c-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-2q2tj\" (UID: \"aadfbe03-82fd-48a8-840d-53c85d26653c\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2q2tj" Jul 22 21:14:37.437506 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:37.437479 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg8hc\" (UniqueName: \"kubernetes.io/projected/aadfbe03-82fd-48a8-840d-53c85d26653c-kube-api-access-kg8hc\") pod \"llmisvc-controller-manager-777c6bd7d5-2q2tj\" (UID: \"aadfbe03-82fd-48a8-840d-53c85d26653c\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2q2tj" Jul 22 21:14:37.453775 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:37.453722 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gcskt" podStartSLOduration=1.786879022 podStartE2EDuration="4.453708003s" totalImagesPullingTime="2.666828981s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 21:14:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 21:14:37.452594577 +0000 UTC m=+532.566986957" watchObservedRunningTime="2026-07-22 21:14:37.453708003 +0000 UTC m=+532.568100409" Jul 22 21:14:37.549954 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:37.549920 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2q2tj" Jul 22 21:14:37.698271 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:37.698247 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-2q2tj"] Jul 22 21:14:37.701238 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:14:37.701214 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podaadfbe03_82fd_48a8_840d_53c85d26653c.slice/crio-31aebf6c54f4d0faf98e5c22e4ab5423c3bbc1c7cad0a2b97b8c105b7ca5862b WatchSource:0}: Error finding container 31aebf6c54f4d0faf98e5c22e4ab5423c3bbc1c7cad0a2b97b8c105b7ca5862b: Status 404 returned error can't find the container with id 31aebf6c54f4d0faf98e5c22e4ab5423c3bbc1c7cad0a2b97b8c105b7ca5862b Jul 22 21:14:38.256047 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:38.256015 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gcskt" Jul 22 21:14:38.261301 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:38.261274 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gcskt" Jul 22 21:14:38.417630 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:38.417601 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2q2tj" event={"ID":"aadfbe03-82fd-48a8-840d-53c85d26653c","Type":"ContainerStarted","Data":"31aebf6c54f4d0faf98e5c22e4ab5423c3bbc1c7cad0a2b97b8c105b7ca5862b"} Jul 22 21:14:38.417826 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:38.417805 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gcskt" Jul 22 21:14:38.418915 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:38.418893 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gcskt" Jul 22 21:14:41.435252 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:41.435210 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2q2tj" event={"ID":"aadfbe03-82fd-48a8-840d-53c85d26653c","Type":"ContainerStarted","Data":"5e2716880c2ee0136a523be15679f9b1137d3c9f74c55431f8597755eac678a9"} Jul 22 21:14:41.435625 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:41.435333 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2q2tj" Jul 22 21:14:41.451876 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:14:41.451834 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2q2tj" podStartSLOduration=1.48048245 podStartE2EDuration="4.451820961s" totalImagesPullingTime="2.971338511s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 21:14:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 21:14:41.450789328 +0000 UTC m=+536.565181672" watchObservedRunningTime="2026-07-22 21:14:41.451820961 +0000 UTC m=+536.566213305" Jul 22 21:15:12.442590 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:12.442557 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2q2tj" Jul 22 21:15:26.878348 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:26.878303 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-v6nbb"] Jul 22 21:15:26.886201 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:26.886150 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-v6nbb" Jul 22 21:15:26.888851 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:26.888824 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 22 21:15:26.890139 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:26.889918 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 22 21:15:26.890139 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:26.890088 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-v6nbb"] Jul 22 21:15:26.890632 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:26.890606 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-xhrpl\"" Jul 22 21:15:26.951254 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:26.951224 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb5xm\" (UniqueName: \"kubernetes.io/projected/b33c8dc3-39f6-440e-86ab-297792bb50e2-kube-api-access-gb5xm\") pod \"kuadrant-operator-catalog-v6nbb\" (UID: \"b33c8dc3-39f6-440e-86ab-297792bb50e2\") " pod="kuadrant-system/kuadrant-operator-catalog-v6nbb" Jul 22 21:15:27.051867 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:27.051827 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb5xm\" (UniqueName: \"kubernetes.io/projected/b33c8dc3-39f6-440e-86ab-297792bb50e2-kube-api-access-gb5xm\") pod \"kuadrant-operator-catalog-v6nbb\" (UID: \"b33c8dc3-39f6-440e-86ab-297792bb50e2\") " pod="kuadrant-system/kuadrant-operator-catalog-v6nbb" Jul 22 21:15:27.059788 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:27.059757 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb5xm\" (UniqueName: \"kubernetes.io/projected/b33c8dc3-39f6-440e-86ab-297792bb50e2-kube-api-access-gb5xm\") pod \"kuadrant-operator-catalog-v6nbb\" (UID: \"b33c8dc3-39f6-440e-86ab-297792bb50e2\") " pod="kuadrant-system/kuadrant-operator-catalog-v6nbb" Jul 22 21:15:27.198136 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:27.198050 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-v6nbb" Jul 22 21:15:27.240878 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:27.240851 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-v6nbb"] Jul 22 21:15:27.332904 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:27.332875 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-v6nbb"] Jul 22 21:15:27.334537 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:15:27.334516 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb33c8dc3_39f6_440e_86ab_297792bb50e2.slice/crio-60bfd64c9e83bdbd165888e6cb2d18c333268b4691bca54c14ccbb8e580034cf WatchSource:0}: Error finding container 60bfd64c9e83bdbd165888e6cb2d18c333268b4691bca54c14ccbb8e580034cf: Status 404 returned error can't find the container with id 60bfd64c9e83bdbd165888e6cb2d18c333268b4691bca54c14ccbb8e580034cf Jul 22 21:15:27.660021 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:27.659982 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-v6nbb" event={"ID":"b33c8dc3-39f6-440e-86ab-297792bb50e2","Type":"ContainerStarted","Data":"60bfd64c9e83bdbd165888e6cb2d18c333268b4691bca54c14ccbb8e580034cf"} Jul 22 21:15:29.674964 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:29.674924 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-v6nbb" event={"ID":"b33c8dc3-39f6-440e-86ab-297792bb50e2","Type":"ContainerStarted","Data":"1d8447c23970f87eeaddeab30b8901b2ff6149d99ff0260973ee41d4789c6914"} Jul 22 21:15:29.675494 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:29.675018 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-v6nbb" podUID="b33c8dc3-39f6-440e-86ab-297792bb50e2" containerName="registry-server" containerID="cri-o://1d8447c23970f87eeaddeab30b8901b2ff6149d99ff0260973ee41d4789c6914" gracePeriod=2 Jul 22 21:15:29.690437 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:29.690397 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-v6nbb" podStartSLOduration=1.866989238 podStartE2EDuration="3.690379535s" totalImagesPullingTime="1.823390297s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 21:15:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 21:15:29.688984641 +0000 UTC m=+584.803376987" watchObservedRunningTime="2026-07-22 21:15:29.690379535 +0000 UTC m=+584.804771878" Jul 22 21:15:29.929603 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:29.929547 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-v6nbb" Jul 22 21:15:29.972834 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:29.972802 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gb5xm\" (UniqueName: \"kubernetes.io/projected/b33c8dc3-39f6-440e-86ab-297792bb50e2-kube-api-access-gb5xm\") pod \"b33c8dc3-39f6-440e-86ab-297792bb50e2\" (UID: \"b33c8dc3-39f6-440e-86ab-297792bb50e2\") " Jul 22 21:15:29.975017 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:29.974992 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b33c8dc3-39f6-440e-86ab-297792bb50e2-kube-api-access-gb5xm" (OuterVolumeSpecName: "kube-api-access-gb5xm") pod "b33c8dc3-39f6-440e-86ab-297792bb50e2" (UID: "b33c8dc3-39f6-440e-86ab-297792bb50e2"). InnerVolumeSpecName "kube-api-access-gb5xm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 21:15:30.073635 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:30.073603 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gb5xm\" (UniqueName: \"kubernetes.io/projected/b33c8dc3-39f6-440e-86ab-297792bb50e2-kube-api-access-gb5xm\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:15:30.680752 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:30.680721 2572 generic.go:356] "Generic (PLEG): container finished" podID="b33c8dc3-39f6-440e-86ab-297792bb50e2" containerID="1d8447c23970f87eeaddeab30b8901b2ff6149d99ff0260973ee41d4789c6914" exitCode=0 Jul 22 21:15:30.681202 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:30.680768 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-v6nbb" Jul 22 21:15:30.681202 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:30.680808 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-v6nbb" event={"ID":"b33c8dc3-39f6-440e-86ab-297792bb50e2","Type":"ContainerDied","Data":"1d8447c23970f87eeaddeab30b8901b2ff6149d99ff0260973ee41d4789c6914"} Jul 22 21:15:30.681202 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:30.680859 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-v6nbb" event={"ID":"b33c8dc3-39f6-440e-86ab-297792bb50e2","Type":"ContainerDied","Data":"60bfd64c9e83bdbd165888e6cb2d18c333268b4691bca54c14ccbb8e580034cf"} Jul 22 21:15:30.681202 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:30.680879 2572 scope.go:117] "RemoveContainer" containerID="1d8447c23970f87eeaddeab30b8901b2ff6149d99ff0260973ee41d4789c6914" Jul 22 21:15:30.690648 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:30.690627 2572 scope.go:117] "RemoveContainer" containerID="1d8447c23970f87eeaddeab30b8901b2ff6149d99ff0260973ee41d4789c6914" Jul 22 21:15:30.690901 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:15:30.690881 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d8447c23970f87eeaddeab30b8901b2ff6149d99ff0260973ee41d4789c6914\": container with ID starting with 1d8447c23970f87eeaddeab30b8901b2ff6149d99ff0260973ee41d4789c6914 not found: ID does not exist" containerID="1d8447c23970f87eeaddeab30b8901b2ff6149d99ff0260973ee41d4789c6914" Jul 22 21:15:30.690957 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:30.690910 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d8447c23970f87eeaddeab30b8901b2ff6149d99ff0260973ee41d4789c6914"} err="failed to get container status \"1d8447c23970f87eeaddeab30b8901b2ff6149d99ff0260973ee41d4789c6914\": rpc error: code = NotFound desc = could not find container \"1d8447c23970f87eeaddeab30b8901b2ff6149d99ff0260973ee41d4789c6914\": container with ID starting with 1d8447c23970f87eeaddeab30b8901b2ff6149d99ff0260973ee41d4789c6914 not found: ID does not exist" Jul 22 21:15:30.701550 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:30.701510 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-v6nbb"] Jul 22 21:15:30.707749 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:30.707729 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-v6nbb"] Jul 22 21:15:31.383558 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:31.383525 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b33c8dc3-39f6-440e-86ab-297792bb50e2" path="/var/lib/kubelet/pods/b33c8dc3-39f6-440e-86ab-297792bb50e2/volumes" Jul 22 21:15:42.266007 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:42.265971 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rbwgs"] Jul 22 21:15:42.266434 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:42.266395 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b33c8dc3-39f6-440e-86ab-297792bb50e2" containerName="registry-server" Jul 22 21:15:42.266434 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:42.266408 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="b33c8dc3-39f6-440e-86ab-297792bb50e2" containerName="registry-server" Jul 22 21:15:42.266518 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:42.266461 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="b33c8dc3-39f6-440e-86ab-297792bb50e2" containerName="registry-server" Jul 22 21:15:42.269913 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:42.269896 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rbwgs" Jul 22 21:15:42.272084 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:42.272063 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 22 21:15:42.272236 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:42.272219 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 22 21:15:42.273012 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:42.272997 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-269gt\"" Jul 22 21:15:42.276509 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:42.276488 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rbwgs"] Jul 22 21:15:42.384389 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:42.384354 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdb7h\" (UniqueName: \"kubernetes.io/projected/1c2d0ac7-0c4e-4c4a-9acc-75be526b7a79-kube-api-access-rdb7h\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rbwgs\" (UID: \"1c2d0ac7-0c4e-4c4a-9acc-75be526b7a79\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rbwgs" Jul 22 21:15:42.384582 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:42.384444 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1c2d0ac7-0c4e-4c4a-9acc-75be526b7a79-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rbwgs\" (UID: \"1c2d0ac7-0c4e-4c4a-9acc-75be526b7a79\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rbwgs" Jul 22 21:15:42.384582 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:42.384490 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1c2d0ac7-0c4e-4c4a-9acc-75be526b7a79-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rbwgs\" (UID: \"1c2d0ac7-0c4e-4c4a-9acc-75be526b7a79\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rbwgs" Jul 22 21:15:42.484893 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:42.484853 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdb7h\" (UniqueName: \"kubernetes.io/projected/1c2d0ac7-0c4e-4c4a-9acc-75be526b7a79-kube-api-access-rdb7h\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rbwgs\" (UID: \"1c2d0ac7-0c4e-4c4a-9acc-75be526b7a79\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rbwgs" Jul 22 21:15:42.485080 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:42.484927 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1c2d0ac7-0c4e-4c4a-9acc-75be526b7a79-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rbwgs\" (UID: \"1c2d0ac7-0c4e-4c4a-9acc-75be526b7a79\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rbwgs" Jul 22 21:15:42.485080 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:42.484969 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1c2d0ac7-0c4e-4c4a-9acc-75be526b7a79-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rbwgs\" (UID: \"1c2d0ac7-0c4e-4c4a-9acc-75be526b7a79\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rbwgs" Jul 22 21:15:42.485361 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:42.485342 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1c2d0ac7-0c4e-4c4a-9acc-75be526b7a79-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rbwgs\" (UID: \"1c2d0ac7-0c4e-4c4a-9acc-75be526b7a79\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rbwgs" Jul 22 21:15:42.485436 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:42.485375 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1c2d0ac7-0c4e-4c4a-9acc-75be526b7a79-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rbwgs\" (UID: \"1c2d0ac7-0c4e-4c4a-9acc-75be526b7a79\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rbwgs" Jul 22 21:15:42.493496 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:42.493469 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdb7h\" (UniqueName: \"kubernetes.io/projected/1c2d0ac7-0c4e-4c4a-9acc-75be526b7a79-kube-api-access-rdb7h\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rbwgs\" (UID: \"1c2d0ac7-0c4e-4c4a-9acc-75be526b7a79\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rbwgs" Jul 22 21:15:42.580844 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:42.580818 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rbwgs" Jul 22 21:15:42.716613 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:42.716580 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rbwgs"] Jul 22 21:15:42.720552 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:15:42.720508 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c2d0ac7_0c4e_4c4a_9acc_75be526b7a79.slice/crio-e5ee566af9eab44c14ef1c31a48484b9754d738a9c4d2a84c5911b421b064475 WatchSource:0}: Error finding container e5ee566af9eab44c14ef1c31a48484b9754d738a9c4d2a84c5911b421b064475: Status 404 returned error can't find the container with id e5ee566af9eab44c14ef1c31a48484b9754d738a9c4d2a84c5911b421b064475 Jul 22 21:15:42.745595 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:42.745567 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rbwgs" event={"ID":"1c2d0ac7-0c4e-4c4a-9acc-75be526b7a79","Type":"ContainerStarted","Data":"e5ee566af9eab44c14ef1c31a48484b9754d738a9c4d2a84c5911b421b064475"} Jul 22 21:15:42.864317 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:42.864235 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19r4kn"] Jul 22 21:15:42.867746 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:42.867724 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19r4kn" Jul 22 21:15:42.875810 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:42.875787 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19r4kn"] Jul 22 21:15:42.989240 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:42.989197 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1642c073-6026-4a1c-b51a-467b60e75087-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19r4kn\" (UID: \"1642c073-6026-4a1c-b51a-467b60e75087\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19r4kn" Jul 22 21:15:42.989240 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:42.989238 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cmlt\" (UniqueName: \"kubernetes.io/projected/1642c073-6026-4a1c-b51a-467b60e75087-kube-api-access-9cmlt\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19r4kn\" (UID: \"1642c073-6026-4a1c-b51a-467b60e75087\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19r4kn" Jul 22 21:15:42.989447 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:42.989335 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1642c073-6026-4a1c-b51a-467b60e75087-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19r4kn\" (UID: \"1642c073-6026-4a1c-b51a-467b60e75087\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19r4kn" Jul 22 21:15:43.090201 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:43.090141 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1642c073-6026-4a1c-b51a-467b60e75087-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19r4kn\" (UID: \"1642c073-6026-4a1c-b51a-467b60e75087\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19r4kn" Jul 22 21:15:43.090396 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:43.090208 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1642c073-6026-4a1c-b51a-467b60e75087-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19r4kn\" (UID: \"1642c073-6026-4a1c-b51a-467b60e75087\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19r4kn" Jul 22 21:15:43.090396 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:43.090238 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cmlt\" (UniqueName: \"kubernetes.io/projected/1642c073-6026-4a1c-b51a-467b60e75087-kube-api-access-9cmlt\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19r4kn\" (UID: \"1642c073-6026-4a1c-b51a-467b60e75087\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19r4kn" Jul 22 21:15:43.090544 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:43.090519 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1642c073-6026-4a1c-b51a-467b60e75087-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19r4kn\" (UID: \"1642c073-6026-4a1c-b51a-467b60e75087\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19r4kn" Jul 22 21:15:43.090601 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:43.090545 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1642c073-6026-4a1c-b51a-467b60e75087-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19r4kn\" (UID: \"1642c073-6026-4a1c-b51a-467b60e75087\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19r4kn" Jul 22 21:15:43.098730 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:43.098706 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cmlt\" (UniqueName: \"kubernetes.io/projected/1642c073-6026-4a1c-b51a-467b60e75087-kube-api-access-9cmlt\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19r4kn\" (UID: \"1642c073-6026-4a1c-b51a-467b60e75087\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19r4kn" Jul 22 21:15:43.190478 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:43.190401 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19r4kn" Jul 22 21:15:43.281688 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:43.281652 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594h74l"] Jul 22 21:15:43.287441 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:43.287415 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594h74l" Jul 22 21:15:43.292611 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:43.292584 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594h74l"] Jul 22 21:15:43.332225 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:43.332203 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19r4kn"] Jul 22 21:15:43.334432 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:15:43.334403 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1642c073_6026_4a1c_b51a_467b60e75087.slice/crio-d2be9a81f94fcb454821794fbcb6f41eabf93d613db4ffba7ce4fdce9a5f8b11 WatchSource:0}: Error finding container d2be9a81f94fcb454821794fbcb6f41eabf93d613db4ffba7ce4fdce9a5f8b11: Status 404 returned error can't find the container with id d2be9a81f94fcb454821794fbcb6f41eabf93d613db4ffba7ce4fdce9a5f8b11 Jul 22 21:15:43.392591 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:43.392565 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/abe06a85-7716-4827-9767-9f9deef29783-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594h74l\" (UID: \"abe06a85-7716-4827-9767-9f9deef29783\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594h74l" Jul 22 21:15:43.392702 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:43.392621 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2t5xj\" (UniqueName: \"kubernetes.io/projected/abe06a85-7716-4827-9767-9f9deef29783-kube-api-access-2t5xj\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594h74l\" (UID: \"abe06a85-7716-4827-9767-9f9deef29783\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594h74l" Jul 22 21:15:43.392702 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:43.392684 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/abe06a85-7716-4827-9767-9f9deef29783-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594h74l\" (UID: \"abe06a85-7716-4827-9767-9f9deef29783\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594h74l" Jul 22 21:15:43.493310 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:43.493215 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/abe06a85-7716-4827-9767-9f9deef29783-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594h74l\" (UID: \"abe06a85-7716-4827-9767-9f9deef29783\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594h74l" Jul 22 21:15:43.493310 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:43.493276 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/abe06a85-7716-4827-9767-9f9deef29783-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594h74l\" (UID: \"abe06a85-7716-4827-9767-9f9deef29783\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594h74l" Jul 22 21:15:43.493535 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:43.493326 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t5xj\" (UniqueName: \"kubernetes.io/projected/abe06a85-7716-4827-9767-9f9deef29783-kube-api-access-2t5xj\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594h74l\" (UID: \"abe06a85-7716-4827-9767-9f9deef29783\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594h74l" Jul 22 21:15:43.493694 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:43.493671 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/abe06a85-7716-4827-9767-9f9deef29783-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594h74l\" (UID: \"abe06a85-7716-4827-9767-9f9deef29783\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594h74l" Jul 22 21:15:43.493754 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:43.493737 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/abe06a85-7716-4827-9767-9f9deef29783-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594h74l\" (UID: \"abe06a85-7716-4827-9767-9f9deef29783\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594h74l" Jul 22 21:15:43.502923 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:43.502903 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2t5xj\" (UniqueName: \"kubernetes.io/projected/abe06a85-7716-4827-9767-9f9deef29783-kube-api-access-2t5xj\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594h74l\" (UID: \"abe06a85-7716-4827-9767-9f9deef29783\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594h74l" Jul 22 21:15:43.600977 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:43.600929 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594h74l" Jul 22 21:15:43.737086 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:43.737055 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594h74l"] Jul 22 21:15:43.738721 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:15:43.738694 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabe06a85_7716_4827_9767_9f9deef29783.slice/crio-40233f8e5114f17edf6d644b04f96d016ad8026442a97a6c6db1a01e4bfeda9f WatchSource:0}: Error finding container 40233f8e5114f17edf6d644b04f96d016ad8026442a97a6c6db1a01e4bfeda9f: Status 404 returned error can't find the container with id 40233f8e5114f17edf6d644b04f96d016ad8026442a97a6c6db1a01e4bfeda9f Jul 22 21:15:43.752293 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:43.752259 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594h74l" event={"ID":"abe06a85-7716-4827-9767-9f9deef29783","Type":"ContainerStarted","Data":"40233f8e5114f17edf6d644b04f96d016ad8026442a97a6c6db1a01e4bfeda9f"} Jul 22 21:15:43.753795 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:43.753772 2572 generic.go:356] "Generic (PLEG): container finished" podID="1642c073-6026-4a1c-b51a-467b60e75087" containerID="0f33d13a4a7fc93bdb10126f5b593d0a2bd22712cb12cda47bb0e689033d8739" exitCode=0 Jul 22 21:15:43.753896 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:43.753856 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19r4kn" event={"ID":"1642c073-6026-4a1c-b51a-467b60e75087","Type":"ContainerDied","Data":"0f33d13a4a7fc93bdb10126f5b593d0a2bd22712cb12cda47bb0e689033d8739"} Jul 22 21:15:43.753961 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:43.753904 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19r4kn" event={"ID":"1642c073-6026-4a1c-b51a-467b60e75087","Type":"ContainerStarted","Data":"d2be9a81f94fcb454821794fbcb6f41eabf93d613db4ffba7ce4fdce9a5f8b11"} Jul 22 21:15:43.755557 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:43.755534 2572 generic.go:356] "Generic (PLEG): container finished" podID="1c2d0ac7-0c4e-4c4a-9acc-75be526b7a79" containerID="9422668ec753c9e09bb29d186528559b3874b18e0ee7f198ab5bd7d9f6a75a81" exitCode=0 Jul 22 21:15:43.755663 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:43.755579 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rbwgs" event={"ID":"1c2d0ac7-0c4e-4c4a-9acc-75be526b7a79","Type":"ContainerDied","Data":"9422668ec753c9e09bb29d186528559b3874b18e0ee7f198ab5bd7d9f6a75a81"} Jul 22 21:15:44.761497 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:44.761467 2572 generic.go:356] "Generic (PLEG): container finished" podID="abe06a85-7716-4827-9767-9f9deef29783" containerID="1045d0ffcd18f9a5c9b736f1143b95874ce351f4c26293654c9b95411e83d723" exitCode=0 Jul 22 21:15:44.761873 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:44.761555 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594h74l" event={"ID":"abe06a85-7716-4827-9767-9f9deef29783","Type":"ContainerDied","Data":"1045d0ffcd18f9a5c9b736f1143b95874ce351f4c26293654c9b95411e83d723"} Jul 22 21:15:44.763587 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:44.763563 2572 generic.go:356] "Generic (PLEG): container finished" podID="1642c073-6026-4a1c-b51a-467b60e75087" containerID="2e7f6270955550c5020b1c53f06608bdd930a5156bee8abfbe49ba9c48dcdccf" exitCode=0 Jul 22 21:15:44.763675 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:44.763627 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19r4kn" event={"ID":"1642c073-6026-4a1c-b51a-467b60e75087","Type":"ContainerDied","Data":"2e7f6270955550c5020b1c53f06608bdd930a5156bee8abfbe49ba9c48dcdccf"} Jul 22 21:15:45.771133 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:45.771103 2572 generic.go:356] "Generic (PLEG): container finished" podID="1c2d0ac7-0c4e-4c4a-9acc-75be526b7a79" containerID="47e9ab851c8aad8cc674901e66dbe94b5855c95154f2e16db68d239ebb16fea6" exitCode=0 Jul 22 21:15:45.771631 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:45.771200 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rbwgs" event={"ID":"1c2d0ac7-0c4e-4c4a-9acc-75be526b7a79","Type":"ContainerDied","Data":"47e9ab851c8aad8cc674901e66dbe94b5855c95154f2e16db68d239ebb16fea6"} Jul 22 21:15:45.773466 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:45.773445 2572 generic.go:356] "Generic (PLEG): container finished" podID="1642c073-6026-4a1c-b51a-467b60e75087" containerID="9693d3e0be6fa4daa16c392e9f6427c519ed55ec0fd262bbdac29a6684f47518" exitCode=0 Jul 22 21:15:45.773562 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:45.773530 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19r4kn" event={"ID":"1642c073-6026-4a1c-b51a-467b60e75087","Type":"ContainerDied","Data":"9693d3e0be6fa4daa16c392e9f6427c519ed55ec0fd262bbdac29a6684f47518"} Jul 22 21:15:46.780781 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:46.780743 2572 generic.go:356] "Generic (PLEG): container finished" podID="1c2d0ac7-0c4e-4c4a-9acc-75be526b7a79" containerID="e485154ff12c9625a58db0320f1a1ef1fd9ab92a05b692de988ec1c476c5d686" exitCode=0 Jul 22 21:15:46.781208 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:46.780824 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rbwgs" event={"ID":"1c2d0ac7-0c4e-4c4a-9acc-75be526b7a79","Type":"ContainerDied","Data":"e485154ff12c9625a58db0320f1a1ef1fd9ab92a05b692de988ec1c476c5d686"} Jul 22 21:15:46.782614 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:46.782593 2572 generic.go:356] "Generic (PLEG): container finished" podID="abe06a85-7716-4827-9767-9f9deef29783" containerID="045a6f256c0eb6302d5aeb620701f6012e12b5d59922a8aba696570920fc8bb4" exitCode=0 Jul 22 21:15:46.782732 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:46.782684 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594h74l" event={"ID":"abe06a85-7716-4827-9767-9f9deef29783","Type":"ContainerDied","Data":"045a6f256c0eb6302d5aeb620701f6012e12b5d59922a8aba696570920fc8bb4"} Jul 22 21:15:46.925302 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:46.925278 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19r4kn" Jul 22 21:15:47.021531 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:47.021495 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1642c073-6026-4a1c-b51a-467b60e75087-util\") pod \"1642c073-6026-4a1c-b51a-467b60e75087\" (UID: \"1642c073-6026-4a1c-b51a-467b60e75087\") " Jul 22 21:15:47.021695 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:47.021542 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9cmlt\" (UniqueName: \"kubernetes.io/projected/1642c073-6026-4a1c-b51a-467b60e75087-kube-api-access-9cmlt\") pod \"1642c073-6026-4a1c-b51a-467b60e75087\" (UID: \"1642c073-6026-4a1c-b51a-467b60e75087\") " Jul 22 21:15:47.021695 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:47.021586 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1642c073-6026-4a1c-b51a-467b60e75087-bundle\") pod \"1642c073-6026-4a1c-b51a-467b60e75087\" (UID: \"1642c073-6026-4a1c-b51a-467b60e75087\") " Jul 22 21:15:47.022198 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:47.022145 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1642c073-6026-4a1c-b51a-467b60e75087-bundle" (OuterVolumeSpecName: "bundle") pod "1642c073-6026-4a1c-b51a-467b60e75087" (UID: "1642c073-6026-4a1c-b51a-467b60e75087"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 21:15:47.023851 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:47.023815 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1642c073-6026-4a1c-b51a-467b60e75087-kube-api-access-9cmlt" (OuterVolumeSpecName: "kube-api-access-9cmlt") pod "1642c073-6026-4a1c-b51a-467b60e75087" (UID: "1642c073-6026-4a1c-b51a-467b60e75087"). InnerVolumeSpecName "kube-api-access-9cmlt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 21:15:47.026478 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:47.026460 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1642c073-6026-4a1c-b51a-467b60e75087-util" (OuterVolumeSpecName: "util") pod "1642c073-6026-4a1c-b51a-467b60e75087" (UID: "1642c073-6026-4a1c-b51a-467b60e75087"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 21:15:47.122459 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:47.122423 2572 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1642c073-6026-4a1c-b51a-467b60e75087-util\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:15:47.122459 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:47.122455 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cmlt\" (UniqueName: \"kubernetes.io/projected/1642c073-6026-4a1c-b51a-467b60e75087-kube-api-access-9cmlt\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:15:47.122459 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:47.122465 2572 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1642c073-6026-4a1c-b51a-467b60e75087-bundle\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:15:47.723780 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:47.723702 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-66b6fd96f9-qtfdc"] Jul 22 21:15:47.724065 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:47.724053 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1642c073-6026-4a1c-b51a-467b60e75087" containerName="extract" Jul 22 21:15:47.724108 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:47.724067 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="1642c073-6026-4a1c-b51a-467b60e75087" containerName="extract" Jul 22 21:15:47.724108 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:47.724077 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1642c073-6026-4a1c-b51a-467b60e75087" containerName="util" Jul 22 21:15:47.724108 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:47.724082 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="1642c073-6026-4a1c-b51a-467b60e75087" containerName="util" Jul 22 21:15:47.724108 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:47.724097 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1642c073-6026-4a1c-b51a-467b60e75087" containerName="pull" Jul 22 21:15:47.724108 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:47.724102 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="1642c073-6026-4a1c-b51a-467b60e75087" containerName="pull" Jul 22 21:15:47.724297 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:47.724154 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="1642c073-6026-4a1c-b51a-467b60e75087" containerName="extract" Jul 22 21:15:47.727328 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:47.727307 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66b6fd96f9-qtfdc" Jul 22 21:15:47.736519 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:47.736493 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66b6fd96f9-qtfdc"] Jul 22 21:15:47.790025 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:47.789987 2572 generic.go:356] "Generic (PLEG): container finished" podID="abe06a85-7716-4827-9767-9f9deef29783" containerID="4734a016acf2eed5aa0e8b0edf9b950afac23e40bf99271cd61f503cb4af2908" exitCode=0 Jul 22 21:15:47.790462 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:47.790068 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594h74l" event={"ID":"abe06a85-7716-4827-9767-9f9deef29783","Type":"ContainerDied","Data":"4734a016acf2eed5aa0e8b0edf9b950afac23e40bf99271cd61f503cb4af2908"} Jul 22 21:15:47.791866 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:47.791839 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19r4kn" event={"ID":"1642c073-6026-4a1c-b51a-467b60e75087","Type":"ContainerDied","Data":"d2be9a81f94fcb454821794fbcb6f41eabf93d613db4ffba7ce4fdce9a5f8b11"} Jul 22 21:15:47.791973 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:47.791889 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2be9a81f94fcb454821794fbcb6f41eabf93d613db4ffba7ce4fdce9a5f8b11" Jul 22 21:15:47.791973 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:47.791854 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19r4kn" Jul 22 21:15:47.828361 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:47.828323 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2b365fa5-795b-45ba-af51-038572c0e32e-service-ca\") pod \"console-66b6fd96f9-qtfdc\" (UID: \"2b365fa5-795b-45ba-af51-038572c0e32e\") " pod="openshift-console/console-66b6fd96f9-qtfdc" Jul 22 21:15:47.828361 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:47.828367 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b365fa5-795b-45ba-af51-038572c0e32e-trusted-ca-bundle\") pod \"console-66b6fd96f9-qtfdc\" (UID: \"2b365fa5-795b-45ba-af51-038572c0e32e\") " pod="openshift-console/console-66b6fd96f9-qtfdc" Jul 22 21:15:47.828596 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:47.828463 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2b365fa5-795b-45ba-af51-038572c0e32e-console-serving-cert\") pod \"console-66b6fd96f9-qtfdc\" (UID: \"2b365fa5-795b-45ba-af51-038572c0e32e\") " pod="openshift-console/console-66b6fd96f9-qtfdc" Jul 22 21:15:47.828596 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:47.828493 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2b365fa5-795b-45ba-af51-038572c0e32e-console-oauth-config\") pod \"console-66b6fd96f9-qtfdc\" (UID: \"2b365fa5-795b-45ba-af51-038572c0e32e\") " pod="openshift-console/console-66b6fd96f9-qtfdc" Jul 22 21:15:47.828596 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:47.828516 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdc2l\" (UniqueName: \"kubernetes.io/projected/2b365fa5-795b-45ba-af51-038572c0e32e-kube-api-access-pdc2l\") pod \"console-66b6fd96f9-qtfdc\" (UID: \"2b365fa5-795b-45ba-af51-038572c0e32e\") " pod="openshift-console/console-66b6fd96f9-qtfdc" Jul 22 21:15:47.828738 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:47.828625 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2b365fa5-795b-45ba-af51-038572c0e32e-console-config\") pod \"console-66b6fd96f9-qtfdc\" (UID: \"2b365fa5-795b-45ba-af51-038572c0e32e\") " pod="openshift-console/console-66b6fd96f9-qtfdc" Jul 22 21:15:47.828738 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:47.828659 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2b365fa5-795b-45ba-af51-038572c0e32e-oauth-serving-cert\") pod \"console-66b6fd96f9-qtfdc\" (UID: \"2b365fa5-795b-45ba-af51-038572c0e32e\") " pod="openshift-console/console-66b6fd96f9-qtfdc" Jul 22 21:15:47.923471 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:47.923447 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rbwgs" Jul 22 21:15:47.929798 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:47.929772 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b365fa5-795b-45ba-af51-038572c0e32e-trusted-ca-bundle\") pod \"console-66b6fd96f9-qtfdc\" (UID: \"2b365fa5-795b-45ba-af51-038572c0e32e\") " pod="openshift-console/console-66b6fd96f9-qtfdc" Jul 22 21:15:47.929880 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:47.929859 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2b365fa5-795b-45ba-af51-038572c0e32e-console-serving-cert\") pod \"console-66b6fd96f9-qtfdc\" (UID: \"2b365fa5-795b-45ba-af51-038572c0e32e\") " pod="openshift-console/console-66b6fd96f9-qtfdc" Jul 22 21:15:47.929924 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:47.929891 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2b365fa5-795b-45ba-af51-038572c0e32e-console-oauth-config\") pod \"console-66b6fd96f9-qtfdc\" (UID: \"2b365fa5-795b-45ba-af51-038572c0e32e\") " pod="openshift-console/console-66b6fd96f9-qtfdc" Jul 22 21:15:47.929924 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:47.929916 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdc2l\" (UniqueName: \"kubernetes.io/projected/2b365fa5-795b-45ba-af51-038572c0e32e-kube-api-access-pdc2l\") pod \"console-66b6fd96f9-qtfdc\" (UID: \"2b365fa5-795b-45ba-af51-038572c0e32e\") " pod="openshift-console/console-66b6fd96f9-qtfdc" Jul 22 21:15:47.930026 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:47.929991 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2b365fa5-795b-45ba-af51-038572c0e32e-console-config\") pod \"console-66b6fd96f9-qtfdc\" (UID: \"2b365fa5-795b-45ba-af51-038572c0e32e\") " pod="openshift-console/console-66b6fd96f9-qtfdc" Jul 22 21:15:47.930026 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:47.930016 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2b365fa5-795b-45ba-af51-038572c0e32e-oauth-serving-cert\") pod \"console-66b6fd96f9-qtfdc\" (UID: \"2b365fa5-795b-45ba-af51-038572c0e32e\") " pod="openshift-console/console-66b6fd96f9-qtfdc" Jul 22 21:15:47.930111 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:47.930068 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2b365fa5-795b-45ba-af51-038572c0e32e-service-ca\") pod \"console-66b6fd96f9-qtfdc\" (UID: \"2b365fa5-795b-45ba-af51-038572c0e32e\") " pod="openshift-console/console-66b6fd96f9-qtfdc" Jul 22 21:15:47.930762 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:47.930708 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2b365fa5-795b-45ba-af51-038572c0e32e-console-config\") pod \"console-66b6fd96f9-qtfdc\" (UID: \"2b365fa5-795b-45ba-af51-038572c0e32e\") " pod="openshift-console/console-66b6fd96f9-qtfdc" Jul 22 21:15:47.930762 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:47.930722 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b365fa5-795b-45ba-af51-038572c0e32e-trusted-ca-bundle\") pod \"console-66b6fd96f9-qtfdc\" (UID: \"2b365fa5-795b-45ba-af51-038572c0e32e\") " pod="openshift-console/console-66b6fd96f9-qtfdc" Jul 22 21:15:47.930762 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:47.930748 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2b365fa5-795b-45ba-af51-038572c0e32e-service-ca\") pod \"console-66b6fd96f9-qtfdc\" (UID: \"2b365fa5-795b-45ba-af51-038572c0e32e\") " pod="openshift-console/console-66b6fd96f9-qtfdc" Jul 22 21:15:47.930963 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:47.930799 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2b365fa5-795b-45ba-af51-038572c0e32e-oauth-serving-cert\") pod \"console-66b6fd96f9-qtfdc\" (UID: \"2b365fa5-795b-45ba-af51-038572c0e32e\") " pod="openshift-console/console-66b6fd96f9-qtfdc" Jul 22 21:15:47.932552 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:47.932536 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2b365fa5-795b-45ba-af51-038572c0e32e-console-serving-cert\") pod \"console-66b6fd96f9-qtfdc\" (UID: \"2b365fa5-795b-45ba-af51-038572c0e32e\") " pod="openshift-console/console-66b6fd96f9-qtfdc" Jul 22 21:15:47.932624 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:47.932588 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2b365fa5-795b-45ba-af51-038572c0e32e-console-oauth-config\") pod \"console-66b6fd96f9-qtfdc\" (UID: \"2b365fa5-795b-45ba-af51-038572c0e32e\") " pod="openshift-console/console-66b6fd96f9-qtfdc" Jul 22 21:15:47.938082 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:47.938058 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdc2l\" (UniqueName: \"kubernetes.io/projected/2b365fa5-795b-45ba-af51-038572c0e32e-kube-api-access-pdc2l\") pod \"console-66b6fd96f9-qtfdc\" (UID: \"2b365fa5-795b-45ba-af51-038572c0e32e\") " pod="openshift-console/console-66b6fd96f9-qtfdc" Jul 22 21:15:48.030744 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:48.030658 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1c2d0ac7-0c4e-4c4a-9acc-75be526b7a79-bundle\") pod \"1c2d0ac7-0c4e-4c4a-9acc-75be526b7a79\" (UID: \"1c2d0ac7-0c4e-4c4a-9acc-75be526b7a79\") " Jul 22 21:15:48.030744 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:48.030712 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rdb7h\" (UniqueName: \"kubernetes.io/projected/1c2d0ac7-0c4e-4c4a-9acc-75be526b7a79-kube-api-access-rdb7h\") pod \"1c2d0ac7-0c4e-4c4a-9acc-75be526b7a79\" (UID: \"1c2d0ac7-0c4e-4c4a-9acc-75be526b7a79\") " Jul 22 21:15:48.030976 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:48.030754 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1c2d0ac7-0c4e-4c4a-9acc-75be526b7a79-util\") pod \"1c2d0ac7-0c4e-4c4a-9acc-75be526b7a79\" (UID: \"1c2d0ac7-0c4e-4c4a-9acc-75be526b7a79\") " Jul 22 21:15:48.031258 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:48.031226 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c2d0ac7-0c4e-4c4a-9acc-75be526b7a79-bundle" (OuterVolumeSpecName: "bundle") pod "1c2d0ac7-0c4e-4c4a-9acc-75be526b7a79" (UID: "1c2d0ac7-0c4e-4c4a-9acc-75be526b7a79"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 21:15:48.032962 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:48.032935 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c2d0ac7-0c4e-4c4a-9acc-75be526b7a79-kube-api-access-rdb7h" (OuterVolumeSpecName: "kube-api-access-rdb7h") pod "1c2d0ac7-0c4e-4c4a-9acc-75be526b7a79" (UID: "1c2d0ac7-0c4e-4c4a-9acc-75be526b7a79"). InnerVolumeSpecName "kube-api-access-rdb7h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 21:15:48.035673 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:48.035643 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c2d0ac7-0c4e-4c4a-9acc-75be526b7a79-util" (OuterVolumeSpecName: "util") pod "1c2d0ac7-0c4e-4c4a-9acc-75be526b7a79" (UID: "1c2d0ac7-0c4e-4c4a-9acc-75be526b7a79"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 21:15:48.037123 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:48.037106 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66b6fd96f9-qtfdc" Jul 22 21:15:48.131391 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:48.131362 2572 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1c2d0ac7-0c4e-4c4a-9acc-75be526b7a79-bundle\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:15:48.131391 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:48.131389 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdb7h\" (UniqueName: \"kubernetes.io/projected/1c2d0ac7-0c4e-4c4a-9acc-75be526b7a79-kube-api-access-rdb7h\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:15:48.131573 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:48.131401 2572 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1c2d0ac7-0c4e-4c4a-9acc-75be526b7a79-util\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:15:48.170228 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:48.170203 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66b6fd96f9-qtfdc"] Jul 22 21:15:48.171786 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:15:48.171759 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b365fa5_795b_45ba_af51_038572c0e32e.slice/crio-26459e96ec1da9a7828055cb20d46c03447f884b852af410949809236c94bc26 WatchSource:0}: Error finding container 26459e96ec1da9a7828055cb20d46c03447f884b852af410949809236c94bc26: Status 404 returned error can't find the container with id 26459e96ec1da9a7828055cb20d46c03447f884b852af410949809236c94bc26 Jul 22 21:15:48.799198 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:48.799141 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rbwgs" event={"ID":"1c2d0ac7-0c4e-4c4a-9acc-75be526b7a79","Type":"ContainerDied","Data":"e5ee566af9eab44c14ef1c31a48484b9754d738a9c4d2a84c5911b421b064475"} Jul 22 21:15:48.799643 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:48.799215 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e5ee566af9eab44c14ef1c31a48484b9754d738a9c4d2a84c5911b421b064475" Jul 22 21:15:48.799643 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:48.799152 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rbwgs" Jul 22 21:15:48.800763 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:48.800737 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66b6fd96f9-qtfdc" event={"ID":"2b365fa5-795b-45ba-af51-038572c0e32e","Type":"ContainerStarted","Data":"a6aeed3d190a16e01070c1f825a48517efc04b26d882a487c096b2b3cdba4960"} Jul 22 21:15:48.800841 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:48.800784 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66b6fd96f9-qtfdc" event={"ID":"2b365fa5-795b-45ba-af51-038572c0e32e","Type":"ContainerStarted","Data":"26459e96ec1da9a7828055cb20d46c03447f884b852af410949809236c94bc26"} Jul 22 21:15:48.818359 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:48.818302 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-66b6fd96f9-qtfdc" podStartSLOduration=1.818278073 podStartE2EDuration="1.818278073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 21:15:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 21:15:48.817487774 +0000 UTC m=+603.931880133" watchObservedRunningTime="2026-07-22 21:15:48.818278073 +0000 UTC m=+603.932670420" Jul 22 21:15:48.935361 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:48.935335 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594h74l" Jul 22 21:15:49.040909 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:49.040875 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2t5xj\" (UniqueName: \"kubernetes.io/projected/abe06a85-7716-4827-9767-9f9deef29783-kube-api-access-2t5xj\") pod \"abe06a85-7716-4827-9767-9f9deef29783\" (UID: \"abe06a85-7716-4827-9767-9f9deef29783\") " Jul 22 21:15:49.040909 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:49.040911 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/abe06a85-7716-4827-9767-9f9deef29783-util\") pod \"abe06a85-7716-4827-9767-9f9deef29783\" (UID: \"abe06a85-7716-4827-9767-9f9deef29783\") " Jul 22 21:15:49.041140 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:49.040968 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/abe06a85-7716-4827-9767-9f9deef29783-bundle\") pod \"abe06a85-7716-4827-9767-9f9deef29783\" (UID: \"abe06a85-7716-4827-9767-9f9deef29783\") " Jul 22 21:15:49.041502 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:49.041470 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abe06a85-7716-4827-9767-9f9deef29783-bundle" (OuterVolumeSpecName: "bundle") pod "abe06a85-7716-4827-9767-9f9deef29783" (UID: "abe06a85-7716-4827-9767-9f9deef29783"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 21:15:49.043133 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:49.043110 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abe06a85-7716-4827-9767-9f9deef29783-kube-api-access-2t5xj" (OuterVolumeSpecName: "kube-api-access-2t5xj") pod "abe06a85-7716-4827-9767-9f9deef29783" (UID: "abe06a85-7716-4827-9767-9f9deef29783"). InnerVolumeSpecName "kube-api-access-2t5xj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 21:15:49.087270 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:49.087230 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abe06a85-7716-4827-9767-9f9deef29783-util" (OuterVolumeSpecName: "util") pod "abe06a85-7716-4827-9767-9f9deef29783" (UID: "abe06a85-7716-4827-9767-9f9deef29783"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 21:15:49.141767 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:49.141729 2572 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/abe06a85-7716-4827-9767-9f9deef29783-bundle\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:15:49.141767 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:49.141762 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2t5xj\" (UniqueName: \"kubernetes.io/projected/abe06a85-7716-4827-9767-9f9deef29783-kube-api-access-2t5xj\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:15:49.141767 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:49.141776 2572 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/abe06a85-7716-4827-9767-9f9deef29783-util\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:15:49.808415 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:49.808392 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594h74l" Jul 22 21:15:49.808799 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:49.808409 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594h74l" event={"ID":"abe06a85-7716-4827-9767-9f9deef29783","Type":"ContainerDied","Data":"40233f8e5114f17edf6d644b04f96d016ad8026442a97a6c6db1a01e4bfeda9f"} Jul 22 21:15:49.808799 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:49.808492 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40233f8e5114f17edf6d644b04f96d016ad8026442a97a6c6db1a01e4bfeda9f" Jul 22 21:15:58.037489 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:58.037451 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-66b6fd96f9-qtfdc" Jul 22 21:15:58.037896 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:58.037517 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-66b6fd96f9-qtfdc" Jul 22 21:15:58.042435 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:58.042412 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-66b6fd96f9-qtfdc" Jul 22 21:15:58.859147 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:58.859123 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-66b6fd96f9-qtfdc" Jul 22 21:15:58.937309 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:15:58.937277 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-84856587d6-k8ntg"] Jul 22 21:16:02.547224 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:02.547184 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9vt55"] Jul 22 21:16:02.547714 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:02.547523 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c2d0ac7-0c4e-4c4a-9acc-75be526b7a79" containerName="extract" Jul 22 21:16:02.547714 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:02.547534 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c2d0ac7-0c4e-4c4a-9acc-75be526b7a79" containerName="extract" Jul 22 21:16:02.547714 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:02.547547 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abe06a85-7716-4827-9767-9f9deef29783" containerName="pull" Jul 22 21:16:02.547714 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:02.547553 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="abe06a85-7716-4827-9767-9f9deef29783" containerName="pull" Jul 22 21:16:02.547714 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:02.547562 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c2d0ac7-0c4e-4c4a-9acc-75be526b7a79" containerName="util" Jul 22 21:16:02.547714 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:02.547568 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c2d0ac7-0c4e-4c4a-9acc-75be526b7a79" containerName="util" Jul 22 21:16:02.547714 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:02.547576 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abe06a85-7716-4827-9767-9f9deef29783" containerName="util" Jul 22 21:16:02.547714 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:02.547582 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="abe06a85-7716-4827-9767-9f9deef29783" containerName="util" Jul 22 21:16:02.547714 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:02.547588 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abe06a85-7716-4827-9767-9f9deef29783" containerName="extract" Jul 22 21:16:02.547714 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:02.547593 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="abe06a85-7716-4827-9767-9f9deef29783" containerName="extract" Jul 22 21:16:02.547714 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:02.547606 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c2d0ac7-0c4e-4c4a-9acc-75be526b7a79" containerName="pull" Jul 22 21:16:02.547714 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:02.547611 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c2d0ac7-0c4e-4c4a-9acc-75be526b7a79" containerName="pull" Jul 22 21:16:02.547714 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:02.547664 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c2d0ac7-0c4e-4c4a-9acc-75be526b7a79" containerName="extract" Jul 22 21:16:02.547714 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:02.547670 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="abe06a85-7716-4827-9767-9f9deef29783" containerName="extract" Jul 22 21:16:02.552627 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:02.552611 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9vt55" Jul 22 21:16:02.555142 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:02.555116 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-5jb7f\"" Jul 22 21:16:02.555329 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:02.555280 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 22 21:16:02.556064 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:02.556043 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 22 21:16:02.558928 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:02.558904 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9vt55"] Jul 22 21:16:02.666217 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:02.666144 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t29b5\" (UniqueName: \"kubernetes.io/projected/17fe000e-0482-4930-97be-95d404a9aee5-kube-api-access-t29b5\") pod \"limitador-operator-controller-manager-859d7b78bd-9vt55\" (UID: \"17fe000e-0482-4930-97be-95d404a9aee5\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9vt55" Jul 22 21:16:02.767252 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:02.767226 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t29b5\" (UniqueName: \"kubernetes.io/projected/17fe000e-0482-4930-97be-95d404a9aee5-kube-api-access-t29b5\") pod \"limitador-operator-controller-manager-859d7b78bd-9vt55\" (UID: \"17fe000e-0482-4930-97be-95d404a9aee5\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9vt55" Jul 22 21:16:02.775663 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:02.775630 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t29b5\" (UniqueName: \"kubernetes.io/projected/17fe000e-0482-4930-97be-95d404a9aee5-kube-api-access-t29b5\") pod \"limitador-operator-controller-manager-859d7b78bd-9vt55\" (UID: \"17fe000e-0482-4930-97be-95d404a9aee5\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9vt55" Jul 22 21:16:02.863516 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:02.863473 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9vt55" Jul 22 21:16:03.009249 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:03.009225 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9vt55"] Jul 22 21:16:03.011874 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:16:03.011846 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17fe000e_0482_4930_97be_95d404a9aee5.slice/crio-d4a3373c2b77db82812e0469e3230fea3151e301952ff513867760a84f80593c WatchSource:0}: Error finding container d4a3373c2b77db82812e0469e3230fea3151e301952ff513867760a84f80593c: Status 404 returned error can't find the container with id d4a3373c2b77db82812e0469e3230fea3151e301952ff513867760a84f80593c Jul 22 21:16:03.883854 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:03.883819 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9vt55" event={"ID":"17fe000e-0482-4930-97be-95d404a9aee5","Type":"ContainerStarted","Data":"d4a3373c2b77db82812e0469e3230fea3151e301952ff513867760a84f80593c"} Jul 22 21:16:05.896220 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:05.896103 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9vt55" event={"ID":"17fe000e-0482-4930-97be-95d404a9aee5","Type":"ContainerStarted","Data":"ac1c97fe4a78f01785dd0f274aaa459ef228ecee1040be76db3dfdf9a8a2c603"} Jul 22 21:16:05.896607 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:05.896219 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9vt55" Jul 22 21:16:05.925900 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:05.925849 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9vt55" podStartSLOduration=1.344797456 podStartE2EDuration="3.925834956s" totalImagesPullingTime="2.5810375s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 21:16:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 21:16:05.922655831 +0000 UTC m=+621.037048174" watchObservedRunningTime="2026-07-22 21:16:05.925834956 +0000 UTC m=+621.040227299" Jul 22 21:16:16.903983 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:16.903953 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9vt55" Jul 22 21:16:23.957680 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:23.957640 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-84856587d6-k8ntg" podUID="2631535b-cad2-48f9-b739-f96f22a0707a" containerName="console" containerID="cri-o://e14020f083b8ae5d56ca9a871cf253cf33e6986cf9b657036811d832666bc913" gracePeriod=15 Jul 22 21:16:24.213384 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:24.213322 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84856587d6-k8ntg" Jul 22 21:16:24.252761 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:24.252730 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gblgv\" (UniqueName: \"kubernetes.io/projected/2631535b-cad2-48f9-b739-f96f22a0707a-kube-api-access-gblgv\") pod \"2631535b-cad2-48f9-b739-f96f22a0707a\" (UID: \"2631535b-cad2-48f9-b739-f96f22a0707a\") " Jul 22 21:16:24.252937 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:24.252770 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2631535b-cad2-48f9-b739-f96f22a0707a-service-ca\") pod \"2631535b-cad2-48f9-b739-f96f22a0707a\" (UID: \"2631535b-cad2-48f9-b739-f96f22a0707a\") " Jul 22 21:16:24.252937 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:24.252818 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2631535b-cad2-48f9-b739-f96f22a0707a-trusted-ca-bundle\") pod \"2631535b-cad2-48f9-b739-f96f22a0707a\" (UID: \"2631535b-cad2-48f9-b739-f96f22a0707a\") " Jul 22 21:16:24.252937 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:24.252865 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2631535b-cad2-48f9-b739-f96f22a0707a-console-config\") pod \"2631535b-cad2-48f9-b739-f96f22a0707a\" (UID: \"2631535b-cad2-48f9-b739-f96f22a0707a\") " Jul 22 21:16:24.252937 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:24.252891 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2631535b-cad2-48f9-b739-f96f22a0707a-console-serving-cert\") pod \"2631535b-cad2-48f9-b739-f96f22a0707a\" (UID: \"2631535b-cad2-48f9-b739-f96f22a0707a\") " Jul 22 21:16:24.253185 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:24.252946 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2631535b-cad2-48f9-b739-f96f22a0707a-console-oauth-config\") pod \"2631535b-cad2-48f9-b739-f96f22a0707a\" (UID: \"2631535b-cad2-48f9-b739-f96f22a0707a\") " Jul 22 21:16:24.253185 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:24.253007 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2631535b-cad2-48f9-b739-f96f22a0707a-oauth-serving-cert\") pod \"2631535b-cad2-48f9-b739-f96f22a0707a\" (UID: \"2631535b-cad2-48f9-b739-f96f22a0707a\") " Jul 22 21:16:24.253360 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:24.253328 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2631535b-cad2-48f9-b739-f96f22a0707a-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "2631535b-cad2-48f9-b739-f96f22a0707a" (UID: "2631535b-cad2-48f9-b739-f96f22a0707a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 21:16:24.253576 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:24.253475 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2631535b-cad2-48f9-b739-f96f22a0707a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "2631535b-cad2-48f9-b739-f96f22a0707a" (UID: "2631535b-cad2-48f9-b739-f96f22a0707a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 21:16:24.253576 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:24.253581 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2631535b-cad2-48f9-b739-f96f22a0707a-console-config" (OuterVolumeSpecName: "console-config") pod "2631535b-cad2-48f9-b739-f96f22a0707a" (UID: "2631535b-cad2-48f9-b739-f96f22a0707a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 21:16:24.253764 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:24.253608 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2631535b-cad2-48f9-b739-f96f22a0707a-service-ca" (OuterVolumeSpecName: "service-ca") pod "2631535b-cad2-48f9-b739-f96f22a0707a" (UID: "2631535b-cad2-48f9-b739-f96f22a0707a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 21:16:24.255188 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:24.255129 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2631535b-cad2-48f9-b739-f96f22a0707a-kube-api-access-gblgv" (OuterVolumeSpecName: "kube-api-access-gblgv") pod "2631535b-cad2-48f9-b739-f96f22a0707a" (UID: "2631535b-cad2-48f9-b739-f96f22a0707a"). InnerVolumeSpecName "kube-api-access-gblgv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 21:16:24.255355 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:24.255333 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2631535b-cad2-48f9-b739-f96f22a0707a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "2631535b-cad2-48f9-b739-f96f22a0707a" (UID: "2631535b-cad2-48f9-b739-f96f22a0707a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 21:16:24.255417 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:24.255361 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2631535b-cad2-48f9-b739-f96f22a0707a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "2631535b-cad2-48f9-b739-f96f22a0707a" (UID: "2631535b-cad2-48f9-b739-f96f22a0707a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 21:16:24.354635 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:24.354601 2572 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2631535b-cad2-48f9-b739-f96f22a0707a-trusted-ca-bundle\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:16:24.354635 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:24.354629 2572 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2631535b-cad2-48f9-b739-f96f22a0707a-console-config\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:16:24.354635 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:24.354639 2572 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2631535b-cad2-48f9-b739-f96f22a0707a-console-serving-cert\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:16:24.354868 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:24.354648 2572 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2631535b-cad2-48f9-b739-f96f22a0707a-console-oauth-config\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:16:24.354868 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:24.354657 2572 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2631535b-cad2-48f9-b739-f96f22a0707a-oauth-serving-cert\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:16:24.354868 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:24.354665 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gblgv\" (UniqueName: \"kubernetes.io/projected/2631535b-cad2-48f9-b739-f96f22a0707a-kube-api-access-gblgv\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:16:24.354868 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:24.354674 2572 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2631535b-cad2-48f9-b739-f96f22a0707a-service-ca\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:16:24.993060 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:24.993022 2572 generic.go:356] "Generic (PLEG): container finished" podID="2631535b-cad2-48f9-b739-f96f22a0707a" containerID="e14020f083b8ae5d56ca9a871cf253cf33e6986cf9b657036811d832666bc913" exitCode=0 Jul 22 21:16:24.993483 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:24.993079 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84856587d6-k8ntg" Jul 22 21:16:24.993483 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:24.993123 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84856587d6-k8ntg" event={"ID":"2631535b-cad2-48f9-b739-f96f22a0707a","Type":"ContainerDied","Data":"e14020f083b8ae5d56ca9a871cf253cf33e6986cf9b657036811d832666bc913"} Jul 22 21:16:24.993483 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:24.993199 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84856587d6-k8ntg" event={"ID":"2631535b-cad2-48f9-b739-f96f22a0707a","Type":"ContainerDied","Data":"7dc480be8487172b42f6801f1cda9478a88cbb5973313b58876243eda0e85414"} Jul 22 21:16:24.993483 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:24.993225 2572 scope.go:117] "RemoveContainer" containerID="e14020f083b8ae5d56ca9a871cf253cf33e6986cf9b657036811d832666bc913" Jul 22 21:16:25.004790 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:25.004763 2572 scope.go:117] "RemoveContainer" containerID="e14020f083b8ae5d56ca9a871cf253cf33e6986cf9b657036811d832666bc913" Jul 22 21:16:25.006284 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:16:25.006250 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e14020f083b8ae5d56ca9a871cf253cf33e6986cf9b657036811d832666bc913\": container with ID starting with e14020f083b8ae5d56ca9a871cf253cf33e6986cf9b657036811d832666bc913 not found: ID does not exist" containerID="e14020f083b8ae5d56ca9a871cf253cf33e6986cf9b657036811d832666bc913" Jul 22 21:16:25.006395 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:25.006295 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e14020f083b8ae5d56ca9a871cf253cf33e6986cf9b657036811d832666bc913"} err="failed to get container status \"e14020f083b8ae5d56ca9a871cf253cf33e6986cf9b657036811d832666bc913\": rpc error: code = NotFound desc = could not find container \"e14020f083b8ae5d56ca9a871cf253cf33e6986cf9b657036811d832666bc913\": container with ID starting with e14020f083b8ae5d56ca9a871cf253cf33e6986cf9b657036811d832666bc913 not found: ID does not exist" Jul 22 21:16:25.020760 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:25.020730 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-84856587d6-k8ntg"] Jul 22 21:16:25.028953 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:25.028926 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-84856587d6-k8ntg"] Jul 22 21:16:25.382870 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:25.382834 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2631535b-cad2-48f9-b739-f96f22a0707a" path="/var/lib/kubelet/pods/2631535b-cad2-48f9-b739-f96f22a0707a/volumes" Jul 22 21:16:29.422320 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:29.422288 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-5tr8p"] Jul 22 21:16:29.422691 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:29.422643 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2631535b-cad2-48f9-b739-f96f22a0707a" containerName="console" Jul 22 21:16:29.422691 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:29.422654 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="2631535b-cad2-48f9-b739-f96f22a0707a" containerName="console" Jul 22 21:16:29.422764 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:29.422724 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="2631535b-cad2-48f9-b739-f96f22a0707a" containerName="console" Jul 22 21:16:29.426986 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:29.426966 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-5tr8p" Jul 22 21:16:29.430371 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:29.430347 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-zhkzn\"" Jul 22 21:16:29.438789 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:29.438766 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-5tr8p"] Jul 22 21:16:29.505538 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:29.505491 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4v5k\" (UniqueName: \"kubernetes.io/projected/29091eb7-d24c-4d61-8166-833c85788926-kube-api-access-k4v5k\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-5tr8p\" (UID: \"29091eb7-d24c-4d61-8166-833c85788926\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-5tr8p" Jul 22 21:16:29.505718 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:29.505553 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/29091eb7-d24c-4d61-8166-833c85788926-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-5tr8p\" (UID: \"29091eb7-d24c-4d61-8166-833c85788926\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-5tr8p" Jul 22 21:16:29.606005 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:29.605965 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4v5k\" (UniqueName: \"kubernetes.io/projected/29091eb7-d24c-4d61-8166-833c85788926-kube-api-access-k4v5k\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-5tr8p\" (UID: \"29091eb7-d24c-4d61-8166-833c85788926\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-5tr8p" Jul 22 21:16:29.606181 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:29.606036 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/29091eb7-d24c-4d61-8166-833c85788926-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-5tr8p\" (UID: \"29091eb7-d24c-4d61-8166-833c85788926\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-5tr8p" Jul 22 21:16:29.606457 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:29.606438 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/29091eb7-d24c-4d61-8166-833c85788926-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-5tr8p\" (UID: \"29091eb7-d24c-4d61-8166-833c85788926\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-5tr8p" Jul 22 21:16:29.619854 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:29.619820 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4v5k\" (UniqueName: \"kubernetes.io/projected/29091eb7-d24c-4d61-8166-833c85788926-kube-api-access-k4v5k\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-5tr8p\" (UID: \"29091eb7-d24c-4d61-8166-833c85788926\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-5tr8p" Jul 22 21:16:29.739433 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:29.739346 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-5tr8p" Jul 22 21:16:29.874724 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:29.874692 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-5tr8p"] Jul 22 21:16:30.022362 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:30.022274 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-5tr8p" event={"ID":"29091eb7-d24c-4d61-8166-833c85788926","Type":"ContainerStarted","Data":"cfefbe38becbfcb9453d74fedccc9da93d3a6fa385f7a03aa4688d7cc322de04"} Jul 22 21:16:30.199472 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:30.199439 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-5tr8p"] Jul 22 21:16:30.203949 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:30.203913 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-5tr8p"] Jul 22 21:16:30.223187 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:30.223141 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lgx7g"] Jul 22 21:16:30.227759 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:30.227738 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lgx7g" Jul 22 21:16:30.235180 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:30.235137 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9vt55"] Jul 22 21:16:30.235540 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:30.235387 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9vt55" podUID="17fe000e-0482-4930-97be-95d404a9aee5" containerName="manager" containerID="cri-o://ac1c97fe4a78f01785dd0f274aaa459ef228ecee1040be76db3dfdf9a8a2c603" gracePeriod=2 Jul 22 21:16:30.240506 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:30.240487 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lgx7g"] Jul 22 21:16:30.247435 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:30.247408 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9vt55"] Jul 22 21:16:30.254973 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:16:30.254939 2572 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-9vt55\" is forbidden: User \"system:node:ip-10-0-135-37.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-37.ec2.internal' and this object" podUID="17fe000e-0482-4930-97be-95d404a9aee5" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9vt55" Jul 22 21:16:30.266851 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:30.266826 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8f4vc"] Jul 22 21:16:30.267412 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:30.267393 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17fe000e-0482-4930-97be-95d404a9aee5" containerName="manager" Jul 22 21:16:30.267459 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:30.267419 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="17fe000e-0482-4930-97be-95d404a9aee5" containerName="manager" Jul 22 21:16:30.267559 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:30.267547 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="17fe000e-0482-4930-97be-95d404a9aee5" containerName="manager" Jul 22 21:16:30.271032 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:30.271013 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8f4vc" Jul 22 21:16:30.285126 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:30.284939 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8f4vc"] Jul 22 21:16:30.302538 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:16:30.302491 2572 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-9vt55\" is forbidden: User \"system:node:ip-10-0-135-37.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-37.ec2.internal' and this object" podUID="17fe000e-0482-4930-97be-95d404a9aee5" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9vt55" Jul 22 21:16:30.313341 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:30.313321 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mfw6\" (UniqueName: \"kubernetes.io/projected/69662004-5105-4783-909b-3393ca843304-kube-api-access-8mfw6\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-lgx7g\" (UID: \"69662004-5105-4783-909b-3393ca843304\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lgx7g" Jul 22 21:16:30.313457 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:30.313356 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/69662004-5105-4783-909b-3393ca843304-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-lgx7g\" (UID: \"69662004-5105-4783-909b-3393ca843304\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lgx7g" Jul 22 21:16:30.313516 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:30.313491 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29t59\" (UniqueName: \"kubernetes.io/projected/6f940cfb-15b1-4301-ba08-2d7d86e25c66-kube-api-access-29t59\") pod \"limitador-operator-controller-manager-859d7b78bd-8f4vc\" (UID: \"6f940cfb-15b1-4301-ba08-2d7d86e25c66\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8f4vc" Jul 22 21:16:30.415883 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:30.414911 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29t59\" (UniqueName: \"kubernetes.io/projected/6f940cfb-15b1-4301-ba08-2d7d86e25c66-kube-api-access-29t59\") pod \"limitador-operator-controller-manager-859d7b78bd-8f4vc\" (UID: \"6f940cfb-15b1-4301-ba08-2d7d86e25c66\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8f4vc" Jul 22 21:16:30.415883 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:30.414994 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mfw6\" (UniqueName: \"kubernetes.io/projected/69662004-5105-4783-909b-3393ca843304-kube-api-access-8mfw6\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-lgx7g\" (UID: \"69662004-5105-4783-909b-3393ca843304\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lgx7g" Jul 22 21:16:30.415883 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:30.415040 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/69662004-5105-4783-909b-3393ca843304-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-lgx7g\" (UID: \"69662004-5105-4783-909b-3393ca843304\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lgx7g" Jul 22 21:16:30.415883 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:30.415505 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/69662004-5105-4783-909b-3393ca843304-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-lgx7g\" (UID: \"69662004-5105-4783-909b-3393ca843304\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lgx7g" Jul 22 21:16:30.424983 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:30.424954 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29t59\" (UniqueName: \"kubernetes.io/projected/6f940cfb-15b1-4301-ba08-2d7d86e25c66-kube-api-access-29t59\") pod \"limitador-operator-controller-manager-859d7b78bd-8f4vc\" (UID: \"6f940cfb-15b1-4301-ba08-2d7d86e25c66\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8f4vc" Jul 22 21:16:30.425349 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:30.425038 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mfw6\" (UniqueName: \"kubernetes.io/projected/69662004-5105-4783-909b-3393ca843304-kube-api-access-8mfw6\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-lgx7g\" (UID: \"69662004-5105-4783-909b-3393ca843304\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lgx7g" Jul 22 21:16:30.467926 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:30.467904 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9vt55" Jul 22 21:16:30.470239 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:16:30.470208 2572 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-9vt55\" is forbidden: User \"system:node:ip-10-0-135-37.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-37.ec2.internal' and this object" podUID="17fe000e-0482-4930-97be-95d404a9aee5" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9vt55" Jul 22 21:16:30.515444 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:30.515410 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t29b5\" (UniqueName: \"kubernetes.io/projected/17fe000e-0482-4930-97be-95d404a9aee5-kube-api-access-t29b5\") pod \"17fe000e-0482-4930-97be-95d404a9aee5\" (UID: \"17fe000e-0482-4930-97be-95d404a9aee5\") " Jul 22 21:16:30.518063 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:30.518038 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17fe000e-0482-4930-97be-95d404a9aee5-kube-api-access-t29b5" (OuterVolumeSpecName: "kube-api-access-t29b5") pod "17fe000e-0482-4930-97be-95d404a9aee5" (UID: "17fe000e-0482-4930-97be-95d404a9aee5"). InnerVolumeSpecName "kube-api-access-t29b5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 21:16:30.543743 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:30.543678 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lgx7g" Jul 22 21:16:30.606604 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:30.606442 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8f4vc" Jul 22 21:16:30.617371 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:30.617338 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t29b5\" (UniqueName: \"kubernetes.io/projected/17fe000e-0482-4930-97be-95d404a9aee5-kube-api-access-t29b5\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:16:30.721836 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:30.721753 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lgx7g"] Jul 22 21:16:30.723399 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:16:30.723359 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69662004_5105_4783_909b_3393ca843304.slice/crio-b55ab1f377e785e5e010a43129f204251e691d7cf52d0d521a61a58ed118f82d WatchSource:0}: Error finding container b55ab1f377e785e5e010a43129f204251e691d7cf52d0d521a61a58ed118f82d: Status 404 returned error can't find the container with id b55ab1f377e785e5e010a43129f204251e691d7cf52d0d521a61a58ed118f82d Jul 22 21:16:30.781559 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:30.781529 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8f4vc"] Jul 22 21:16:30.783907 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:16:30.783873 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f940cfb_15b1_4301_ba08_2d7d86e25c66.slice/crio-397a2c08b56cc524933e406c0a5dc6ab1dfba19577473bb71b5a9ccda3258638 WatchSource:0}: Error finding container 397a2c08b56cc524933e406c0a5dc6ab1dfba19577473bb71b5a9ccda3258638: Status 404 returned error can't find the container with id 397a2c08b56cc524933e406c0a5dc6ab1dfba19577473bb71b5a9ccda3258638 Jul 22 21:16:31.031031 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:31.030968 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8f4vc" event={"ID":"6f940cfb-15b1-4301-ba08-2d7d86e25c66","Type":"ContainerStarted","Data":"2bb33eabdf81e29cdb8a69f36926a6952d02727b3658154e924fae62f1387910"} Jul 22 21:16:31.031253 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:31.031043 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8f4vc" Jul 22 21:16:31.031253 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:31.031069 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8f4vc" event={"ID":"6f940cfb-15b1-4301-ba08-2d7d86e25c66","Type":"ContainerStarted","Data":"397a2c08b56cc524933e406c0a5dc6ab1dfba19577473bb71b5a9ccda3258638"} Jul 22 21:16:31.032402 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:31.032379 2572 generic.go:356] "Generic (PLEG): container finished" podID="17fe000e-0482-4930-97be-95d404a9aee5" containerID="ac1c97fe4a78f01785dd0f274aaa459ef228ecee1040be76db3dfdf9a8a2c603" exitCode=0 Jul 22 21:16:31.032528 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:31.032428 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9vt55" Jul 22 21:16:31.032528 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:31.032487 2572 scope.go:117] "RemoveContainer" containerID="ac1c97fe4a78f01785dd0f274aaa459ef228ecee1040be76db3dfdf9a8a2c603" Jul 22 21:16:31.033359 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:16:31.033309 2572 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-9vt55\" is forbidden: User \"system:node:ip-10-0-135-37.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-37.ec2.internal' and this object" podUID="17fe000e-0482-4930-97be-95d404a9aee5" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9vt55" Jul 22 21:16:31.034269 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:31.034241 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lgx7g" event={"ID":"69662004-5105-4783-909b-3393ca843304","Type":"ContainerStarted","Data":"b55ab1f377e785e5e010a43129f204251e691d7cf52d0d521a61a58ed118f82d"} Jul 22 21:16:31.043132 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:31.043110 2572 scope.go:117] "RemoveContainer" containerID="ac1c97fe4a78f01785dd0f274aaa459ef228ecee1040be76db3dfdf9a8a2c603" Jul 22 21:16:31.043477 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:16:31.043459 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac1c97fe4a78f01785dd0f274aaa459ef228ecee1040be76db3dfdf9a8a2c603\": container with ID starting with ac1c97fe4a78f01785dd0f274aaa459ef228ecee1040be76db3dfdf9a8a2c603 not found: ID does not exist" containerID="ac1c97fe4a78f01785dd0f274aaa459ef228ecee1040be76db3dfdf9a8a2c603" Jul 22 21:16:31.043550 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:31.043483 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac1c97fe4a78f01785dd0f274aaa459ef228ecee1040be76db3dfdf9a8a2c603"} err="failed to get container status \"ac1c97fe4a78f01785dd0f274aaa459ef228ecee1040be76db3dfdf9a8a2c603\": rpc error: code = NotFound desc = could not find container \"ac1c97fe4a78f01785dd0f274aaa459ef228ecee1040be76db3dfdf9a8a2c603\": container with ID starting with ac1c97fe4a78f01785dd0f274aaa459ef228ecee1040be76db3dfdf9a8a2c603 not found: ID does not exist" Jul 22 21:16:31.052885 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:31.052816 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8f4vc" podStartSLOduration=1.05280121 podStartE2EDuration="1.05280121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 21:16:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 21:16:31.052018335 +0000 UTC m=+646.166410710" watchObservedRunningTime="2026-07-22 21:16:31.05280121 +0000 UTC m=+646.167193555" Jul 22 21:16:31.054014 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:16:31.053978 2572 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-9vt55\" is forbidden: User \"system:node:ip-10-0-135-37.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-37.ec2.internal' and this object" podUID="17fe000e-0482-4930-97be-95d404a9aee5" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9vt55" Jul 22 21:16:31.386558 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:31.386525 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17fe000e-0482-4930-97be-95d404a9aee5" path="/var/lib/kubelet/pods/17fe000e-0482-4930-97be-95d404a9aee5/volumes" Jul 22 21:16:35.061547 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:35.061503 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-5tr8p" podUID="29091eb7-d24c-4d61-8166-833c85788926" containerName="manager" containerID="cri-o://d5f47dd7a630cf971e50f678599d05919712135284b463414cc84a2ceb6e84b7" gracePeriod=2 Jul 22 21:16:35.063179 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:35.063140 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lgx7g" event={"ID":"69662004-5105-4783-909b-3393ca843304","Type":"ContainerStarted","Data":"e26037f492957ab1e215dd221389ab80f5e83f63c2c4504fc87fb41d67f8db8c"} Jul 22 21:16:35.063280 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:35.063234 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lgx7g" Jul 22 21:16:35.082374 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:35.082332 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lgx7g" podStartSLOduration=1.6945159680000002 podStartE2EDuration="5.08231646s" totalImagesPullingTime="3.387800492s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 21:16:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 21:16:35.07943071 +0000 UTC m=+650.193823074" watchObservedRunningTime="2026-07-22 21:16:35.08231646 +0000 UTC m=+650.196708804" Jul 22 21:16:35.312440 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:35.312382 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-5tr8p" Jul 22 21:16:35.314783 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:16:35.314758 2572 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-5tr8p\" is forbidden: User \"system:node:ip-10-0-135-37.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-37.ec2.internal' and this object" podUID="29091eb7-d24c-4d61-8166-833c85788926" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-5tr8p" Jul 22 21:16:35.361323 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:35.361288 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/29091eb7-d24c-4d61-8166-833c85788926-extensions-socket-volume\") pod \"29091eb7-d24c-4d61-8166-833c85788926\" (UID: \"29091eb7-d24c-4d61-8166-833c85788926\") " Jul 22 21:16:35.361488 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:35.361339 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k4v5k\" (UniqueName: \"kubernetes.io/projected/29091eb7-d24c-4d61-8166-833c85788926-kube-api-access-k4v5k\") pod \"29091eb7-d24c-4d61-8166-833c85788926\" (UID: \"29091eb7-d24c-4d61-8166-833c85788926\") " Jul 22 21:16:35.361570 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:35.361546 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29091eb7-d24c-4d61-8166-833c85788926-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "29091eb7-d24c-4d61-8166-833c85788926" (UID: "29091eb7-d24c-4d61-8166-833c85788926"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 21:16:35.363454 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:35.363433 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29091eb7-d24c-4d61-8166-833c85788926-kube-api-access-k4v5k" (OuterVolumeSpecName: "kube-api-access-k4v5k") pod "29091eb7-d24c-4d61-8166-833c85788926" (UID: "29091eb7-d24c-4d61-8166-833c85788926"). InnerVolumeSpecName "kube-api-access-k4v5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 21:16:35.381066 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:16:35.381033 2572 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-5tr8p\" is forbidden: User \"system:node:ip-10-0-135-37.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-37.ec2.internal' and this object" podUID="29091eb7-d24c-4d61-8166-833c85788926" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-5tr8p" Jul 22 21:16:35.382471 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:35.382450 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29091eb7-d24c-4d61-8166-833c85788926" path="/var/lib/kubelet/pods/29091eb7-d24c-4d61-8166-833c85788926/volumes" Jul 22 21:16:35.462420 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:35.462391 2572 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/29091eb7-d24c-4d61-8166-833c85788926-extensions-socket-volume\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:16:35.462420 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:35.462414 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4v5k\" (UniqueName: \"kubernetes.io/projected/29091eb7-d24c-4d61-8166-833c85788926-kube-api-access-k4v5k\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:16:36.069815 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:36.069784 2572 generic.go:356] "Generic (PLEG): container finished" podID="29091eb7-d24c-4d61-8166-833c85788926" containerID="d5f47dd7a630cf971e50f678599d05919712135284b463414cc84a2ceb6e84b7" exitCode=2 Jul 22 21:16:36.069815 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:36.069821 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-5tr8p" Jul 22 21:16:36.070335 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:36.069884 2572 scope.go:117] "RemoveContainer" containerID="d5f47dd7a630cf971e50f678599d05919712135284b463414cc84a2ceb6e84b7" Jul 22 21:16:36.074278 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:16:36.074250 2572 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-5tr8p\" is forbidden: User \"system:node:ip-10-0-135-37.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-37.ec2.internal' and this object" podUID="29091eb7-d24c-4d61-8166-833c85788926" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-5tr8p" Jul 22 21:16:36.078914 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:36.078894 2572 scope.go:117] "RemoveContainer" containerID="d5f47dd7a630cf971e50f678599d05919712135284b463414cc84a2ceb6e84b7" Jul 22 21:16:36.079141 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:16:36.079124 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5f47dd7a630cf971e50f678599d05919712135284b463414cc84a2ceb6e84b7\": container with ID starting with d5f47dd7a630cf971e50f678599d05919712135284b463414cc84a2ceb6e84b7 not found: ID does not exist" containerID="d5f47dd7a630cf971e50f678599d05919712135284b463414cc84a2ceb6e84b7" Jul 22 21:16:36.079227 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:36.079148 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5f47dd7a630cf971e50f678599d05919712135284b463414cc84a2ceb6e84b7"} err="failed to get container status \"d5f47dd7a630cf971e50f678599d05919712135284b463414cc84a2ceb6e84b7\": rpc error: code = NotFound desc = could not find container \"d5f47dd7a630cf971e50f678599d05919712135284b463414cc84a2ceb6e84b7\": container with ID starting with d5f47dd7a630cf971e50f678599d05919712135284b463414cc84a2ceb6e84b7 not found: ID does not exist" Jul 22 21:16:42.045252 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:42.045219 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8f4vc" Jul 22 21:16:45.303022 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:45.302983 2572 scope.go:117] "RemoveContainer" containerID="c2f1f79d1b1a75b9a79a2ddc67123f2842b82864430d1922e6f0fc4e1f073d66" Jul 22 21:16:45.311082 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:45.311065 2572 scope.go:117] "RemoveContainer" containerID="b556b0d347e996528144d7ed431cdb2e701dd47724beebf5cf024a07fa8c8e24" Jul 22 21:16:45.318708 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:45.318691 2572 scope.go:117] "RemoveContainer" containerID="3c93264ff08292125c50471a1281e3b6bac25a76f59f8d5448e08a450c0bd35e" Jul 22 21:16:45.325666 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:45.325649 2572 scope.go:117] "RemoveContainer" containerID="fba6a4055ada9a4b5007c1d701c405fa3c6071b5b38e1c256bada8111c8ac589" Jul 22 21:16:45.332883 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:45.332864 2572 scope.go:117] "RemoveContainer" containerID="9f3f0cb1e6d6358a070b364c742be14f7f760ef8b19053e03542fb44dc9957ce" Jul 22 21:16:45.339764 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:45.339749 2572 scope.go:117] "RemoveContainer" containerID="7c4aecb3c740e48eb8f153301ced1b582b22114e1aa88e5894a1e76ce5fff3c2" Jul 22 21:16:45.346942 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:45.346925 2572 scope.go:117] "RemoveContainer" containerID="01c19e0f452dc6a90d7e42c79fe38fe6b585a9ea8c839b000fa2939b28763e73" Jul 22 21:16:46.072336 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:46.072304 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lgx7g" Jul 22 21:16:58.637742 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:58.637711 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lgx7g"] Jul 22 21:16:58.638258 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:58.637969 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lgx7g" podUID="69662004-5105-4783-909b-3393ca843304" containerName="manager" containerID="cri-o://e26037f492957ab1e215dd221389ab80f5e83f63c2c4504fc87fb41d67f8db8c" gracePeriod=10 Jul 22 21:16:58.890080 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:58.890022 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lgx7g" Jul 22 21:16:59.075369 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:59.075331 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8mfw6\" (UniqueName: \"kubernetes.io/projected/69662004-5105-4783-909b-3393ca843304-kube-api-access-8mfw6\") pod \"69662004-5105-4783-909b-3393ca843304\" (UID: \"69662004-5105-4783-909b-3393ca843304\") " Jul 22 21:16:59.075562 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:59.075457 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/69662004-5105-4783-909b-3393ca843304-extensions-socket-volume\") pod \"69662004-5105-4783-909b-3393ca843304\" (UID: \"69662004-5105-4783-909b-3393ca843304\") " Jul 22 21:16:59.075827 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:59.075803 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69662004-5105-4783-909b-3393ca843304-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "69662004-5105-4783-909b-3393ca843304" (UID: "69662004-5105-4783-909b-3393ca843304"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 21:16:59.077615 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:59.077594 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69662004-5105-4783-909b-3393ca843304-kube-api-access-8mfw6" (OuterVolumeSpecName: "kube-api-access-8mfw6") pod "69662004-5105-4783-909b-3393ca843304" (UID: "69662004-5105-4783-909b-3393ca843304"). InnerVolumeSpecName "kube-api-access-8mfw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 21:16:59.176344 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:59.176261 2572 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/69662004-5105-4783-909b-3393ca843304-extensions-socket-volume\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:16:59.176344 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:59.176290 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mfw6\" (UniqueName: \"kubernetes.io/projected/69662004-5105-4783-909b-3393ca843304-kube-api-access-8mfw6\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:16:59.195074 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:59.195040 2572 generic.go:356] "Generic (PLEG): container finished" podID="69662004-5105-4783-909b-3393ca843304" containerID="e26037f492957ab1e215dd221389ab80f5e83f63c2c4504fc87fb41d67f8db8c" exitCode=0 Jul 22 21:16:59.195258 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:59.195098 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lgx7g" Jul 22 21:16:59.195258 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:59.195116 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lgx7g" event={"ID":"69662004-5105-4783-909b-3393ca843304","Type":"ContainerDied","Data":"e26037f492957ab1e215dd221389ab80f5e83f63c2c4504fc87fb41d67f8db8c"} Jul 22 21:16:59.195258 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:59.195184 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lgx7g" event={"ID":"69662004-5105-4783-909b-3393ca843304","Type":"ContainerDied","Data":"b55ab1f377e785e5e010a43129f204251e691d7cf52d0d521a61a58ed118f82d"} Jul 22 21:16:59.195258 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:59.195213 2572 scope.go:117] "RemoveContainer" containerID="e26037f492957ab1e215dd221389ab80f5e83f63c2c4504fc87fb41d67f8db8c" Jul 22 21:16:59.204449 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:59.204430 2572 scope.go:117] "RemoveContainer" containerID="e26037f492957ab1e215dd221389ab80f5e83f63c2c4504fc87fb41d67f8db8c" Jul 22 21:16:59.204692 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:16:59.204674 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e26037f492957ab1e215dd221389ab80f5e83f63c2c4504fc87fb41d67f8db8c\": container with ID starting with e26037f492957ab1e215dd221389ab80f5e83f63c2c4504fc87fb41d67f8db8c not found: ID does not exist" containerID="e26037f492957ab1e215dd221389ab80f5e83f63c2c4504fc87fb41d67f8db8c" Jul 22 21:16:59.204761 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:59.204703 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e26037f492957ab1e215dd221389ab80f5e83f63c2c4504fc87fb41d67f8db8c"} err="failed to get container status \"e26037f492957ab1e215dd221389ab80f5e83f63c2c4504fc87fb41d67f8db8c\": rpc error: code = NotFound desc = could not find container \"e26037f492957ab1e215dd221389ab80f5e83f63c2c4504fc87fb41d67f8db8c\": container with ID starting with e26037f492957ab1e215dd221389ab80f5e83f63c2c4504fc87fb41d67f8db8c not found: ID does not exist" Jul 22 21:16:59.220352 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:59.220329 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lgx7g"] Jul 22 21:16:59.224547 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:59.224525 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-lgx7g"] Jul 22 21:16:59.382946 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:16:59.382912 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69662004-5105-4783-909b-3393ca843304" path="/var/lib/kubelet/pods/69662004-5105-4783-909b-3393ca843304/volumes" Jul 22 21:17:02.418576 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:02.418538 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-chp6r"] Jul 22 21:17:02.419061 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:02.419041 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69662004-5105-4783-909b-3393ca843304" containerName="manager" Jul 22 21:17:02.419142 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:02.419066 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="69662004-5105-4783-909b-3393ca843304" containerName="manager" Jul 22 21:17:02.419142 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:02.419092 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29091eb7-d24c-4d61-8166-833c85788926" containerName="manager" Jul 22 21:17:02.419142 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:02.419101 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="29091eb7-d24c-4d61-8166-833c85788926" containerName="manager" Jul 22 21:17:02.419284 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:02.419189 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="29091eb7-d24c-4d61-8166-833c85788926" containerName="manager" Jul 22 21:17:02.419284 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:02.419201 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="69662004-5105-4783-909b-3393ca843304" containerName="manager" Jul 22 21:17:02.424037 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:02.424020 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-chp6r" Jul 22 21:17:02.426459 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:02.426439 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-hx5ct\"" Jul 22 21:17:02.434074 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:02.434051 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-chp6r"] Jul 22 21:17:02.603442 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:02.603402 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqw4g\" (UniqueName: \"kubernetes.io/projected/40faf561-504b-45df-9c9b-211772e1e444-kube-api-access-rqw4g\") pod \"maas-default-gateway-openshift-default-56654dcf7-chp6r\" (UID: \"40faf561-504b-45df-9c9b-211772e1e444\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-chp6r" Jul 22 21:17:02.603620 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:02.603453 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40faf561-504b-45df-9c9b-211772e1e444-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-chp6r\" (UID: \"40faf561-504b-45df-9c9b-211772e1e444\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-chp6r" Jul 22 21:17:02.603620 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:02.603479 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40faf561-504b-45df-9c9b-211772e1e444-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-chp6r\" (UID: \"40faf561-504b-45df-9c9b-211772e1e444\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-chp6r" Jul 22 21:17:02.603620 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:02.603504 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40faf561-504b-45df-9c9b-211772e1e444-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-chp6r\" (UID: \"40faf561-504b-45df-9c9b-211772e1e444\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-chp6r" Jul 22 21:17:02.603620 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:02.603533 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40faf561-504b-45df-9c9b-211772e1e444-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-chp6r\" (UID: \"40faf561-504b-45df-9c9b-211772e1e444\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-chp6r" Jul 22 21:17:02.603620 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:02.603551 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40faf561-504b-45df-9c9b-211772e1e444-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-chp6r\" (UID: \"40faf561-504b-45df-9c9b-211772e1e444\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-chp6r" Jul 22 21:17:02.603793 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:02.603614 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40faf561-504b-45df-9c9b-211772e1e444-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-chp6r\" (UID: \"40faf561-504b-45df-9c9b-211772e1e444\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-chp6r" Jul 22 21:17:02.603793 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:02.603673 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40faf561-504b-45df-9c9b-211772e1e444-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-chp6r\" (UID: \"40faf561-504b-45df-9c9b-211772e1e444\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-chp6r" Jul 22 21:17:02.603793 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:02.603728 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40faf561-504b-45df-9c9b-211772e1e444-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-chp6r\" (UID: \"40faf561-504b-45df-9c9b-211772e1e444\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-chp6r" Jul 22 21:17:02.705236 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:02.705122 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40faf561-504b-45df-9c9b-211772e1e444-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-chp6r\" (UID: \"40faf561-504b-45df-9c9b-211772e1e444\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-chp6r" Jul 22 21:17:02.705236 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:02.705193 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40faf561-504b-45df-9c9b-211772e1e444-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-chp6r\" (UID: \"40faf561-504b-45df-9c9b-211772e1e444\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-chp6r" Jul 22 21:17:02.705446 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:02.705245 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqw4g\" (UniqueName: \"kubernetes.io/projected/40faf561-504b-45df-9c9b-211772e1e444-kube-api-access-rqw4g\") pod \"maas-default-gateway-openshift-default-56654dcf7-chp6r\" (UID: \"40faf561-504b-45df-9c9b-211772e1e444\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-chp6r" Jul 22 21:17:02.705446 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:02.705271 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40faf561-504b-45df-9c9b-211772e1e444-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-chp6r\" (UID: \"40faf561-504b-45df-9c9b-211772e1e444\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-chp6r" Jul 22 21:17:02.705446 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:02.705294 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40faf561-504b-45df-9c9b-211772e1e444-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-chp6r\" (UID: \"40faf561-504b-45df-9c9b-211772e1e444\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-chp6r" Jul 22 21:17:02.705446 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:02.705309 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40faf561-504b-45df-9c9b-211772e1e444-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-chp6r\" (UID: \"40faf561-504b-45df-9c9b-211772e1e444\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-chp6r" Jul 22 21:17:02.705446 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:02.705331 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40faf561-504b-45df-9c9b-211772e1e444-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-chp6r\" (UID: \"40faf561-504b-45df-9c9b-211772e1e444\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-chp6r" Jul 22 21:17:02.705446 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:02.705355 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40faf561-504b-45df-9c9b-211772e1e444-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-chp6r\" (UID: \"40faf561-504b-45df-9c9b-211772e1e444\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-chp6r" Jul 22 21:17:02.705446 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:02.705381 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40faf561-504b-45df-9c9b-211772e1e444-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-chp6r\" (UID: \"40faf561-504b-45df-9c9b-211772e1e444\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-chp6r" Jul 22 21:17:02.705799 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:02.705648 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40faf561-504b-45df-9c9b-211772e1e444-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-chp6r\" (UID: \"40faf561-504b-45df-9c9b-211772e1e444\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-chp6r" Jul 22 21:17:02.705799 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:02.705681 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40faf561-504b-45df-9c9b-211772e1e444-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-chp6r\" (UID: \"40faf561-504b-45df-9c9b-211772e1e444\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-chp6r" Jul 22 21:17:02.705799 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:02.705793 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40faf561-504b-45df-9c9b-211772e1e444-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-chp6r\" (UID: \"40faf561-504b-45df-9c9b-211772e1e444\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-chp6r" Jul 22 21:17:02.705963 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:02.705850 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40faf561-504b-45df-9c9b-211772e1e444-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-chp6r\" (UID: \"40faf561-504b-45df-9c9b-211772e1e444\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-chp6r" Jul 22 21:17:02.706197 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:02.706147 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40faf561-504b-45df-9c9b-211772e1e444-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-chp6r\" (UID: \"40faf561-504b-45df-9c9b-211772e1e444\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-chp6r" Jul 22 21:17:02.707541 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:02.707515 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40faf561-504b-45df-9c9b-211772e1e444-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-chp6r\" (UID: \"40faf561-504b-45df-9c9b-211772e1e444\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-chp6r" Jul 22 21:17:02.707831 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:02.707812 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40faf561-504b-45df-9c9b-211772e1e444-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-chp6r\" (UID: \"40faf561-504b-45df-9c9b-211772e1e444\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-chp6r" Jul 22 21:17:02.713032 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:02.713010 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40faf561-504b-45df-9c9b-211772e1e444-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-chp6r\" (UID: \"40faf561-504b-45df-9c9b-211772e1e444\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-chp6r" Jul 22 21:17:02.713321 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:02.713300 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqw4g\" (UniqueName: \"kubernetes.io/projected/40faf561-504b-45df-9c9b-211772e1e444-kube-api-access-rqw4g\") pod \"maas-default-gateway-openshift-default-56654dcf7-chp6r\" (UID: \"40faf561-504b-45df-9c9b-211772e1e444\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-chp6r" Jul 22 21:17:02.735152 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:02.735127 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-chp6r" Jul 22 21:17:02.872485 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:02.872460 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-chp6r"] Jul 22 21:17:02.875098 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:17:02.875068 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40faf561_504b_45df_9c9b_211772e1e444.slice/crio-55dc040d510b503881907b4f3047a85b77efc64e2dbd88067bcf19a2f9adf703 WatchSource:0}: Error finding container 55dc040d510b503881907b4f3047a85b77efc64e2dbd88067bcf19a2f9adf703: Status 404 returned error can't find the container with id 55dc040d510b503881907b4f3047a85b77efc64e2dbd88067bcf19a2f9adf703 Jul 22 21:17:02.877251 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:02.877210 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 22 21:17:02.877353 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:02.877290 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 22 21:17:02.877353 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:02.877323 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 22 21:17:03.220766 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:03.220724 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-chp6r" event={"ID":"40faf561-504b-45df-9c9b-211772e1e444","Type":"ContainerStarted","Data":"e84350d914eec06546a4bea56cf053cf9b0806beae06f1828c9b41e5694c94c0"} Jul 22 21:17:03.220978 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:03.220795 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-chp6r" event={"ID":"40faf561-504b-45df-9c9b-211772e1e444","Type":"ContainerStarted","Data":"55dc040d510b503881907b4f3047a85b77efc64e2dbd88067bcf19a2f9adf703"} Jul 22 21:17:03.240680 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:03.240631 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-chp6r" podStartSLOduration=1.24061645 podStartE2EDuration="1.24061645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 21:17:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 21:17:03.238453501 +0000 UTC m=+678.352845860" watchObservedRunningTime="2026-07-22 21:17:03.24061645 +0000 UTC m=+678.355008793" Jul 22 21:17:03.736340 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:03.736304 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-chp6r" Jul 22 21:17:03.741244 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:03.741222 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-chp6r" Jul 22 21:17:04.226422 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:04.226392 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-chp6r" Jul 22 21:17:04.227402 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:04.227383 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-chp6r" Jul 22 21:17:15.324032 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:15.323995 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:17:15.327538 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:15.327522 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-88kxw" Jul 22 21:17:15.330534 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:15.330508 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-269gt\"" Jul 22 21:17:15.331308 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:15.331289 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 22 21:17:15.354481 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:15.354451 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:17:15.409781 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:15.409753 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fk5p\" (UniqueName: \"kubernetes.io/projected/e8f04df2-b7b0-4e87-b649-9593bde9faf9-kube-api-access-6fk5p\") pod \"limitador-limitador-54d6d988d5-88kxw\" (UID: \"e8f04df2-b7b0-4e87-b649-9593bde9faf9\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-88kxw" Jul 22 21:17:15.409952 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:15.409798 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e8f04df2-b7b0-4e87-b649-9593bde9faf9-config-file\") pod \"limitador-limitador-54d6d988d5-88kxw\" (UID: \"e8f04df2-b7b0-4e87-b649-9593bde9faf9\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-88kxw" Jul 22 21:17:15.412701 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:15.412673 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:17:15.510822 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:15.510783 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fk5p\" (UniqueName: \"kubernetes.io/projected/e8f04df2-b7b0-4e87-b649-9593bde9faf9-kube-api-access-6fk5p\") pod \"limitador-limitador-54d6d988d5-88kxw\" (UID: \"e8f04df2-b7b0-4e87-b649-9593bde9faf9\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-88kxw" Jul 22 21:17:15.510822 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:15.510828 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e8f04df2-b7b0-4e87-b649-9593bde9faf9-config-file\") pod \"limitador-limitador-54d6d988d5-88kxw\" (UID: \"e8f04df2-b7b0-4e87-b649-9593bde9faf9\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-88kxw" Jul 22 21:17:15.511504 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:15.511486 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e8f04df2-b7b0-4e87-b649-9593bde9faf9-config-file\") pod \"limitador-limitador-54d6d988d5-88kxw\" (UID: \"e8f04df2-b7b0-4e87-b649-9593bde9faf9\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-88kxw" Jul 22 21:17:15.519580 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:15.519557 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fk5p\" (UniqueName: \"kubernetes.io/projected/e8f04df2-b7b0-4e87-b649-9593bde9faf9-kube-api-access-6fk5p\") pod \"limitador-limitador-54d6d988d5-88kxw\" (UID: \"e8f04df2-b7b0-4e87-b649-9593bde9faf9\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-88kxw" Jul 22 21:17:15.638292 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:15.638195 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-88kxw" Jul 22 21:17:15.773512 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:15.773489 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:17:15.775567 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:17:15.775534 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8f04df2_b7b0_4e87_b649_9593bde9faf9.slice/crio-99ab13ad26fbc54219ea97c04ba58f02e608908a70911e56ca0434c454138bfe WatchSource:0}: Error finding container 99ab13ad26fbc54219ea97c04ba58f02e608908a70911e56ca0434c454138bfe: Status 404 returned error can't find the container with id 99ab13ad26fbc54219ea97c04ba58f02e608908a70911e56ca0434c454138bfe Jul 22 21:17:15.777447 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:15.777431 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 21:17:15.820039 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:15.820008 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-5vs4m"] Jul 22 21:17:15.824624 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:15.824609 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-5vs4m" Jul 22 21:17:15.827091 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:15.827076 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-n7qhp\"" Jul 22 21:17:15.830005 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:15.829986 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-5vs4m"] Jul 22 21:17:15.914067 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:15.913983 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64pjn\" (UniqueName: \"kubernetes.io/projected/44bf769b-37e1-4d40-bdc8-5fb06e6e5beb-kube-api-access-64pjn\") pod \"authorino-75d556bd77-5vs4m\" (UID: \"44bf769b-37e1-4d40-bdc8-5fb06e6e5beb\") " pod="kuadrant-system/authorino-75d556bd77-5vs4m" Jul 22 21:17:16.015060 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:16.015026 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64pjn\" (UniqueName: \"kubernetes.io/projected/44bf769b-37e1-4d40-bdc8-5fb06e6e5beb-kube-api-access-64pjn\") pod \"authorino-75d556bd77-5vs4m\" (UID: \"44bf769b-37e1-4d40-bdc8-5fb06e6e5beb\") " pod="kuadrant-system/authorino-75d556bd77-5vs4m" Jul 22 21:17:16.022939 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:16.022911 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64pjn\" (UniqueName: \"kubernetes.io/projected/44bf769b-37e1-4d40-bdc8-5fb06e6e5beb-kube-api-access-64pjn\") pod \"authorino-75d556bd77-5vs4m\" (UID: \"44bf769b-37e1-4d40-bdc8-5fb06e6e5beb\") " pod="kuadrant-system/authorino-75d556bd77-5vs4m" Jul 22 21:17:16.135730 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:16.135698 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-5vs4m" Jul 22 21:17:16.266045 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:16.265999 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-5vs4m"] Jul 22 21:17:16.268153 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:17:16.268129 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod44bf769b_37e1_4d40_bdc8_5fb06e6e5beb.slice/crio-7666204566ee5af7aecfa889bb3018d7f2441b135130beb38b9fb97e8433b71e WatchSource:0}: Error finding container 7666204566ee5af7aecfa889bb3018d7f2441b135130beb38b9fb97e8433b71e: Status 404 returned error can't find the container with id 7666204566ee5af7aecfa889bb3018d7f2441b135130beb38b9fb97e8433b71e Jul 22 21:17:16.288746 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:16.288716 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-88kxw" event={"ID":"e8f04df2-b7b0-4e87-b649-9593bde9faf9","Type":"ContainerStarted","Data":"99ab13ad26fbc54219ea97c04ba58f02e608908a70911e56ca0434c454138bfe"} Jul 22 21:17:16.290005 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:16.289984 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-5vs4m" event={"ID":"44bf769b-37e1-4d40-bdc8-5fb06e6e5beb","Type":"ContainerStarted","Data":"7666204566ee5af7aecfa889bb3018d7f2441b135130beb38b9fb97e8433b71e"} Jul 22 21:17:21.331432 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:21.331390 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-5vs4m" event={"ID":"44bf769b-37e1-4d40-bdc8-5fb06e6e5beb","Type":"ContainerStarted","Data":"2fadee350aceffe350d77b746b1fcc46e8f346ac351a062527a41bd3693fd979"} Jul 22 21:17:21.332882 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:21.332852 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-88kxw" event={"ID":"e8f04df2-b7b0-4e87-b649-9593bde9faf9","Type":"ContainerStarted","Data":"114c75d4bc1dea6c5527c72d6ea7cd9fe8600faee3bfa580a4c3db0a78aa432e"} Jul 22 21:17:21.333021 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:21.333005 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-88kxw" Jul 22 21:17:21.345728 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:21.345680 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-5vs4m" podStartSLOduration=2.347547366 podStartE2EDuration="6.345664906s" totalImagesPullingTime="3.99811754s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 21:17:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 21:17:21.344805195 +0000 UTC m=+696.459197538" watchObservedRunningTime="2026-07-22 21:17:21.345664906 +0000 UTC m=+696.460057250" Jul 22 21:17:21.364936 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:21.364869 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-88kxw" podStartSLOduration=1.886014811 podStartE2EDuration="6.364846792s" totalImagesPullingTime="4.478831981s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 21:17:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 21:17:21.36107376 +0000 UTC m=+696.475466106" watchObservedRunningTime="2026-07-22 21:17:21.364846792 +0000 UTC m=+696.479239138" Jul 22 21:17:32.339371 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:32.339337 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-88kxw" Jul 22 21:17:50.913454 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:50.913413 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-5vs4m"] Jul 22 21:17:50.913969 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:50.913585 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-5vs4m" podUID="44bf769b-37e1-4d40-bdc8-5fb06e6e5beb" containerName="authorino" containerID="cri-o://2fadee350aceffe350d77b746b1fcc46e8f346ac351a062527a41bd3693fd979" gracePeriod=30 Jul 22 21:17:51.161718 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:51.161695 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-5vs4m" Jul 22 21:17:51.351524 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:51.351491 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-64pjn\" (UniqueName: \"kubernetes.io/projected/44bf769b-37e1-4d40-bdc8-5fb06e6e5beb-kube-api-access-64pjn\") pod \"44bf769b-37e1-4d40-bdc8-5fb06e6e5beb\" (UID: \"44bf769b-37e1-4d40-bdc8-5fb06e6e5beb\") " Jul 22 21:17:51.353643 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:51.353610 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44bf769b-37e1-4d40-bdc8-5fb06e6e5beb-kube-api-access-64pjn" (OuterVolumeSpecName: "kube-api-access-64pjn") pod "44bf769b-37e1-4d40-bdc8-5fb06e6e5beb" (UID: "44bf769b-37e1-4d40-bdc8-5fb06e6e5beb"). InnerVolumeSpecName "kube-api-access-64pjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 21:17:51.452699 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:51.452669 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64pjn\" (UniqueName: \"kubernetes.io/projected/44bf769b-37e1-4d40-bdc8-5fb06e6e5beb-kube-api-access-64pjn\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:17:51.497748 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:51.497714 2572 generic.go:356] "Generic (PLEG): container finished" podID="44bf769b-37e1-4d40-bdc8-5fb06e6e5beb" containerID="2fadee350aceffe350d77b746b1fcc46e8f346ac351a062527a41bd3693fd979" exitCode=0 Jul 22 21:17:51.497889 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:51.497781 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-5vs4m" Jul 22 21:17:51.497889 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:51.497789 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-5vs4m" event={"ID":"44bf769b-37e1-4d40-bdc8-5fb06e6e5beb","Type":"ContainerDied","Data":"2fadee350aceffe350d77b746b1fcc46e8f346ac351a062527a41bd3693fd979"} Jul 22 21:17:51.497889 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:51.497833 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-5vs4m" event={"ID":"44bf769b-37e1-4d40-bdc8-5fb06e6e5beb","Type":"ContainerDied","Data":"7666204566ee5af7aecfa889bb3018d7f2441b135130beb38b9fb97e8433b71e"} Jul 22 21:17:51.497889 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:51.497849 2572 scope.go:117] "RemoveContainer" containerID="2fadee350aceffe350d77b746b1fcc46e8f346ac351a062527a41bd3693fd979" Jul 22 21:17:51.506495 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:51.506475 2572 scope.go:117] "RemoveContainer" containerID="2fadee350aceffe350d77b746b1fcc46e8f346ac351a062527a41bd3693fd979" Jul 22 21:17:51.506759 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:17:51.506736 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fadee350aceffe350d77b746b1fcc46e8f346ac351a062527a41bd3693fd979\": container with ID starting with 2fadee350aceffe350d77b746b1fcc46e8f346ac351a062527a41bd3693fd979 not found: ID does not exist" containerID="2fadee350aceffe350d77b746b1fcc46e8f346ac351a062527a41bd3693fd979" Jul 22 21:17:51.506811 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:51.506768 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fadee350aceffe350d77b746b1fcc46e8f346ac351a062527a41bd3693fd979"} err="failed to get container status \"2fadee350aceffe350d77b746b1fcc46e8f346ac351a062527a41bd3693fd979\": rpc error: code = NotFound desc = could not find container \"2fadee350aceffe350d77b746b1fcc46e8f346ac351a062527a41bd3693fd979\": container with ID starting with 2fadee350aceffe350d77b746b1fcc46e8f346ac351a062527a41bd3693fd979 not found: ID does not exist" Jul 22 21:17:51.518178 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:51.518136 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-5vs4m"] Jul 22 21:17:51.528340 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:51.528318 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-5vs4m"] Jul 22 21:17:53.382847 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:17:53.382811 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44bf769b-37e1-4d40-bdc8-5fb06e6e5beb" path="/var/lib/kubelet/pods/44bf769b-37e1-4d40-bdc8-5fb06e6e5beb/volumes" Jul 22 21:18:33.953512 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:18:33.953474 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:18:41.762873 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:18:41.762840 2572 generic.go:356] "Generic (PLEG): container finished" podID="40faf561-504b-45df-9c9b-211772e1e444" containerID="e84350d914eec06546a4bea56cf053cf9b0806beae06f1828c9b41e5694c94c0" exitCode=0 Jul 22 21:18:41.763300 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:18:41.762918 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-chp6r" event={"ID":"40faf561-504b-45df-9c9b-211772e1e444","Type":"ContainerDied","Data":"e84350d914eec06546a4bea56cf053cf9b0806beae06f1828c9b41e5694c94c0"} Jul 22 21:18:41.763353 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:18:41.763339 2572 scope.go:117] "RemoveContainer" containerID="e84350d914eec06546a4bea56cf053cf9b0806beae06f1828c9b41e5694c94c0" Jul 22 21:18:41.764414 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:18:41.764384 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 22 21:18:41.764489 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:18:41.764454 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 22 21:18:41.764525 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:18:41.764486 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 22 21:18:42.770428 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:18:42.770393 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-chp6r" event={"ID":"40faf561-504b-45df-9c9b-211772e1e444","Type":"ContainerStarted","Data":"7dc2c2c37c67459aec213c3adcdec1289f45c72f5c52ee66cba2c5a9e2ebf430"} Jul 22 21:18:43.735427 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:18:43.735387 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-chp6r" Jul 22 21:18:43.735631 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:18:43.735606 2572 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-chp6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 22 21:18:43.735677 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:18:43.735659 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-chp6r" podUID="40faf561-504b-45df-9c9b-211772e1e444" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 22 21:18:44.736023 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:18:44.735982 2572 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-chp6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 22 21:18:44.736441 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:18:44.736041 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-chp6r" podUID="40faf561-504b-45df-9c9b-211772e1e444" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 22 21:18:45.736231 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:18:45.736128 2572 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-chp6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 22 21:18:45.736231 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:18:45.736210 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-chp6r" podUID="40faf561-504b-45df-9c9b-211772e1e444" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 22 21:18:46.777098 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:18:46.777071 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-chp6r" Jul 22 21:18:46.777505 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:18:46.777448 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-chp6r" Jul 22 21:18:46.778057 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:18:46.778039 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-chp6r" Jul 22 21:19:05.442999 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:19:05.442966 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:19:25.042903 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:19:25.042867 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:19:56.937274 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:19:56.937238 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:20:00.438336 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:20:00.438299 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkj7rj"] Jul 22 21:20:00.438865 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:20:00.438845 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44bf769b-37e1-4d40-bdc8-5fb06e6e5beb" containerName="authorino" Jul 22 21:20:00.438953 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:20:00.438869 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="44bf769b-37e1-4d40-bdc8-5fb06e6e5beb" containerName="authorino" Jul 22 21:20:00.439006 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:20:00.438969 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="44bf769b-37e1-4d40-bdc8-5fb06e6e5beb" containerName="authorino" Jul 22 21:20:00.442052 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:20:00.442030 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkj7rj" Jul 22 21:20:00.444464 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:20:00.444444 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 22 21:20:00.444574 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:20:00.444464 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-tm5ws\"" Jul 22 21:20:00.445498 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:20:00.445483 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 22 21:20:00.445572 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:20:00.445521 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 22 21:20:00.454751 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:20:00.454731 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkj7rj"] Jul 22 21:20:00.574264 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:20:00.574231 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9r24\" (UniqueName: \"kubernetes.io/projected/1a53c4ca-1b67-4422-8131-368b274b4dfa-kube-api-access-p9r24\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkj7rj\" (UID: \"1a53c4ca-1b67-4422-8131-368b274b4dfa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkj7rj" Jul 22 21:20:00.574264 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:20:00.574268 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a53c4ca-1b67-4422-8131-368b274b4dfa-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkj7rj\" (UID: \"1a53c4ca-1b67-4422-8131-368b274b4dfa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkj7rj" Jul 22 21:20:00.574467 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:20:00.574315 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a53c4ca-1b67-4422-8131-368b274b4dfa-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkj7rj\" (UID: \"1a53c4ca-1b67-4422-8131-368b274b4dfa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkj7rj" Jul 22 21:20:00.574467 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:20:00.574375 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a53c4ca-1b67-4422-8131-368b274b4dfa-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkj7rj\" (UID: \"1a53c4ca-1b67-4422-8131-368b274b4dfa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkj7rj" Jul 22 21:20:00.574467 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:20:00.574397 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a53c4ca-1b67-4422-8131-368b274b4dfa-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkj7rj\" (UID: \"1a53c4ca-1b67-4422-8131-368b274b4dfa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkj7rj" Jul 22 21:20:00.574467 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:20:00.574422 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a53c4ca-1b67-4422-8131-368b274b4dfa-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkj7rj\" (UID: \"1a53c4ca-1b67-4422-8131-368b274b4dfa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkj7rj" Jul 22 21:20:00.675105 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:20:00.675067 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a53c4ca-1b67-4422-8131-368b274b4dfa-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkj7rj\" (UID: \"1a53c4ca-1b67-4422-8131-368b274b4dfa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkj7rj" Jul 22 21:20:00.675306 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:20:00.675134 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a53c4ca-1b67-4422-8131-368b274b4dfa-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkj7rj\" (UID: \"1a53c4ca-1b67-4422-8131-368b274b4dfa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkj7rj" Jul 22 21:20:00.675306 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:20:00.675156 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a53c4ca-1b67-4422-8131-368b274b4dfa-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkj7rj\" (UID: \"1a53c4ca-1b67-4422-8131-368b274b4dfa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkj7rj" Jul 22 21:20:00.675306 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:20:00.675204 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a53c4ca-1b67-4422-8131-368b274b4dfa-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkj7rj\" (UID: \"1a53c4ca-1b67-4422-8131-368b274b4dfa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkj7rj" Jul 22 21:20:00.675306 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:20:00.675242 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9r24\" (UniqueName: \"kubernetes.io/projected/1a53c4ca-1b67-4422-8131-368b274b4dfa-kube-api-access-p9r24\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkj7rj\" (UID: \"1a53c4ca-1b67-4422-8131-368b274b4dfa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkj7rj" Jul 22 21:20:00.675499 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:20:00.675362 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a53c4ca-1b67-4422-8131-368b274b4dfa-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkj7rj\" (UID: \"1a53c4ca-1b67-4422-8131-368b274b4dfa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkj7rj" Jul 22 21:20:00.675601 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:20:00.675583 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a53c4ca-1b67-4422-8131-368b274b4dfa-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkj7rj\" (UID: \"1a53c4ca-1b67-4422-8131-368b274b4dfa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkj7rj" Jul 22 21:20:00.675669 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:20:00.675642 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a53c4ca-1b67-4422-8131-368b274b4dfa-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkj7rj\" (UID: \"1a53c4ca-1b67-4422-8131-368b274b4dfa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkj7rj" Jul 22 21:20:00.675730 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:20:00.675712 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a53c4ca-1b67-4422-8131-368b274b4dfa-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkj7rj\" (UID: \"1a53c4ca-1b67-4422-8131-368b274b4dfa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkj7rj" Jul 22 21:20:00.677658 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:20:00.677637 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a53c4ca-1b67-4422-8131-368b274b4dfa-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkj7rj\" (UID: \"1a53c4ca-1b67-4422-8131-368b274b4dfa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkj7rj" Jul 22 21:20:00.677812 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:20:00.677797 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a53c4ca-1b67-4422-8131-368b274b4dfa-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkj7rj\" (UID: \"1a53c4ca-1b67-4422-8131-368b274b4dfa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkj7rj" Jul 22 21:20:00.685929 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:20:00.685905 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9r24\" (UniqueName: \"kubernetes.io/projected/1a53c4ca-1b67-4422-8131-368b274b4dfa-kube-api-access-p9r24\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkj7rj\" (UID: \"1a53c4ca-1b67-4422-8131-368b274b4dfa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkj7rj" Jul 22 21:20:00.752309 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:20:00.752217 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkj7rj" Jul 22 21:20:00.890266 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:20:00.890239 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkj7rj"] Jul 22 21:20:00.892051 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:20:00.892024 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a53c4ca_1b67_4422_8131_368b274b4dfa.slice/crio-3b866bf427401ef8da129fcc95a8d1e4edce9ec5dd176dbe282b1163f89b9a19 WatchSource:0}: Error finding container 3b866bf427401ef8da129fcc95a8d1e4edce9ec5dd176dbe282b1163f89b9a19: Status 404 returned error can't find the container with id 3b866bf427401ef8da129fcc95a8d1e4edce9ec5dd176dbe282b1163f89b9a19 Jul 22 21:20:01.196607 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:20:01.196572 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkj7rj" event={"ID":"1a53c4ca-1b67-4422-8131-368b274b4dfa","Type":"ContainerStarted","Data":"3b866bf427401ef8da129fcc95a8d1e4edce9ec5dd176dbe282b1163f89b9a19"} Jul 22 21:20:02.203565 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:20:02.203503 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkj7rj" event={"ID":"1a53c4ca-1b67-4422-8131-368b274b4dfa","Type":"ContainerStarted","Data":"179ada66804d27dbdf59fb8e4f1b9e885b1c455ceaa69eb95900c2f6466379b0"} Jul 22 21:20:02.221123 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:20:02.221073 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkj7rj" podStartSLOduration=1.018830032 podStartE2EDuration="2.221057069s" totalImagesPullingTime="1.202227037s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 21:20:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 21:20:02.219299705 +0000 UTC m=+857.333692060" watchObservedRunningTime="2026-07-22 21:20:02.221057069 +0000 UTC m=+857.335449412" Jul 22 21:20:02.243888 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:20:02.243859 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:20:10.752768 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:20:10.752727 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkj7rj" Jul 22 21:20:10.753271 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:20:10.752804 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkj7rj" Jul 22 21:20:10.765114 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:20:10.765090 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkj7rj" Jul 22 21:20:11.271043 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:20:11.271015 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkj7rj" Jul 22 21:20:23.341193 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:20:23.341149 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:20:25.635891 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:20:25.635857 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-gksj5"] Jul 22 21:20:25.640773 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:20:25.640752 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gksj5" Jul 22 21:20:25.643225 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:20:25.643207 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 22 21:20:25.647450 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:20:25.647427 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-gksj5"] Jul 22 21:20:25.787374 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:20:25.787346 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42dd4d0d-9036-4fbd-97ad-9e3567b97cc2-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-gksj5\" (UID: \"42dd4d0d-9036-4fbd-97ad-9e3567b97cc2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gksj5" Jul 22 21:20:25.787498 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:20:25.787380 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42dd4d0d-9036-4fbd-97ad-9e3567b97cc2-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-gksj5\" (UID: \"42dd4d0d-9036-4fbd-97ad-9e3567b97cc2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gksj5" Jul 22 21:20:25.787498 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:20:25.787451 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42dd4d0d-9036-4fbd-97ad-9e3567b97cc2-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-gksj5\" (UID: \"42dd4d0d-9036-4fbd-97ad-9e3567b97cc2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gksj5" Jul 22 21:20:25.787498 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:20:25.787488 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42dd4d0d-9036-4fbd-97ad-9e3567b97cc2-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-gksj5\" (UID: \"42dd4d0d-9036-4fbd-97ad-9e3567b97cc2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gksj5" Jul 22 21:20:25.787606 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:20:25.787515 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42dd4d0d-9036-4fbd-97ad-9e3567b97cc2-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-gksj5\" (UID: \"42dd4d0d-9036-4fbd-97ad-9e3567b97cc2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gksj5" Jul 22 21:20:25.787606 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:20:25.787535 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mtws\" (UniqueName: \"kubernetes.io/projected/42dd4d0d-9036-4fbd-97ad-9e3567b97cc2-kube-api-access-2mtws\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-gksj5\" (UID: \"42dd4d0d-9036-4fbd-97ad-9e3567b97cc2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gksj5" Jul 22 21:20:25.888543 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:20:25.888467 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42dd4d0d-9036-4fbd-97ad-9e3567b97cc2-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-gksj5\" (UID: \"42dd4d0d-9036-4fbd-97ad-9e3567b97cc2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gksj5" Jul 22 21:20:25.888543 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:20:25.888497 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42dd4d0d-9036-4fbd-97ad-9e3567b97cc2-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-gksj5\" (UID: \"42dd4d0d-9036-4fbd-97ad-9e3567b97cc2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gksj5" Jul 22 21:20:25.888543 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:20:25.888516 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42dd4d0d-9036-4fbd-97ad-9e3567b97cc2-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-gksj5\" (UID: \"42dd4d0d-9036-4fbd-97ad-9e3567b97cc2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gksj5" Jul 22 21:20:25.888543 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:20:25.888535 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mtws\" (UniqueName: \"kubernetes.io/projected/42dd4d0d-9036-4fbd-97ad-9e3567b97cc2-kube-api-access-2mtws\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-gksj5\" (UID: \"42dd4d0d-9036-4fbd-97ad-9e3567b97cc2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gksj5" Jul 22 21:20:25.888865 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:20:25.888578 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42dd4d0d-9036-4fbd-97ad-9e3567b97cc2-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-gksj5\" (UID: \"42dd4d0d-9036-4fbd-97ad-9e3567b97cc2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gksj5" Jul 22 21:20:25.888865 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:20:25.888741 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42dd4d0d-9036-4fbd-97ad-9e3567b97cc2-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-gksj5\" (UID: \"42dd4d0d-9036-4fbd-97ad-9e3567b97cc2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gksj5" Jul 22 21:20:25.888980 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:20:25.888959 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42dd4d0d-9036-4fbd-97ad-9e3567b97cc2-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-gksj5\" (UID: \"42dd4d0d-9036-4fbd-97ad-9e3567b97cc2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gksj5" Jul 22 21:20:25.889041 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:20:25.889011 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42dd4d0d-9036-4fbd-97ad-9e3567b97cc2-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-gksj5\" (UID: \"42dd4d0d-9036-4fbd-97ad-9e3567b97cc2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gksj5" Jul 22 21:20:25.889080 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:20:25.889046 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42dd4d0d-9036-4fbd-97ad-9e3567b97cc2-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-gksj5\" (UID: \"42dd4d0d-9036-4fbd-97ad-9e3567b97cc2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gksj5" Jul 22 21:20:25.890932 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:20:25.890913 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42dd4d0d-9036-4fbd-97ad-9e3567b97cc2-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-gksj5\" (UID: \"42dd4d0d-9036-4fbd-97ad-9e3567b97cc2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gksj5" Jul 22 21:20:25.891191 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:20:25.891155 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42dd4d0d-9036-4fbd-97ad-9e3567b97cc2-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-gksj5\" (UID: \"42dd4d0d-9036-4fbd-97ad-9e3567b97cc2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gksj5" Jul 22 21:20:25.899538 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:20:25.899520 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mtws\" (UniqueName: \"kubernetes.io/projected/42dd4d0d-9036-4fbd-97ad-9e3567b97cc2-kube-api-access-2mtws\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-gksj5\" (UID: \"42dd4d0d-9036-4fbd-97ad-9e3567b97cc2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gksj5" Jul 22 21:20:25.951677 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:20:25.951652 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gksj5" Jul 22 21:20:26.082564 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:20:26.082535 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-gksj5"] Jul 22 21:20:26.084142 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:20:26.084113 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod42dd4d0d_9036_4fbd_97ad_9e3567b97cc2.slice/crio-d9ec533fe896fde39084d5dbb2b14ec625d9483cda0cd04f74c1534f2a811e4f WatchSource:0}: Error finding container d9ec533fe896fde39084d5dbb2b14ec625d9483cda0cd04f74c1534f2a811e4f: Status 404 returned error can't find the container with id d9ec533fe896fde39084d5dbb2b14ec625d9483cda0cd04f74c1534f2a811e4f Jul 22 21:20:26.335891 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:20:26.335855 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gksj5" event={"ID":"42dd4d0d-9036-4fbd-97ad-9e3567b97cc2","Type":"ContainerStarted","Data":"d9ec533fe896fde39084d5dbb2b14ec625d9483cda0cd04f74c1534f2a811e4f"} Jul 22 21:20:27.342494 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:20:27.342460 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gksj5" event={"ID":"42dd4d0d-9036-4fbd-97ad-9e3567b97cc2","Type":"ContainerStarted","Data":"d638dfd0808a2479f507e2ff6cb441c94303b613e1995bef176bd275a96cc7ca"} Jul 22 21:20:27.358611 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:20:27.358561 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gksj5" podStartSLOduration=2.060285865 podStartE2EDuration="2.358545557s" totalImagesPullingTime="298.259692ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 21:20:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 21:20:27.35737819 +0000 UTC m=+882.471770560" watchObservedRunningTime="2026-07-22 21:20:27.358545557 +0000 UTC m=+882.472937893" Jul 22 21:20:34.743352 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:20:34.743318 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:20:35.952289 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:20:35.952249 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gksj5" Jul 22 21:20:35.952678 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:20:35.952306 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gksj5" Jul 22 21:20:35.964789 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:20:35.964766 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gksj5" Jul 22 21:20:36.404597 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:20:36.404565 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-gksj5" Jul 22 21:21:27.398788 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:21:27.398757 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5b9c6cff9-gp6cx"] Jul 22 21:21:27.402103 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:21:27.402086 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b9c6cff9-gp6cx" Jul 22 21:21:27.405265 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:21:27.405244 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 22 21:21:27.405390 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:21:27.405291 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-n7qhp\"" Jul 22 21:21:27.409665 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:21:27.409466 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5b9c6cff9-gp6cx"] Jul 22 21:21:27.495015 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:21:27.494990 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6r8p6\" (UniqueName: \"kubernetes.io/projected/57c6e48c-5b69-40e8-b774-21c4b4100d5d-kube-api-access-6r8p6\") pod \"authorino-5b9c6cff9-gp6cx\" (UID: \"57c6e48c-5b69-40e8-b774-21c4b4100d5d\") " pod="kuadrant-system/authorino-5b9c6cff9-gp6cx" Jul 22 21:21:27.495128 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:21:27.495041 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/57c6e48c-5b69-40e8-b774-21c4b4100d5d-tls-cert\") pod \"authorino-5b9c6cff9-gp6cx\" (UID: \"57c6e48c-5b69-40e8-b774-21c4b4100d5d\") " pod="kuadrant-system/authorino-5b9c6cff9-gp6cx" Jul 22 21:21:27.596449 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:21:27.596422 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r8p6\" (UniqueName: \"kubernetes.io/projected/57c6e48c-5b69-40e8-b774-21c4b4100d5d-kube-api-access-6r8p6\") pod \"authorino-5b9c6cff9-gp6cx\" (UID: \"57c6e48c-5b69-40e8-b774-21c4b4100d5d\") " pod="kuadrant-system/authorino-5b9c6cff9-gp6cx" Jul 22 21:21:27.596551 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:21:27.596477 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/57c6e48c-5b69-40e8-b774-21c4b4100d5d-tls-cert\") pod \"authorino-5b9c6cff9-gp6cx\" (UID: \"57c6e48c-5b69-40e8-b774-21c4b4100d5d\") " pod="kuadrant-system/authorino-5b9c6cff9-gp6cx" Jul 22 21:21:27.598932 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:21:27.598911 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/57c6e48c-5b69-40e8-b774-21c4b4100d5d-tls-cert\") pod \"authorino-5b9c6cff9-gp6cx\" (UID: \"57c6e48c-5b69-40e8-b774-21c4b4100d5d\") " pod="kuadrant-system/authorino-5b9c6cff9-gp6cx" Jul 22 21:21:27.605215 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:21:27.605195 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r8p6\" (UniqueName: \"kubernetes.io/projected/57c6e48c-5b69-40e8-b774-21c4b4100d5d-kube-api-access-6r8p6\") pod \"authorino-5b9c6cff9-gp6cx\" (UID: \"57c6e48c-5b69-40e8-b774-21c4b4100d5d\") " pod="kuadrant-system/authorino-5b9c6cff9-gp6cx" Jul 22 21:21:27.712860 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:21:27.712792 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b9c6cff9-gp6cx" Jul 22 21:21:27.851120 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:21:27.851095 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5b9c6cff9-gp6cx"] Jul 22 21:21:27.852873 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:21:27.852848 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod57c6e48c_5b69_40e8_b774_21c4b4100d5d.slice/crio-1cec78ccaea5284148844ad90f59e03f2c14084cd486e1f4cec6f49f6c11eda3 WatchSource:0}: Error finding container 1cec78ccaea5284148844ad90f59e03f2c14084cd486e1f4cec6f49f6c11eda3: Status 404 returned error can't find the container with id 1cec78ccaea5284148844ad90f59e03f2c14084cd486e1f4cec6f49f6c11eda3 Jul 22 21:21:28.667553 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:21:28.667519 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5b9c6cff9-gp6cx" event={"ID":"57c6e48c-5b69-40e8-b774-21c4b4100d5d","Type":"ContainerStarted","Data":"3e1e3bf75e92f3a24c0e6ab260da6829b99e366a2c9e299d7ecf9fabada8b1e2"} Jul 22 21:21:28.667553 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:21:28.667561 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5b9c6cff9-gp6cx" event={"ID":"57c6e48c-5b69-40e8-b774-21c4b4100d5d","Type":"ContainerStarted","Data":"1cec78ccaea5284148844ad90f59e03f2c14084cd486e1f4cec6f49f6c11eda3"} Jul 22 21:21:28.689389 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:21:28.689337 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5b9c6cff9-gp6cx" podStartSLOduration=1.249144367 podStartE2EDuration="1.689321216s" totalImagesPullingTime="440.176849ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 21:21:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 21:21:28.688777713 +0000 UTC m=+943.803170059" watchObservedRunningTime="2026-07-22 21:21:28.689321216 +0000 UTC m=+943.803713560" Jul 22 21:21:58.041695 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:21:58.041657 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:22:07.242237 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:22:07.242124 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:22:18.448390 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:22:18.448349 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:23:58.847385 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:23:58.847346 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:24:37.337517 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:24:37.337482 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:24:46.836744 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:24:46.836704 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:24:55.755692 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:24:55.755658 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:25:06.839236 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:25:06.839125 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:25:17.544623 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:25:17.544587 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:25:27.743759 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:25:27.743726 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:25:38.738706 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:25:38.738667 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:25:48.643370 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:25:48.643329 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:26:10.740840 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:26:10.740803 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:26:22.637316 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:26:22.637274 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:26:45.238922 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:26:45.238830 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:26:55.339615 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:26:55.339581 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:27:01.550877 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:27:01.550838 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:27:11.545621 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:27:11.545571 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:27:22.745302 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:27:22.745257 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:27:31.945366 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:27:31.945328 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:27:41.233372 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:27:41.233339 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:27:50.340625 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:27:50.340591 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:29:38.147589 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:29:38.147515 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:29:50.150926 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:29:50.150887 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:29:58.970215 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:29:58.970179 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:30:11.595601 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:30:11.595566 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:30:21.007082 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:30:21.006957 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:30:31.522349 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:30:31.522316 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:30:40.395546 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:30:40.395516 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:30:51.685692 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:30:51.685659 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:31:01.288418 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:31:01.288384 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:31:12.095239 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:31:12.095149 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:31:25.601702 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:31:25.601665 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:31:43.887675 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:31:43.887636 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:31:54.352652 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:31:54.352614 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:31:59.656295 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:31:59.656257 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-71e1eaff-kserve-56b46df9f-7tkl9"] Jul 22 21:31:59.659827 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:31:59.659811 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-71e1eaff-kserve-56b46df9f-7tkl9" Jul 22 21:31:59.662626 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:31:59.662606 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-71e1eaff-kserve-self-signed-certs\"" Jul 22 21:31:59.677565 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:31:59.677541 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-71e1eaff-kserve-56b46df9f-7tkl9"] Jul 22 21:31:59.727299 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:31:59.727265 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a05c72d3-a6bb-48db-be0b-c4c70977832b-tmp-dir\") pod \"e2e-same-id-a-71e1eaff-kserve-56b46df9f-7tkl9\" (UID: \"a05c72d3-a6bb-48db-be0b-c4c70977832b\") " pod="llm/e2e-same-id-a-71e1eaff-kserve-56b46df9f-7tkl9" Jul 22 21:31:59.727454 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:31:59.727310 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvt4k\" (UniqueName: \"kubernetes.io/projected/a05c72d3-a6bb-48db-be0b-c4c70977832b-kube-api-access-rvt4k\") pod \"e2e-same-id-a-71e1eaff-kserve-56b46df9f-7tkl9\" (UID: \"a05c72d3-a6bb-48db-be0b-c4c70977832b\") " pod="llm/e2e-same-id-a-71e1eaff-kserve-56b46df9f-7tkl9" Jul 22 21:31:59.727454 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:31:59.727429 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a05c72d3-a6bb-48db-be0b-c4c70977832b-dshm\") pod \"e2e-same-id-a-71e1eaff-kserve-56b46df9f-7tkl9\" (UID: \"a05c72d3-a6bb-48db-be0b-c4c70977832b\") " pod="llm/e2e-same-id-a-71e1eaff-kserve-56b46df9f-7tkl9" Jul 22 21:31:59.727541 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:31:59.727453 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a05c72d3-a6bb-48db-be0b-c4c70977832b-model-cache\") pod \"e2e-same-id-a-71e1eaff-kserve-56b46df9f-7tkl9\" (UID: \"a05c72d3-a6bb-48db-be0b-c4c70977832b\") " pod="llm/e2e-same-id-a-71e1eaff-kserve-56b46df9f-7tkl9" Jul 22 21:31:59.727541 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:31:59.727472 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a05c72d3-a6bb-48db-be0b-c4c70977832b-home\") pod \"e2e-same-id-a-71e1eaff-kserve-56b46df9f-7tkl9\" (UID: \"a05c72d3-a6bb-48db-be0b-c4c70977832b\") " pod="llm/e2e-same-id-a-71e1eaff-kserve-56b46df9f-7tkl9" Jul 22 21:31:59.727615 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:31:59.727561 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a05c72d3-a6bb-48db-be0b-c4c70977832b-tls-certs\") pod \"e2e-same-id-a-71e1eaff-kserve-56b46df9f-7tkl9\" (UID: \"a05c72d3-a6bb-48db-be0b-c4c70977832b\") " pod="llm/e2e-same-id-a-71e1eaff-kserve-56b46df9f-7tkl9" Jul 22 21:31:59.828010 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:31:59.827975 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a05c72d3-a6bb-48db-be0b-c4c70977832b-tls-certs\") pod \"e2e-same-id-a-71e1eaff-kserve-56b46df9f-7tkl9\" (UID: \"a05c72d3-a6bb-48db-be0b-c4c70977832b\") " pod="llm/e2e-same-id-a-71e1eaff-kserve-56b46df9f-7tkl9" Jul 22 21:31:59.828010 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:31:59.828012 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a05c72d3-a6bb-48db-be0b-c4c70977832b-tmp-dir\") pod \"e2e-same-id-a-71e1eaff-kserve-56b46df9f-7tkl9\" (UID: \"a05c72d3-a6bb-48db-be0b-c4c70977832b\") " pod="llm/e2e-same-id-a-71e1eaff-kserve-56b46df9f-7tkl9" Jul 22 21:31:59.828294 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:31:59.828036 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvt4k\" (UniqueName: \"kubernetes.io/projected/a05c72d3-a6bb-48db-be0b-c4c70977832b-kube-api-access-rvt4k\") pod \"e2e-same-id-a-71e1eaff-kserve-56b46df9f-7tkl9\" (UID: \"a05c72d3-a6bb-48db-be0b-c4c70977832b\") " pod="llm/e2e-same-id-a-71e1eaff-kserve-56b46df9f-7tkl9" Jul 22 21:31:59.828294 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:31:59.828109 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a05c72d3-a6bb-48db-be0b-c4c70977832b-dshm\") pod \"e2e-same-id-a-71e1eaff-kserve-56b46df9f-7tkl9\" (UID: \"a05c72d3-a6bb-48db-be0b-c4c70977832b\") " pod="llm/e2e-same-id-a-71e1eaff-kserve-56b46df9f-7tkl9" Jul 22 21:31:59.828294 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:31:59.828124 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a05c72d3-a6bb-48db-be0b-c4c70977832b-model-cache\") pod \"e2e-same-id-a-71e1eaff-kserve-56b46df9f-7tkl9\" (UID: \"a05c72d3-a6bb-48db-be0b-c4c70977832b\") " pod="llm/e2e-same-id-a-71e1eaff-kserve-56b46df9f-7tkl9" Jul 22 21:31:59.828294 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:31:59.828143 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a05c72d3-a6bb-48db-be0b-c4c70977832b-home\") pod \"e2e-same-id-a-71e1eaff-kserve-56b46df9f-7tkl9\" (UID: \"a05c72d3-a6bb-48db-be0b-c4c70977832b\") " pod="llm/e2e-same-id-a-71e1eaff-kserve-56b46df9f-7tkl9" Jul 22 21:31:59.828488 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:31:59.828470 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a05c72d3-a6bb-48db-be0b-c4c70977832b-tmp-dir\") pod \"e2e-same-id-a-71e1eaff-kserve-56b46df9f-7tkl9\" (UID: \"a05c72d3-a6bb-48db-be0b-c4c70977832b\") " pod="llm/e2e-same-id-a-71e1eaff-kserve-56b46df9f-7tkl9" Jul 22 21:31:59.828526 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:31:59.828494 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a05c72d3-a6bb-48db-be0b-c4c70977832b-model-cache\") pod \"e2e-same-id-a-71e1eaff-kserve-56b46df9f-7tkl9\" (UID: \"a05c72d3-a6bb-48db-be0b-c4c70977832b\") " pod="llm/e2e-same-id-a-71e1eaff-kserve-56b46df9f-7tkl9" Jul 22 21:31:59.828615 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:31:59.828587 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a05c72d3-a6bb-48db-be0b-c4c70977832b-home\") pod \"e2e-same-id-a-71e1eaff-kserve-56b46df9f-7tkl9\" (UID: \"a05c72d3-a6bb-48db-be0b-c4c70977832b\") " pod="llm/e2e-same-id-a-71e1eaff-kserve-56b46df9f-7tkl9" Jul 22 21:31:59.830504 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:31:59.830476 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a05c72d3-a6bb-48db-be0b-c4c70977832b-dshm\") pod \"e2e-same-id-a-71e1eaff-kserve-56b46df9f-7tkl9\" (UID: \"a05c72d3-a6bb-48db-be0b-c4c70977832b\") " pod="llm/e2e-same-id-a-71e1eaff-kserve-56b46df9f-7tkl9" Jul 22 21:31:59.830615 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:31:59.830598 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a05c72d3-a6bb-48db-be0b-c4c70977832b-tls-certs\") pod \"e2e-same-id-a-71e1eaff-kserve-56b46df9f-7tkl9\" (UID: \"a05c72d3-a6bb-48db-be0b-c4c70977832b\") " pod="llm/e2e-same-id-a-71e1eaff-kserve-56b46df9f-7tkl9" Jul 22 21:31:59.837288 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:31:59.837267 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvt4k\" (UniqueName: \"kubernetes.io/projected/a05c72d3-a6bb-48db-be0b-c4c70977832b-kube-api-access-rvt4k\") pod \"e2e-same-id-a-71e1eaff-kserve-56b46df9f-7tkl9\" (UID: \"a05c72d3-a6bb-48db-be0b-c4c70977832b\") " pod="llm/e2e-same-id-a-71e1eaff-kserve-56b46df9f-7tkl9" Jul 22 21:31:59.969297 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:31:59.969199 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-71e1eaff-kserve-56b46df9f-7tkl9" Jul 22 21:32:00.120042 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:32:00.120018 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-71e1eaff-kserve-56b46df9f-7tkl9"] Jul 22 21:32:00.121712 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:32:00.121686 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda05c72d3_a6bb_48db_be0b_c4c70977832b.slice/crio-03fe6b0b59b64a3650de3c439db6904db29db5012cc5c363ced1f6e5ea94bae2 WatchSource:0}: Error finding container 03fe6b0b59b64a3650de3c439db6904db29db5012cc5c363ced1f6e5ea94bae2: Status 404 returned error can't find the container with id 03fe6b0b59b64a3650de3c439db6904db29db5012cc5c363ced1f6e5ea94bae2 Jul 22 21:32:01.068814 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:32:01.068778 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-71e1eaff-kserve-56b46df9f-7tkl9" event={"ID":"a05c72d3-a6bb-48db-be0b-c4c70977832b","Type":"ContainerStarted","Data":"3152346c7a7d71e70944e19d8464301c2c84d7ad91da2ace3ddf91133b1e0d34"} Jul 22 21:32:01.068814 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:32:01.068819 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-71e1eaff-kserve-56b46df9f-7tkl9" event={"ID":"a05c72d3-a6bb-48db-be0b-c4c70977832b","Type":"ContainerStarted","Data":"03fe6b0b59b64a3650de3c439db6904db29db5012cc5c363ced1f6e5ea94bae2"} Jul 22 21:32:01.092654 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:32:01.092610 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-71e1eaff-kserve-56b46df9f-7tkl9" podStartSLOduration=2.092595887 podStartE2EDuration="2.092595887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 21:31:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 21:32:01.089643879 +0000 UTC m=+1576.204036223" watchObservedRunningTime="2026-07-22 21:32:01.092595887 +0000 UTC m=+1576.206988231" Jul 22 21:32:03.858998 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:32:03.858958 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:32:09.970318 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:32:09.970237 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-71e1eaff-kserve-56b46df9f-7tkl9" Jul 22 21:32:09.970318 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:32:09.970275 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-71e1eaff-kserve-56b46df9f-7tkl9" Jul 22 21:32:09.984666 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:32:09.984641 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-71e1eaff-kserve-56b46df9f-7tkl9" Jul 22 21:32:10.126423 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:32:10.126395 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-71e1eaff-kserve-56b46df9f-7tkl9" Jul 22 21:32:16.252149 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:32:16.252112 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:32:28.521527 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:32:28.521496 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-71e1eaff-kserve-56b46df9f-7tkl9"] Jul 22 21:32:32.453814 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:32:32.453774 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:32:43.542548 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:32:43.542442 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-71e1eaff-kserve-56b46df9f-7tkl9" podUID="a05c72d3-a6bb-48db-be0b-c4c70977832b" containerName="main" containerID="cri-o://3152346c7a7d71e70944e19d8464301c2c84d7ad91da2ace3ddf91133b1e0d34" gracePeriod=45 Jul 22 21:32:43.795772 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:32:43.795718 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-71e1eaff-kserve-56b46df9f-7tkl9" Jul 22 21:32:43.828751 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:32:43.828723 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a05c72d3-a6bb-48db-be0b-c4c70977832b-dshm\") pod \"a05c72d3-a6bb-48db-be0b-c4c70977832b\" (UID: \"a05c72d3-a6bb-48db-be0b-c4c70977832b\") " Jul 22 21:32:43.828864 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:32:43.828808 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a05c72d3-a6bb-48db-be0b-c4c70977832b-tls-certs\") pod \"a05c72d3-a6bb-48db-be0b-c4c70977832b\" (UID: \"a05c72d3-a6bb-48db-be0b-c4c70977832b\") " Jul 22 21:32:43.828864 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:32:43.828840 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a05c72d3-a6bb-48db-be0b-c4c70977832b-tmp-dir\") pod \"a05c72d3-a6bb-48db-be0b-c4c70977832b\" (UID: \"a05c72d3-a6bb-48db-be0b-c4c70977832b\") " Jul 22 21:32:43.828944 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:32:43.828863 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a05c72d3-a6bb-48db-be0b-c4c70977832b-home\") pod \"a05c72d3-a6bb-48db-be0b-c4c70977832b\" (UID: \"a05c72d3-a6bb-48db-be0b-c4c70977832b\") " Jul 22 21:32:43.828988 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:32:43.828949 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rvt4k\" (UniqueName: \"kubernetes.io/projected/a05c72d3-a6bb-48db-be0b-c4c70977832b-kube-api-access-rvt4k\") pod \"a05c72d3-a6bb-48db-be0b-c4c70977832b\" (UID: \"a05c72d3-a6bb-48db-be0b-c4c70977832b\") " Jul 22 21:32:43.828988 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:32:43.828979 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a05c72d3-a6bb-48db-be0b-c4c70977832b-model-cache\") pod \"a05c72d3-a6bb-48db-be0b-c4c70977832b\" (UID: \"a05c72d3-a6bb-48db-be0b-c4c70977832b\") " Jul 22 21:32:43.829135 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:32:43.829100 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a05c72d3-a6bb-48db-be0b-c4c70977832b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a05c72d3-a6bb-48db-be0b-c4c70977832b" (UID: "a05c72d3-a6bb-48db-be0b-c4c70977832b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 21:32:43.829135 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:32:43.829109 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a05c72d3-a6bb-48db-be0b-c4c70977832b-home" (OuterVolumeSpecName: "home") pod "a05c72d3-a6bb-48db-be0b-c4c70977832b" (UID: "a05c72d3-a6bb-48db-be0b-c4c70977832b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 21:32:43.829356 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:32:43.829332 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a05c72d3-a6bb-48db-be0b-c4c70977832b-model-cache" (OuterVolumeSpecName: "model-cache") pod "a05c72d3-a6bb-48db-be0b-c4c70977832b" (UID: "a05c72d3-a6bb-48db-be0b-c4c70977832b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 21:32:43.829461 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:32:43.829358 2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a05c72d3-a6bb-48db-be0b-c4c70977832b-tmp-dir\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:32:43.829461 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:32:43.829379 2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a05c72d3-a6bb-48db-be0b-c4c70977832b-home\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:32:43.831309 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:32:43.831277 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a05c72d3-a6bb-48db-be0b-c4c70977832b-kube-api-access-rvt4k" (OuterVolumeSpecName: "kube-api-access-rvt4k") pod "a05c72d3-a6bb-48db-be0b-c4c70977832b" (UID: "a05c72d3-a6bb-48db-be0b-c4c70977832b"). InnerVolumeSpecName "kube-api-access-rvt4k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 21:32:43.831416 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:32:43.831365 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a05c72d3-a6bb-48db-be0b-c4c70977832b-dshm" (OuterVolumeSpecName: "dshm") pod "a05c72d3-a6bb-48db-be0b-c4c70977832b" (UID: "a05c72d3-a6bb-48db-be0b-c4c70977832b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 21:32:43.831416 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:32:43.831372 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a05c72d3-a6bb-48db-be0b-c4c70977832b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a05c72d3-a6bb-48db-be0b-c4c70977832b" (UID: "a05c72d3-a6bb-48db-be0b-c4c70977832b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 21:32:43.929961 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:32:43.929936 2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a05c72d3-a6bb-48db-be0b-c4c70977832b-dshm\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:32:43.929961 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:32:43.929959 2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a05c72d3-a6bb-48db-be0b-c4c70977832b-tls-certs\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:32:43.930094 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:32:43.929970 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvt4k\" (UniqueName: \"kubernetes.io/projected/a05c72d3-a6bb-48db-be0b-c4c70977832b-kube-api-access-rvt4k\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:32:43.930094 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:32:43.929980 2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a05c72d3-a6bb-48db-be0b-c4c70977832b-model-cache\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:32:44.301968 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:32:44.301932 2572 generic.go:356] "Generic (PLEG): container finished" podID="a05c72d3-a6bb-48db-be0b-c4c70977832b" containerID="3152346c7a7d71e70944e19d8464301c2c84d7ad91da2ace3ddf91133b1e0d34" exitCode=0 Jul 22 21:32:44.302152 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:32:44.301980 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-71e1eaff-kserve-56b46df9f-7tkl9" Jul 22 21:32:44.302152 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:32:44.302008 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-71e1eaff-kserve-56b46df9f-7tkl9" event={"ID":"a05c72d3-a6bb-48db-be0b-c4c70977832b","Type":"ContainerDied","Data":"3152346c7a7d71e70944e19d8464301c2c84d7ad91da2ace3ddf91133b1e0d34"} Jul 22 21:32:44.302152 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:32:44.302039 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-71e1eaff-kserve-56b46df9f-7tkl9" event={"ID":"a05c72d3-a6bb-48db-be0b-c4c70977832b","Type":"ContainerDied","Data":"03fe6b0b59b64a3650de3c439db6904db29db5012cc5c363ced1f6e5ea94bae2"} Jul 22 21:32:44.302152 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:32:44.302054 2572 scope.go:117] "RemoveContainer" containerID="3152346c7a7d71e70944e19d8464301c2c84d7ad91da2ace3ddf91133b1e0d34" Jul 22 21:32:44.311403 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:32:44.311384 2572 scope.go:117] "RemoveContainer" containerID="3152346c7a7d71e70944e19d8464301c2c84d7ad91da2ace3ddf91133b1e0d34" Jul 22 21:32:44.311648 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:32:44.311627 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3152346c7a7d71e70944e19d8464301c2c84d7ad91da2ace3ddf91133b1e0d34\": container with ID starting with 3152346c7a7d71e70944e19d8464301c2c84d7ad91da2ace3ddf91133b1e0d34 not found: ID does not exist" containerID="3152346c7a7d71e70944e19d8464301c2c84d7ad91da2ace3ddf91133b1e0d34" Jul 22 21:32:44.311709 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:32:44.311657 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3152346c7a7d71e70944e19d8464301c2c84d7ad91da2ace3ddf91133b1e0d34"} err="failed to get container status \"3152346c7a7d71e70944e19d8464301c2c84d7ad91da2ace3ddf91133b1e0d34\": rpc error: code = NotFound desc = could not find container \"3152346c7a7d71e70944e19d8464301c2c84d7ad91da2ace3ddf91133b1e0d34\": container with ID starting with 3152346c7a7d71e70944e19d8464301c2c84d7ad91da2ace3ddf91133b1e0d34 not found: ID does not exist" Jul 22 21:32:44.322608 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:32:44.322587 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-71e1eaff-kserve-56b46df9f-7tkl9"] Jul 22 21:32:44.326176 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:32:44.326144 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-71e1eaff-kserve-56b46df9f-7tkl9"] Jul 22 21:32:45.383260 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:32:45.383225 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a05c72d3-a6bb-48db-be0b-c4c70977832b" path="/var/lib/kubelet/pods/a05c72d3-a6bb-48db-be0b-c4c70977832b/volumes" Jul 22 21:32:46.346548 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:32:46.346511 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:32:53.443020 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:32:53.442984 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:33:09.743101 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:33:09.743063 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:33:19.045987 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:33:19.045951 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:33:28.343385 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:33:28.343354 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:33:36.048973 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:33:36.048934 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:33:45.649202 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:33:45.649146 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:33:55.139961 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:33:55.139929 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:34:09.151528 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:34:09.151448 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:34:18.146180 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:34:18.146132 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:34:27.541609 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:34:27.541571 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:34:39.843987 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:34:39.843950 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:34:48.944312 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:34:48.944278 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:34:55.456506 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:34:55.456471 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:35:05.141886 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:35:05.141855 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:35:13.042677 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:35:13.042635 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:35:25.350406 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:35:25.350368 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:35:34.852897 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:35:34.852859 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:35:49.056085 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:35:49.056043 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:35:58.458680 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:35:58.458642 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:36:08.148092 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:08.148059 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-69774d8e-openshift-default-5979cb8667-j42ng"] Jul 22 21:36:08.148568 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:08.148435 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a05c72d3-a6bb-48db-be0b-c4c70977832b" containerName="main" Jul 22 21:36:08.148568 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:08.148446 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="a05c72d3-a6bb-48db-be0b-c4c70977832b" containerName="main" Jul 22 21:36:08.148568 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:08.148515 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="a05c72d3-a6bb-48db-be0b-c4c70977832b" containerName="main" Jul 22 21:36:08.151199 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:08.151183 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-69774d8e-openshift-default-5979cb8667-j42ng" Jul 22 21:36:08.153760 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:08.153738 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-69774d8e-openshift-default-dockercfg-smmlz\"" Jul 22 21:36:08.176437 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:08.176412 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-69774d8e-openshift-default-5979cb8667-j42ng"] Jul 22 21:36:08.180331 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:08.180310 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84ca8ded-ea64-462f-8966-625649305efb-istio-data\") pod \"e2e-ait-69774d8e-openshift-default-5979cb8667-j42ng\" (UID: \"84ca8ded-ea64-462f-8966-625649305efb\") " pod="openshift-ingress/e2e-ait-69774d8e-openshift-default-5979cb8667-j42ng" Jul 22 21:36:08.180424 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:08.180347 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84ca8ded-ea64-462f-8966-625649305efb-workload-certs\") pod \"e2e-ait-69774d8e-openshift-default-5979cb8667-j42ng\" (UID: \"84ca8ded-ea64-462f-8966-625649305efb\") " pod="openshift-ingress/e2e-ait-69774d8e-openshift-default-5979cb8667-j42ng" Jul 22 21:36:08.180424 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:08.180377 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84ca8ded-ea64-462f-8966-625649305efb-credential-socket\") pod \"e2e-ait-69774d8e-openshift-default-5979cb8667-j42ng\" (UID: \"84ca8ded-ea64-462f-8966-625649305efb\") " pod="openshift-ingress/e2e-ait-69774d8e-openshift-default-5979cb8667-j42ng" Jul 22 21:36:08.180516 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:08.180417 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84ca8ded-ea64-462f-8966-625649305efb-istio-podinfo\") pod \"e2e-ait-69774d8e-openshift-default-5979cb8667-j42ng\" (UID: \"84ca8ded-ea64-462f-8966-625649305efb\") " pod="openshift-ingress/e2e-ait-69774d8e-openshift-default-5979cb8667-j42ng" Jul 22 21:36:08.180557 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:08.180524 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84ca8ded-ea64-462f-8966-625649305efb-workload-socket\") pod \"e2e-ait-69774d8e-openshift-default-5979cb8667-j42ng\" (UID: \"84ca8ded-ea64-462f-8966-625649305efb\") " pod="openshift-ingress/e2e-ait-69774d8e-openshift-default-5979cb8667-j42ng" Jul 22 21:36:08.180598 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:08.180570 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84ca8ded-ea64-462f-8966-625649305efb-istiod-ca-cert\") pod \"e2e-ait-69774d8e-openshift-default-5979cb8667-j42ng\" (UID: \"84ca8ded-ea64-462f-8966-625649305efb\") " pod="openshift-ingress/e2e-ait-69774d8e-openshift-default-5979cb8667-j42ng" Jul 22 21:36:08.180648 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:08.180633 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84ca8ded-ea64-462f-8966-625649305efb-istio-token\") pod \"e2e-ait-69774d8e-openshift-default-5979cb8667-j42ng\" (UID: \"84ca8ded-ea64-462f-8966-625649305efb\") " pod="openshift-ingress/e2e-ait-69774d8e-openshift-default-5979cb8667-j42ng" Jul 22 21:36:08.180705 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:08.180690 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8q88\" (UniqueName: \"kubernetes.io/projected/84ca8ded-ea64-462f-8966-625649305efb-kube-api-access-c8q88\") pod \"e2e-ait-69774d8e-openshift-default-5979cb8667-j42ng\" (UID: \"84ca8ded-ea64-462f-8966-625649305efb\") " pod="openshift-ingress/e2e-ait-69774d8e-openshift-default-5979cb8667-j42ng" Jul 22 21:36:08.180754 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:08.180721 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84ca8ded-ea64-462f-8966-625649305efb-istio-envoy\") pod \"e2e-ait-69774d8e-openshift-default-5979cb8667-j42ng\" (UID: \"84ca8ded-ea64-462f-8966-625649305efb\") " pod="openshift-ingress/e2e-ait-69774d8e-openshift-default-5979cb8667-j42ng" Jul 22 21:36:08.256966 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:08.256932 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:36:08.281778 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:08.281751 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84ca8ded-ea64-462f-8966-625649305efb-istio-data\") pod \"e2e-ait-69774d8e-openshift-default-5979cb8667-j42ng\" (UID: \"84ca8ded-ea64-462f-8966-625649305efb\") " pod="openshift-ingress/e2e-ait-69774d8e-openshift-default-5979cb8667-j42ng" Jul 22 21:36:08.281939 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:08.281787 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84ca8ded-ea64-462f-8966-625649305efb-workload-certs\") pod \"e2e-ait-69774d8e-openshift-default-5979cb8667-j42ng\" (UID: \"84ca8ded-ea64-462f-8966-625649305efb\") " pod="openshift-ingress/e2e-ait-69774d8e-openshift-default-5979cb8667-j42ng" Jul 22 21:36:08.281939 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:08.281816 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84ca8ded-ea64-462f-8966-625649305efb-credential-socket\") pod \"e2e-ait-69774d8e-openshift-default-5979cb8667-j42ng\" (UID: \"84ca8ded-ea64-462f-8966-625649305efb\") " pod="openshift-ingress/e2e-ait-69774d8e-openshift-default-5979cb8667-j42ng" Jul 22 21:36:08.281939 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:08.281838 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84ca8ded-ea64-462f-8966-625649305efb-istio-podinfo\") pod \"e2e-ait-69774d8e-openshift-default-5979cb8667-j42ng\" (UID: \"84ca8ded-ea64-462f-8966-625649305efb\") " pod="openshift-ingress/e2e-ait-69774d8e-openshift-default-5979cb8667-j42ng" Jul 22 21:36:08.281939 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:08.281879 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84ca8ded-ea64-462f-8966-625649305efb-workload-socket\") pod \"e2e-ait-69774d8e-openshift-default-5979cb8667-j42ng\" (UID: \"84ca8ded-ea64-462f-8966-625649305efb\") " pod="openshift-ingress/e2e-ait-69774d8e-openshift-default-5979cb8667-j42ng" Jul 22 21:36:08.281939 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:08.281904 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84ca8ded-ea64-462f-8966-625649305efb-istiod-ca-cert\") pod \"e2e-ait-69774d8e-openshift-default-5979cb8667-j42ng\" (UID: \"84ca8ded-ea64-462f-8966-625649305efb\") " pod="openshift-ingress/e2e-ait-69774d8e-openshift-default-5979cb8667-j42ng" Jul 22 21:36:08.282212 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:08.281991 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84ca8ded-ea64-462f-8966-625649305efb-istio-token\") pod \"e2e-ait-69774d8e-openshift-default-5979cb8667-j42ng\" (UID: \"84ca8ded-ea64-462f-8966-625649305efb\") " pod="openshift-ingress/e2e-ait-69774d8e-openshift-default-5979cb8667-j42ng" Jul 22 21:36:08.282212 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:08.282036 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8q88\" (UniqueName: \"kubernetes.io/projected/84ca8ded-ea64-462f-8966-625649305efb-kube-api-access-c8q88\") pod \"e2e-ait-69774d8e-openshift-default-5979cb8667-j42ng\" (UID: \"84ca8ded-ea64-462f-8966-625649305efb\") " pod="openshift-ingress/e2e-ait-69774d8e-openshift-default-5979cb8667-j42ng" Jul 22 21:36:08.282212 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:08.282061 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84ca8ded-ea64-462f-8966-625649305efb-istio-envoy\") pod \"e2e-ait-69774d8e-openshift-default-5979cb8667-j42ng\" (UID: \"84ca8ded-ea64-462f-8966-625649305efb\") " pod="openshift-ingress/e2e-ait-69774d8e-openshift-default-5979cb8667-j42ng" Jul 22 21:36:08.282379 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:08.282237 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84ca8ded-ea64-462f-8966-625649305efb-istio-data\") pod \"e2e-ait-69774d8e-openshift-default-5979cb8667-j42ng\" (UID: \"84ca8ded-ea64-462f-8966-625649305efb\") " pod="openshift-ingress/e2e-ait-69774d8e-openshift-default-5979cb8667-j42ng" Jul 22 21:36:08.282379 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:08.282293 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84ca8ded-ea64-462f-8966-625649305efb-credential-socket\") pod \"e2e-ait-69774d8e-openshift-default-5979cb8667-j42ng\" (UID: \"84ca8ded-ea64-462f-8966-625649305efb\") " pod="openshift-ingress/e2e-ait-69774d8e-openshift-default-5979cb8667-j42ng" Jul 22 21:36:08.282379 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:08.282345 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84ca8ded-ea64-462f-8966-625649305efb-workload-certs\") pod \"e2e-ait-69774d8e-openshift-default-5979cb8667-j42ng\" (UID: \"84ca8ded-ea64-462f-8966-625649305efb\") " pod="openshift-ingress/e2e-ait-69774d8e-openshift-default-5979cb8667-j42ng" Jul 22 21:36:08.282536 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:08.282478 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84ca8ded-ea64-462f-8966-625649305efb-workload-socket\") pod \"e2e-ait-69774d8e-openshift-default-5979cb8667-j42ng\" (UID: \"84ca8ded-ea64-462f-8966-625649305efb\") " pod="openshift-ingress/e2e-ait-69774d8e-openshift-default-5979cb8667-j42ng" Jul 22 21:36:08.282906 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:08.282885 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84ca8ded-ea64-462f-8966-625649305efb-istiod-ca-cert\") pod \"e2e-ait-69774d8e-openshift-default-5979cb8667-j42ng\" (UID: \"84ca8ded-ea64-462f-8966-625649305efb\") " pod="openshift-ingress/e2e-ait-69774d8e-openshift-default-5979cb8667-j42ng" Jul 22 21:36:08.284903 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:08.284884 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84ca8ded-ea64-462f-8966-625649305efb-istio-envoy\") pod \"e2e-ait-69774d8e-openshift-default-5979cb8667-j42ng\" (UID: \"84ca8ded-ea64-462f-8966-625649305efb\") " pod="openshift-ingress/e2e-ait-69774d8e-openshift-default-5979cb8667-j42ng" Jul 22 21:36:08.285106 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:08.285079 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84ca8ded-ea64-462f-8966-625649305efb-istio-podinfo\") pod \"e2e-ait-69774d8e-openshift-default-5979cb8667-j42ng\" (UID: \"84ca8ded-ea64-462f-8966-625649305efb\") " pod="openshift-ingress/e2e-ait-69774d8e-openshift-default-5979cb8667-j42ng" Jul 22 21:36:08.291847 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:08.291821 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84ca8ded-ea64-462f-8966-625649305efb-istio-token\") pod \"e2e-ait-69774d8e-openshift-default-5979cb8667-j42ng\" (UID: \"84ca8ded-ea64-462f-8966-625649305efb\") " pod="openshift-ingress/e2e-ait-69774d8e-openshift-default-5979cb8667-j42ng" Jul 22 21:36:08.292330 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:08.292310 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8q88\" (UniqueName: \"kubernetes.io/projected/84ca8ded-ea64-462f-8966-625649305efb-kube-api-access-c8q88\") pod \"e2e-ait-69774d8e-openshift-default-5979cb8667-j42ng\" (UID: \"84ca8ded-ea64-462f-8966-625649305efb\") " pod="openshift-ingress/e2e-ait-69774d8e-openshift-default-5979cb8667-j42ng" Jul 22 21:36:08.461419 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:08.461352 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-69774d8e-openshift-default-5979cb8667-j42ng" Jul 22 21:36:08.603252 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:08.603227 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-69774d8e-openshift-default-5979cb8667-j42ng"] Jul 22 21:36:08.604874 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:36:08.604840 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84ca8ded_ea64_462f_8966_625649305efb.slice/crio-5b430f7ff4795c7a5a705da3b5b6ab169848d94e2ef05f85ba65c2d1493f837c WatchSource:0}: Error finding container 5b430f7ff4795c7a5a705da3b5b6ab169848d94e2ef05f85ba65c2d1493f837c: Status 404 returned error can't find the container with id 5b430f7ff4795c7a5a705da3b5b6ab169848d94e2ef05f85ba65c2d1493f837c Jul 22 21:36:08.607234 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:08.607200 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 22 21:36:08.607322 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:08.607269 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 22 21:36:08.607322 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:08.607302 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 22 21:36:09.392466 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:09.392430 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-69774d8e-openshift-default-5979cb8667-j42ng" event={"ID":"84ca8ded-ea64-462f-8966-625649305efb","Type":"ContainerStarted","Data":"9a0f73888792cb8a7e04da989754f719e62d96cadecb4566559162554fe80d75"} Jul 22 21:36:09.392880 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:09.392482 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-69774d8e-openshift-default-5979cb8667-j42ng" event={"ID":"84ca8ded-ea64-462f-8966-625649305efb","Type":"ContainerStarted","Data":"5b430f7ff4795c7a5a705da3b5b6ab169848d94e2ef05f85ba65c2d1493f837c"} Jul 22 21:36:09.417213 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:09.417155 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-69774d8e-openshift-default-5979cb8667-j42ng" podStartSLOduration=1.417140405 podStartE2EDuration="1.417140405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 21:36:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 21:36:09.415093312 +0000 UTC m=+1824.529485718" watchObservedRunningTime="2026-07-22 21:36:09.417140405 +0000 UTC m=+1824.531532750" Jul 22 21:36:09.462010 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:09.461986 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-69774d8e-openshift-default-5979cb8667-j42ng" Jul 22 21:36:09.466557 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:09.466539 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-69774d8e-openshift-default-5979cb8667-j42ng" Jul 22 21:36:10.398095 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:10.398065 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-69774d8e-openshift-default-5979cb8667-j42ng" Jul 22 21:36:10.399013 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:10.398997 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-69774d8e-openshift-default-5979cb8667-j42ng" Jul 22 21:36:18.060480 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:18.060450 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:36:25.910788 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:25.910265 2572 patch_prober.go:28] interesting pod/e2e-ait-69774d8e-openshift-default-5979cb8667-j42ng container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": read tcp 10.134.0.2:50332->10.134.0.50:15021: read: connection reset by peer" start-of-body= Jul 22 21:36:25.910788 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:25.910366 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-69774d8e-openshift-default-5979cb8667-j42ng" podUID="84ca8ded-ea64-462f-8966-625649305efb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": read tcp 10.134.0.2:50332->10.134.0.50:15021: read: connection reset by peer" Jul 22 21:36:26.484116 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:26.484085 2572 generic.go:356] "Generic (PLEG): container finished" podID="84ca8ded-ea64-462f-8966-625649305efb" containerID="9a0f73888792cb8a7e04da989754f719e62d96cadecb4566559162554fe80d75" exitCode=0 Jul 22 21:36:26.484292 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:26.484179 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-69774d8e-openshift-default-5979cb8667-j42ng" event={"ID":"84ca8ded-ea64-462f-8966-625649305efb","Type":"ContainerDied","Data":"9a0f73888792cb8a7e04da989754f719e62d96cadecb4566559162554fe80d75"} Jul 22 21:36:26.484489 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:26.484475 2572 scope.go:117] "RemoveContainer" containerID="9a0f73888792cb8a7e04da989754f719e62d96cadecb4566559162554fe80d75" Jul 22 21:36:26.485426 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:26.485398 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 22 21:36:26.485492 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:26.485470 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 22 21:36:26.485530 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:26.485499 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 22 21:36:27.497443 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:27.497406 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-69774d8e-openshift-default-5979cb8667-j42ng" event={"ID":"84ca8ded-ea64-462f-8966-625649305efb","Type":"ContainerStarted","Data":"3629b10c31459b4f55082df65e2c16e07c1eed200edf60123715a98bc26fe278"} Jul 22 21:36:28.461865 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:28.461832 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-69774d8e-openshift-default-5979cb8667-j42ng" Jul 22 21:36:28.462073 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:28.462050 2572 patch_prober.go:28] interesting pod/e2e-ait-69774d8e-openshift-default-5979cb8667-j42ng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Jul 22 21:36:28.462124 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:28.462099 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-69774d8e-openshift-default-5979cb8667-j42ng" podUID="84ca8ded-ea64-462f-8966-625649305efb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Jul 22 21:36:29.462066 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:29.462032 2572 patch_prober.go:28] interesting pod/e2e-ait-69774d8e-openshift-default-5979cb8667-j42ng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Jul 22 21:36:29.462460 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:29.462081 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-69774d8e-openshift-default-5979cb8667-j42ng" podUID="84ca8ded-ea64-462f-8966-625649305efb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Jul 22 21:36:30.461687 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:30.461651 2572 patch_prober.go:28] interesting pod/e2e-ait-69774d8e-openshift-default-5979cb8667-j42ng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Jul 22 21:36:30.461852 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:30.461716 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-69774d8e-openshift-default-5979cb8667-j42ng" podUID="84ca8ded-ea64-462f-8966-625649305efb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Jul 22 21:36:31.465253 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:31.465221 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-69774d8e-openshift-default-5979cb8667-j42ng" Jul 22 21:36:31.465632 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:31.465478 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-69774d8e-openshift-default-5979cb8667-j42ng" Jul 22 21:36:31.466235 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:31.466217 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-69774d8e-openshift-default-5979cb8667-j42ng" Jul 22 21:36:49.446157 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:49.446120 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:36:51.539993 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:51.539950 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-69774d8e-openshift-default-5979cb8667-j42ng"] Jul 22 21:36:51.540445 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:51.540206 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-69774d8e-openshift-default-5979cb8667-j42ng" podUID="84ca8ded-ea64-462f-8966-625649305efb" containerName="istio-proxy" containerID="cri-o://3629b10c31459b4f55082df65e2c16e07c1eed200edf60123715a98bc26fe278" gracePeriod=30 Jul 22 21:36:56.662391 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:56.662360 2572 generic.go:356] "Generic (PLEG): container finished" podID="84ca8ded-ea64-462f-8966-625649305efb" containerID="3629b10c31459b4f55082df65e2c16e07c1eed200edf60123715a98bc26fe278" exitCode=0 Jul 22 21:36:56.662712 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:56.662431 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-69774d8e-openshift-default-5979cb8667-j42ng" event={"ID":"84ca8ded-ea64-462f-8966-625649305efb","Type":"ContainerDied","Data":"3629b10c31459b4f55082df65e2c16e07c1eed200edf60123715a98bc26fe278"} Jul 22 21:36:56.662712 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:56.662494 2572 scope.go:117] "RemoveContainer" containerID="9a0f73888792cb8a7e04da989754f719e62d96cadecb4566559162554fe80d75" Jul 22 21:36:56.793892 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:56.793871 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-69774d8e-openshift-default-5979cb8667-j42ng" Jul 22 21:36:56.820336 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:56.820314 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84ca8ded-ea64-462f-8966-625649305efb-istio-data\") pod \"84ca8ded-ea64-462f-8966-625649305efb\" (UID: \"84ca8ded-ea64-462f-8966-625649305efb\") " Jul 22 21:36:56.820489 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:56.820345 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84ca8ded-ea64-462f-8966-625649305efb-workload-socket\") pod \"84ca8ded-ea64-462f-8966-625649305efb\" (UID: \"84ca8ded-ea64-462f-8966-625649305efb\") " Jul 22 21:36:56.820489 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:56.820367 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84ca8ded-ea64-462f-8966-625649305efb-istio-token\") pod \"84ca8ded-ea64-462f-8966-625649305efb\" (UID: \"84ca8ded-ea64-462f-8966-625649305efb\") " Jul 22 21:36:56.820489 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:56.820401 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84ca8ded-ea64-462f-8966-625649305efb-istiod-ca-cert\") pod \"84ca8ded-ea64-462f-8966-625649305efb\" (UID: \"84ca8ded-ea64-462f-8966-625649305efb\") " Jul 22 21:36:56.820489 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:56.820416 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84ca8ded-ea64-462f-8966-625649305efb-istio-envoy\") pod \"84ca8ded-ea64-462f-8966-625649305efb\" (UID: \"84ca8ded-ea64-462f-8966-625649305efb\") " Jul 22 21:36:56.820489 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:56.820459 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84ca8ded-ea64-462f-8966-625649305efb-credential-socket\") pod \"84ca8ded-ea64-462f-8966-625649305efb\" (UID: \"84ca8ded-ea64-462f-8966-625649305efb\") " Jul 22 21:36:56.820489 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:56.820488 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84ca8ded-ea64-462f-8966-625649305efb-istio-podinfo\") pod \"84ca8ded-ea64-462f-8966-625649305efb\" (UID: \"84ca8ded-ea64-462f-8966-625649305efb\") " Jul 22 21:36:56.820815 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:56.820518 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84ca8ded-ea64-462f-8966-625649305efb-workload-certs\") pod \"84ca8ded-ea64-462f-8966-625649305efb\" (UID: \"84ca8ded-ea64-462f-8966-625649305efb\") " Jul 22 21:36:56.820815 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:56.820626 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84ca8ded-ea64-462f-8966-625649305efb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "84ca8ded-ea64-462f-8966-625649305efb" (UID: "84ca8ded-ea64-462f-8966-625649305efb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 21:36:56.820815 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:56.820722 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84ca8ded-ea64-462f-8966-625649305efb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "84ca8ded-ea64-462f-8966-625649305efb" (UID: "84ca8ded-ea64-462f-8966-625649305efb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 21:36:56.820979 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:56.820888 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84ca8ded-ea64-462f-8966-625649305efb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "84ca8ded-ea64-462f-8966-625649305efb" (UID: "84ca8ded-ea64-462f-8966-625649305efb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 21:36:56.820979 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:56.820961 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84ca8ded-ea64-462f-8966-625649305efb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "84ca8ded-ea64-462f-8966-625649305efb" (UID: "84ca8ded-ea64-462f-8966-625649305efb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 21:36:56.821095 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:56.821045 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c8q88\" (UniqueName: \"kubernetes.io/projected/84ca8ded-ea64-462f-8966-625649305efb-kube-api-access-c8q88\") pod \"84ca8ded-ea64-462f-8966-625649305efb\" (UID: \"84ca8ded-ea64-462f-8966-625649305efb\") " Jul 22 21:36:56.821386 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:56.821356 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84ca8ded-ea64-462f-8966-625649305efb-istio-data" (OuterVolumeSpecName: "istio-data") pod "84ca8ded-ea64-462f-8966-625649305efb" (UID: "84ca8ded-ea64-462f-8966-625649305efb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 21:36:56.821643 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:56.821625 2572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84ca8ded-ea64-462f-8966-625649305efb-workload-certs\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:36:56.821705 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:56.821650 2572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84ca8ded-ea64-462f-8966-625649305efb-istio-data\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:36:56.821705 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:56.821662 2572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84ca8ded-ea64-462f-8966-625649305efb-workload-socket\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:36:56.821705 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:56.821674 2572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84ca8ded-ea64-462f-8966-625649305efb-istiod-ca-cert\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:36:56.821705 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:56.821689 2572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84ca8ded-ea64-462f-8966-625649305efb-credential-socket\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:36:56.822947 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:56.822923 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/84ca8ded-ea64-462f-8966-625649305efb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "84ca8ded-ea64-462f-8966-625649305efb" (UID: "84ca8ded-ea64-462f-8966-625649305efb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 21:36:56.823144 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:56.823125 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84ca8ded-ea64-462f-8966-625649305efb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "84ca8ded-ea64-462f-8966-625649305efb" (UID: "84ca8ded-ea64-462f-8966-625649305efb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 21:36:56.823236 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:56.823149 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84ca8ded-ea64-462f-8966-625649305efb-istio-token" (OuterVolumeSpecName: "istio-token") pod "84ca8ded-ea64-462f-8966-625649305efb" (UID: "84ca8ded-ea64-462f-8966-625649305efb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 21:36:56.823457 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:56.823435 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84ca8ded-ea64-462f-8966-625649305efb-kube-api-access-c8q88" (OuterVolumeSpecName: "kube-api-access-c8q88") pod "84ca8ded-ea64-462f-8966-625649305efb" (UID: "84ca8ded-ea64-462f-8966-625649305efb"). InnerVolumeSpecName "kube-api-access-c8q88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 21:36:56.922392 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:56.922366 2572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84ca8ded-ea64-462f-8966-625649305efb-istio-envoy\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:36:56.922392 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:56.922389 2572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84ca8ded-ea64-462f-8966-625649305efb-istio-podinfo\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:36:56.922538 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:56.922398 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8q88\" (UniqueName: \"kubernetes.io/projected/84ca8ded-ea64-462f-8966-625649305efb-kube-api-access-c8q88\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:36:56.922538 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:56.922407 2572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84ca8ded-ea64-462f-8966-625649305efb-istio-token\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:36:57.668326 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:57.668244 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-69774d8e-openshift-default-5979cb8667-j42ng" event={"ID":"84ca8ded-ea64-462f-8966-625649305efb","Type":"ContainerDied","Data":"5b430f7ff4795c7a5a705da3b5b6ab169848d94e2ef05f85ba65c2d1493f837c"} Jul 22 21:36:57.668326 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:57.668274 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-69774d8e-openshift-default-5979cb8667-j42ng" Jul 22 21:36:57.668326 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:57.668306 2572 scope.go:117] "RemoveContainer" containerID="3629b10c31459b4f55082df65e2c16e07c1eed200edf60123715a98bc26fe278" Jul 22 21:36:57.686493 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:57.686471 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-69774d8e-openshift-default-5979cb8667-j42ng"] Jul 22 21:36:57.692177 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:57.692144 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-69774d8e-openshift-default-5979cb8667-j42ng"] Jul 22 21:36:59.382595 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:36:59.382564 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84ca8ded-ea64-462f-8966-625649305efb" path="/var/lib/kubelet/pods/84ca8ded-ea64-462f-8966-625649305efb/volumes" Jul 22 21:37:01.561155 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:37:01.561119 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:37:37.943888 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:37:37.943854 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:37:49.549427 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:37:49.549391 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:38:20.850918 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:38:20.850889 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:38:35.139245 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:38:35.139204 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:39:06.725486 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:39:06.725405 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:40:34.179594 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:40:34.179560 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:40:43.749175 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:40:43.749139 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:40:55.979285 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:40:55.979249 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:41:17.388931 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:41:17.388882 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:41:58.453583 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:41:58.453499 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:42:06.067722 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:42:06.067692 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-b765b624/e2e-lifecycle-model-b765b624-kserve-749c7596cf-vjccb"] Jul 22 21:42:06.068088 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:42:06.068052 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84ca8ded-ea64-462f-8966-625649305efb" containerName="istio-proxy" Jul 22 21:42:06.068088 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:42:06.068062 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="84ca8ded-ea64-462f-8966-625649305efb" containerName="istio-proxy" Jul 22 21:42:06.068181 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:42:06.068147 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="84ca8ded-ea64-462f-8966-625649305efb" containerName="istio-proxy" Jul 22 21:42:06.071074 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:42:06.071057 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-b765b624/e2e-lifecycle-model-b765b624-kserve-749c7596cf-vjccb" Jul 22 21:42:06.074556 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:42:06.074536 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-b765b624\"/\"openshift-service-ca.crt\"" Jul 22 21:42:06.075858 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:42:06.075842 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-b765b624\"/\"kube-root-ca.crt\"" Jul 22 21:42:06.075939 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:42:06.075863 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-b765b624\"/\"e2e-lifecycle-model-b765b624-kserve-self-signed-certs\"" Jul 22 21:42:06.075939 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:42:06.075863 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-b765b624\"/\"default-dockercfg-j87rz\"" Jul 22 21:42:06.100129 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:42:06.100104 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-b765b624/e2e-lifecycle-model-b765b624-kserve-749c7596cf-vjccb"] Jul 22 21:42:06.229459 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:42:06.229435 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e64b4646-d034-4155-9cde-054f0793d1e9-tls-certs\") pod \"e2e-lifecycle-model-b765b624-kserve-749c7596cf-vjccb\" (UID: \"e64b4646-d034-4155-9cde-054f0793d1e9\") " pod="ai-tenant-e2e-mt-b765b624/e2e-lifecycle-model-b765b624-kserve-749c7596cf-vjccb" Jul 22 21:42:06.229591 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:42:06.229469 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hp7t\" (UniqueName: \"kubernetes.io/projected/e64b4646-d034-4155-9cde-054f0793d1e9-kube-api-access-9hp7t\") pod \"e2e-lifecycle-model-b765b624-kserve-749c7596cf-vjccb\" (UID: \"e64b4646-d034-4155-9cde-054f0793d1e9\") " pod="ai-tenant-e2e-mt-b765b624/e2e-lifecycle-model-b765b624-kserve-749c7596cf-vjccb" Jul 22 21:42:06.229591 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:42:06.229499 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e64b4646-d034-4155-9cde-054f0793d1e9-tmp-dir\") pod \"e2e-lifecycle-model-b765b624-kserve-749c7596cf-vjccb\" (UID: \"e64b4646-d034-4155-9cde-054f0793d1e9\") " pod="ai-tenant-e2e-mt-b765b624/e2e-lifecycle-model-b765b624-kserve-749c7596cf-vjccb" Jul 22 21:42:06.229591 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:42:06.229548 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e64b4646-d034-4155-9cde-054f0793d1e9-dshm\") pod \"e2e-lifecycle-model-b765b624-kserve-749c7596cf-vjccb\" (UID: \"e64b4646-d034-4155-9cde-054f0793d1e9\") " pod="ai-tenant-e2e-mt-b765b624/e2e-lifecycle-model-b765b624-kserve-749c7596cf-vjccb" Jul 22 21:42:06.229767 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:42:06.229595 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e64b4646-d034-4155-9cde-054f0793d1e9-home\") pod \"e2e-lifecycle-model-b765b624-kserve-749c7596cf-vjccb\" (UID: \"e64b4646-d034-4155-9cde-054f0793d1e9\") " pod="ai-tenant-e2e-mt-b765b624/e2e-lifecycle-model-b765b624-kserve-749c7596cf-vjccb" Jul 22 21:42:06.229767 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:42:06.229625 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e64b4646-d034-4155-9cde-054f0793d1e9-model-cache\") pod \"e2e-lifecycle-model-b765b624-kserve-749c7596cf-vjccb\" (UID: \"e64b4646-d034-4155-9cde-054f0793d1e9\") " pod="ai-tenant-e2e-mt-b765b624/e2e-lifecycle-model-b765b624-kserve-749c7596cf-vjccb" Jul 22 21:42:06.330482 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:42:06.330397 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e64b4646-d034-4155-9cde-054f0793d1e9-home\") pod \"e2e-lifecycle-model-b765b624-kserve-749c7596cf-vjccb\" (UID: \"e64b4646-d034-4155-9cde-054f0793d1e9\") " pod="ai-tenant-e2e-mt-b765b624/e2e-lifecycle-model-b765b624-kserve-749c7596cf-vjccb" Jul 22 21:42:06.330482 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:42:06.330436 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e64b4646-d034-4155-9cde-054f0793d1e9-model-cache\") pod \"e2e-lifecycle-model-b765b624-kserve-749c7596cf-vjccb\" (UID: \"e64b4646-d034-4155-9cde-054f0793d1e9\") " pod="ai-tenant-e2e-mt-b765b624/e2e-lifecycle-model-b765b624-kserve-749c7596cf-vjccb" Jul 22 21:42:06.330690 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:42:06.330495 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e64b4646-d034-4155-9cde-054f0793d1e9-tls-certs\") pod \"e2e-lifecycle-model-b765b624-kserve-749c7596cf-vjccb\" (UID: \"e64b4646-d034-4155-9cde-054f0793d1e9\") " pod="ai-tenant-e2e-mt-b765b624/e2e-lifecycle-model-b765b624-kserve-749c7596cf-vjccb" Jul 22 21:42:06.330690 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:42:06.330526 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hp7t\" (UniqueName: \"kubernetes.io/projected/e64b4646-d034-4155-9cde-054f0793d1e9-kube-api-access-9hp7t\") pod \"e2e-lifecycle-model-b765b624-kserve-749c7596cf-vjccb\" (UID: \"e64b4646-d034-4155-9cde-054f0793d1e9\") " pod="ai-tenant-e2e-mt-b765b624/e2e-lifecycle-model-b765b624-kserve-749c7596cf-vjccb" Jul 22 21:42:06.330690 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:42:06.330580 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e64b4646-d034-4155-9cde-054f0793d1e9-tmp-dir\") pod \"e2e-lifecycle-model-b765b624-kserve-749c7596cf-vjccb\" (UID: \"e64b4646-d034-4155-9cde-054f0793d1e9\") " pod="ai-tenant-e2e-mt-b765b624/e2e-lifecycle-model-b765b624-kserve-749c7596cf-vjccb" Jul 22 21:42:06.330690 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:42:06.330616 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e64b4646-d034-4155-9cde-054f0793d1e9-dshm\") pod \"e2e-lifecycle-model-b765b624-kserve-749c7596cf-vjccb\" (UID: \"e64b4646-d034-4155-9cde-054f0793d1e9\") " pod="ai-tenant-e2e-mt-b765b624/e2e-lifecycle-model-b765b624-kserve-749c7596cf-vjccb" Jul 22 21:42:06.330907 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:42:06.330779 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e64b4646-d034-4155-9cde-054f0793d1e9-home\") pod \"e2e-lifecycle-model-b765b624-kserve-749c7596cf-vjccb\" (UID: \"e64b4646-d034-4155-9cde-054f0793d1e9\") " pod="ai-tenant-e2e-mt-b765b624/e2e-lifecycle-model-b765b624-kserve-749c7596cf-vjccb" Jul 22 21:42:06.330907 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:42:06.330831 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e64b4646-d034-4155-9cde-054f0793d1e9-model-cache\") pod \"e2e-lifecycle-model-b765b624-kserve-749c7596cf-vjccb\" (UID: \"e64b4646-d034-4155-9cde-054f0793d1e9\") " pod="ai-tenant-e2e-mt-b765b624/e2e-lifecycle-model-b765b624-kserve-749c7596cf-vjccb" Jul 22 21:42:06.331000 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:42:06.330902 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e64b4646-d034-4155-9cde-054f0793d1e9-tmp-dir\") pod \"e2e-lifecycle-model-b765b624-kserve-749c7596cf-vjccb\" (UID: \"e64b4646-d034-4155-9cde-054f0793d1e9\") " pod="ai-tenant-e2e-mt-b765b624/e2e-lifecycle-model-b765b624-kserve-749c7596cf-vjccb" Jul 22 21:42:06.332857 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:42:06.332840 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e64b4646-d034-4155-9cde-054f0793d1e9-dshm\") pod \"e2e-lifecycle-model-b765b624-kserve-749c7596cf-vjccb\" (UID: \"e64b4646-d034-4155-9cde-054f0793d1e9\") " pod="ai-tenant-e2e-mt-b765b624/e2e-lifecycle-model-b765b624-kserve-749c7596cf-vjccb" Jul 22 21:42:06.333025 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:42:06.333008 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e64b4646-d034-4155-9cde-054f0793d1e9-tls-certs\") pod \"e2e-lifecycle-model-b765b624-kserve-749c7596cf-vjccb\" (UID: \"e64b4646-d034-4155-9cde-054f0793d1e9\") " pod="ai-tenant-e2e-mt-b765b624/e2e-lifecycle-model-b765b624-kserve-749c7596cf-vjccb" Jul 22 21:42:06.341384 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:42:06.341363 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hp7t\" (UniqueName: \"kubernetes.io/projected/e64b4646-d034-4155-9cde-054f0793d1e9-kube-api-access-9hp7t\") pod \"e2e-lifecycle-model-b765b624-kserve-749c7596cf-vjccb\" (UID: \"e64b4646-d034-4155-9cde-054f0793d1e9\") " pod="ai-tenant-e2e-mt-b765b624/e2e-lifecycle-model-b765b624-kserve-749c7596cf-vjccb" Jul 22 21:42:06.379683 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:42:06.379661 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-b765b624/e2e-lifecycle-model-b765b624-kserve-749c7596cf-vjccb" Jul 22 21:42:06.517641 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:42:06.517612 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-b765b624/e2e-lifecycle-model-b765b624-kserve-749c7596cf-vjccb"] Jul 22 21:42:06.518667 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:42:06.518638 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode64b4646_d034_4155_9cde_054f0793d1e9.slice/crio-c2dfa5627ea2c5bbd9a1057145e4c2ae811d5533618eddb055261dbe61096357 WatchSource:0}: Error finding container c2dfa5627ea2c5bbd9a1057145e4c2ae811d5533618eddb055261dbe61096357: Status 404 returned error can't find the container with id c2dfa5627ea2c5bbd9a1057145e4c2ae811d5533618eddb055261dbe61096357 Jul 22 21:42:07.273368 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:42:07.273336 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-b765b624/e2e-lifecycle-model-b765b624-kserve-749c7596cf-vjccb" event={"ID":"e64b4646-d034-4155-9cde-054f0793d1e9","Type":"ContainerStarted","Data":"5ddca1095e82ca6366503858f4be51b6101f5aad8fdd186eacfc18ea43406881"} Jul 22 21:42:07.273737 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:42:07.273376 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-b765b624/e2e-lifecycle-model-b765b624-kserve-749c7596cf-vjccb" event={"ID":"e64b4646-d034-4155-9cde-054f0793d1e9","Type":"ContainerStarted","Data":"c2dfa5627ea2c5bbd9a1057145e4c2ae811d5533618eddb055261dbe61096357"} Jul 22 21:42:07.311464 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:42:07.311409 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-b765b624/e2e-lifecycle-model-b765b624-kserve-749c7596cf-vjccb" podStartSLOduration=1.311394898 podStartE2EDuration="1.311394898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 21:42:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 21:42:07.309874655 +0000 UTC m=+2182.424266998" watchObservedRunningTime="2026-07-22 21:42:07.311394898 +0000 UTC m=+2182.425787278" Jul 22 21:42:11.849904 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:42:11.849866 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:42:16.380257 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:42:16.380213 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-b765b624/e2e-lifecycle-model-b765b624-kserve-749c7596cf-vjccb" Jul 22 21:42:16.380257 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:42:16.380264 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-b765b624/e2e-lifecycle-model-b765b624-kserve-749c7596cf-vjccb" Jul 22 21:42:16.392496 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:42:16.392477 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-b765b624/e2e-lifecycle-model-b765b624-kserve-749c7596cf-vjccb" Jul 22 21:42:17.336590 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:42:17.336562 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-b765b624/e2e-lifecycle-model-b765b624-kserve-749c7596cf-vjccb" Jul 22 21:42:23.048586 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:42:23.048551 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:42:43.655502 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:42:43.655467 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:42:54.175393 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:42:54.175363 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:42:56.933692 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:42:56.933660 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-b765b624/e2e-lifecycle-model-b765b624-kserve-749c7596cf-vjccb"] Jul 22 21:43:11.953888 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:43:11.953809 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-b765b624/e2e-lifecycle-model-b765b624-kserve-749c7596cf-vjccb" podUID="e64b4646-d034-4155-9cde-054f0793d1e9" containerName="main" containerID="cri-o://5ddca1095e82ca6366503858f4be51b6101f5aad8fdd186eacfc18ea43406881" gracePeriod=45 Jul 22 21:43:12.196990 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:43:12.196965 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-b765b624/e2e-lifecycle-model-b765b624-kserve-749c7596cf-vjccb" Jul 22 21:43:12.273373 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:43:12.273300 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e64b4646-d034-4155-9cde-054f0793d1e9-tls-certs\") pod \"e64b4646-d034-4155-9cde-054f0793d1e9\" (UID: \"e64b4646-d034-4155-9cde-054f0793d1e9\") " Jul 22 21:43:12.273373 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:43:12.273336 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e64b4646-d034-4155-9cde-054f0793d1e9-tmp-dir\") pod \"e64b4646-d034-4155-9cde-054f0793d1e9\" (UID: \"e64b4646-d034-4155-9cde-054f0793d1e9\") " Jul 22 21:43:12.273373 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:43:12.273362 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9hp7t\" (UniqueName: \"kubernetes.io/projected/e64b4646-d034-4155-9cde-054f0793d1e9-kube-api-access-9hp7t\") pod \"e64b4646-d034-4155-9cde-054f0793d1e9\" (UID: \"e64b4646-d034-4155-9cde-054f0793d1e9\") " Jul 22 21:43:12.273643 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:43:12.273391 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e64b4646-d034-4155-9cde-054f0793d1e9-dshm\") pod \"e64b4646-d034-4155-9cde-054f0793d1e9\" (UID: \"e64b4646-d034-4155-9cde-054f0793d1e9\") " Jul 22 21:43:12.273643 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:43:12.273427 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e64b4646-d034-4155-9cde-054f0793d1e9-home\") pod \"e64b4646-d034-4155-9cde-054f0793d1e9\" (UID: \"e64b4646-d034-4155-9cde-054f0793d1e9\") " Jul 22 21:43:12.273643 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:43:12.273440 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e64b4646-d034-4155-9cde-054f0793d1e9-model-cache\") pod \"e64b4646-d034-4155-9cde-054f0793d1e9\" (UID: \"e64b4646-d034-4155-9cde-054f0793d1e9\") " Jul 22 21:43:12.273643 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:43:12.273610 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e64b4646-d034-4155-9cde-054f0793d1e9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e64b4646-d034-4155-9cde-054f0793d1e9" (UID: "e64b4646-d034-4155-9cde-054f0793d1e9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 21:43:12.273861 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:43:12.273700 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e64b4646-d034-4155-9cde-054f0793d1e9-home" (OuterVolumeSpecName: "home") pod "e64b4646-d034-4155-9cde-054f0793d1e9" (UID: "e64b4646-d034-4155-9cde-054f0793d1e9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 21:43:12.273861 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:43:12.273789 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e64b4646-d034-4155-9cde-054f0793d1e9-model-cache" (OuterVolumeSpecName: "model-cache") pod "e64b4646-d034-4155-9cde-054f0793d1e9" (UID: "e64b4646-d034-4155-9cde-054f0793d1e9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 21:43:12.273970 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:43:12.273860 2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e64b4646-d034-4155-9cde-054f0793d1e9-tmp-dir\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:43:12.273970 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:43:12.273879 2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e64b4646-d034-4155-9cde-054f0793d1e9-home\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:43:12.275728 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:43:12.275694 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e64b4646-d034-4155-9cde-054f0793d1e9-kube-api-access-9hp7t" (OuterVolumeSpecName: "kube-api-access-9hp7t") pod "e64b4646-d034-4155-9cde-054f0793d1e9" (UID: "e64b4646-d034-4155-9cde-054f0793d1e9"). InnerVolumeSpecName "kube-api-access-9hp7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 21:43:12.275728 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:43:12.275710 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e64b4646-d034-4155-9cde-054f0793d1e9-dshm" (OuterVolumeSpecName: "dshm") pod "e64b4646-d034-4155-9cde-054f0793d1e9" (UID: "e64b4646-d034-4155-9cde-054f0793d1e9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 21:43:12.275873 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:43:12.275770 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e64b4646-d034-4155-9cde-054f0793d1e9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e64b4646-d034-4155-9cde-054f0793d1e9" (UID: "e64b4646-d034-4155-9cde-054f0793d1e9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 21:43:12.374701 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:43:12.374669 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hp7t\" (UniqueName: \"kubernetes.io/projected/e64b4646-d034-4155-9cde-054f0793d1e9-kube-api-access-9hp7t\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:43:12.374701 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:43:12.374698 2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e64b4646-d034-4155-9cde-054f0793d1e9-dshm\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:43:12.374880 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:43:12.374711 2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e64b4646-d034-4155-9cde-054f0793d1e9-model-cache\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:43:12.374880 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:43:12.374723 2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e64b4646-d034-4155-9cde-054f0793d1e9-tls-certs\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:43:12.615689 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:43:12.615653 2572 generic.go:356] "Generic (PLEG): container finished" podID="e64b4646-d034-4155-9cde-054f0793d1e9" containerID="5ddca1095e82ca6366503858f4be51b6101f5aad8fdd186eacfc18ea43406881" exitCode=0 Jul 22 21:43:12.615867 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:43:12.615695 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-b765b624/e2e-lifecycle-model-b765b624-kserve-749c7596cf-vjccb" Jul 22 21:43:12.615867 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:43:12.615740 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-b765b624/e2e-lifecycle-model-b765b624-kserve-749c7596cf-vjccb" event={"ID":"e64b4646-d034-4155-9cde-054f0793d1e9","Type":"ContainerDied","Data":"5ddca1095e82ca6366503858f4be51b6101f5aad8fdd186eacfc18ea43406881"} Jul 22 21:43:12.615867 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:43:12.615789 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-b765b624/e2e-lifecycle-model-b765b624-kserve-749c7596cf-vjccb" event={"ID":"e64b4646-d034-4155-9cde-054f0793d1e9","Type":"ContainerDied","Data":"c2dfa5627ea2c5bbd9a1057145e4c2ae811d5533618eddb055261dbe61096357"} Jul 22 21:43:12.615867 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:43:12.615811 2572 scope.go:117] "RemoveContainer" containerID="5ddca1095e82ca6366503858f4be51b6101f5aad8fdd186eacfc18ea43406881" Jul 22 21:43:12.625202 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:43:12.625185 2572 scope.go:117] "RemoveContainer" containerID="5ddca1095e82ca6366503858f4be51b6101f5aad8fdd186eacfc18ea43406881" Jul 22 21:43:12.625471 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:43:12.625450 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ddca1095e82ca6366503858f4be51b6101f5aad8fdd186eacfc18ea43406881\": container with ID starting with 5ddca1095e82ca6366503858f4be51b6101f5aad8fdd186eacfc18ea43406881 not found: ID does not exist" containerID="5ddca1095e82ca6366503858f4be51b6101f5aad8fdd186eacfc18ea43406881" Jul 22 21:43:12.625535 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:43:12.625478 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ddca1095e82ca6366503858f4be51b6101f5aad8fdd186eacfc18ea43406881"} err="failed to get container status \"5ddca1095e82ca6366503858f4be51b6101f5aad8fdd186eacfc18ea43406881\": rpc error: code = NotFound desc = could not find container \"5ddca1095e82ca6366503858f4be51b6101f5aad8fdd186eacfc18ea43406881\": container with ID starting with 5ddca1095e82ca6366503858f4be51b6101f5aad8fdd186eacfc18ea43406881 not found: ID does not exist" Jul 22 21:43:12.657779 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:43:12.657754 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-b765b624/e2e-lifecycle-model-b765b624-kserve-749c7596cf-vjccb"] Jul 22 21:43:12.663417 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:43:12.663395 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-b765b624/e2e-lifecycle-model-b765b624-kserve-749c7596cf-vjccb"] Jul 22 21:43:13.383075 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:43:13.383038 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e64b4646-d034-4155-9cde-054f0793d1e9" path="/var/lib/kubelet/pods/e64b4646-d034-4155-9cde-054f0793d1e9/volumes" Jul 22 21:43:44.992455 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:43:44.992416 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:43:54.556083 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:43:54.556048 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:44:45.890561 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:44:45.890473 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:44:55.254402 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:44:55.254372 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:45:05.946127 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:45:05.946094 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-3fc251/test-model-3fc251-kserve-69f58969cc-spdjq"] Jul 22 21:45:05.946788 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:45:05.946768 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e64b4646-d034-4155-9cde-054f0793d1e9" containerName="main" Jul 22 21:45:05.946832 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:45:05.946793 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="e64b4646-d034-4155-9cde-054f0793d1e9" containerName="main" Jul 22 21:45:05.946832 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:45:05.946826 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84ca8ded-ea64-462f-8966-625649305efb" containerName="istio-proxy" Jul 22 21:45:05.946894 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:45:05.946835 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="84ca8ded-ea64-462f-8966-625649305efb" containerName="istio-proxy" Jul 22 21:45:05.946926 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:45:05.946917 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="84ca8ded-ea64-462f-8966-625649305efb" containerName="istio-proxy" Jul 22 21:45:05.946960 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:45:05.946932 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="e64b4646-d034-4155-9cde-054f0793d1e9" containerName="main" Jul 22 21:45:05.950375 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:45:05.950355 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-3fc251/test-model-3fc251-kserve-69f58969cc-spdjq" Jul 22 21:45:05.953906 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:45:05.953884 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-3fc251\"/\"openshift-service-ca.crt\"" Jul 22 21:45:05.954036 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:45:05.953888 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-3fc251\"/\"default-dockercfg-44bjd\"" Jul 22 21:45:05.954036 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:45:05.953913 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-3fc251\"/\"kube-root-ca.crt\"" Jul 22 21:45:05.954036 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:45:05.954020 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-3fc251\"/\"test-model-3fc251-kserve-self-signed-certs\"" Jul 22 21:45:05.960245 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:45:05.960225 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-3fc251/test-model-3fc251-kserve-69f58969cc-spdjq"] Jul 22 21:45:06.110926 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:45:06.110900 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/547a1360-0f83-422f-97ec-36a5778ddb2c-tls-certs\") pod \"test-model-3fc251-kserve-69f58969cc-spdjq\" (UID: \"547a1360-0f83-422f-97ec-36a5778ddb2c\") " pod="ai-tenant-e2e-inf-a-3fc251/test-model-3fc251-kserve-69f58969cc-spdjq" Jul 22 21:45:06.111057 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:45:06.110939 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/547a1360-0f83-422f-97ec-36a5778ddb2c-model-cache\") pod \"test-model-3fc251-kserve-69f58969cc-spdjq\" (UID: \"547a1360-0f83-422f-97ec-36a5778ddb2c\") " pod="ai-tenant-e2e-inf-a-3fc251/test-model-3fc251-kserve-69f58969cc-spdjq" Jul 22 21:45:06.111057 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:45:06.110968 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/547a1360-0f83-422f-97ec-36a5778ddb2c-tmp-dir\") pod \"test-model-3fc251-kserve-69f58969cc-spdjq\" (UID: \"547a1360-0f83-422f-97ec-36a5778ddb2c\") " pod="ai-tenant-e2e-inf-a-3fc251/test-model-3fc251-kserve-69f58969cc-spdjq" Jul 22 21:45:06.111129 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:45:06.111058 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/547a1360-0f83-422f-97ec-36a5778ddb2c-dshm\") pod \"test-model-3fc251-kserve-69f58969cc-spdjq\" (UID: \"547a1360-0f83-422f-97ec-36a5778ddb2c\") " pod="ai-tenant-e2e-inf-a-3fc251/test-model-3fc251-kserve-69f58969cc-spdjq" Jul 22 21:45:06.111129 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:45:06.111088 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wc64c\" (UniqueName: \"kubernetes.io/projected/547a1360-0f83-422f-97ec-36a5778ddb2c-kube-api-access-wc64c\") pod \"test-model-3fc251-kserve-69f58969cc-spdjq\" (UID: \"547a1360-0f83-422f-97ec-36a5778ddb2c\") " pod="ai-tenant-e2e-inf-a-3fc251/test-model-3fc251-kserve-69f58969cc-spdjq" Jul 22 21:45:06.111129 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:45:06.111124 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/547a1360-0f83-422f-97ec-36a5778ddb2c-home\") pod \"test-model-3fc251-kserve-69f58969cc-spdjq\" (UID: \"547a1360-0f83-422f-97ec-36a5778ddb2c\") " pod="ai-tenant-e2e-inf-a-3fc251/test-model-3fc251-kserve-69f58969cc-spdjq" Jul 22 21:45:06.211613 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:45:06.211545 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/547a1360-0f83-422f-97ec-36a5778ddb2c-dshm\") pod \"test-model-3fc251-kserve-69f58969cc-spdjq\" (UID: \"547a1360-0f83-422f-97ec-36a5778ddb2c\") " pod="ai-tenant-e2e-inf-a-3fc251/test-model-3fc251-kserve-69f58969cc-spdjq" Jul 22 21:45:06.211613 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:45:06.211576 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wc64c\" (UniqueName: \"kubernetes.io/projected/547a1360-0f83-422f-97ec-36a5778ddb2c-kube-api-access-wc64c\") pod \"test-model-3fc251-kserve-69f58969cc-spdjq\" (UID: \"547a1360-0f83-422f-97ec-36a5778ddb2c\") " pod="ai-tenant-e2e-inf-a-3fc251/test-model-3fc251-kserve-69f58969cc-spdjq" Jul 22 21:45:06.211613 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:45:06.211598 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/547a1360-0f83-422f-97ec-36a5778ddb2c-home\") pod \"test-model-3fc251-kserve-69f58969cc-spdjq\" (UID: \"547a1360-0f83-422f-97ec-36a5778ddb2c\") " pod="ai-tenant-e2e-inf-a-3fc251/test-model-3fc251-kserve-69f58969cc-spdjq" Jul 22 21:45:06.211836 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:45:06.211666 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/547a1360-0f83-422f-97ec-36a5778ddb2c-tls-certs\") pod \"test-model-3fc251-kserve-69f58969cc-spdjq\" (UID: \"547a1360-0f83-422f-97ec-36a5778ddb2c\") " pod="ai-tenant-e2e-inf-a-3fc251/test-model-3fc251-kserve-69f58969cc-spdjq" Jul 22 21:45:06.211836 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:45:06.211696 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/547a1360-0f83-422f-97ec-36a5778ddb2c-model-cache\") pod \"test-model-3fc251-kserve-69f58969cc-spdjq\" (UID: \"547a1360-0f83-422f-97ec-36a5778ddb2c\") " pod="ai-tenant-e2e-inf-a-3fc251/test-model-3fc251-kserve-69f58969cc-spdjq" Jul 22 21:45:06.211836 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:45:06.211720 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/547a1360-0f83-422f-97ec-36a5778ddb2c-tmp-dir\") pod \"test-model-3fc251-kserve-69f58969cc-spdjq\" (UID: \"547a1360-0f83-422f-97ec-36a5778ddb2c\") " pod="ai-tenant-e2e-inf-a-3fc251/test-model-3fc251-kserve-69f58969cc-spdjq" Jul 22 21:45:06.212029 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:45:06.212004 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/547a1360-0f83-422f-97ec-36a5778ddb2c-home\") pod \"test-model-3fc251-kserve-69f58969cc-spdjq\" (UID: \"547a1360-0f83-422f-97ec-36a5778ddb2c\") " pod="ai-tenant-e2e-inf-a-3fc251/test-model-3fc251-kserve-69f58969cc-spdjq" Jul 22 21:45:06.212121 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:45:06.212101 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/547a1360-0f83-422f-97ec-36a5778ddb2c-model-cache\") pod \"test-model-3fc251-kserve-69f58969cc-spdjq\" (UID: \"547a1360-0f83-422f-97ec-36a5778ddb2c\") " pod="ai-tenant-e2e-inf-a-3fc251/test-model-3fc251-kserve-69f58969cc-spdjq" Jul 22 21:45:06.212203 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:45:06.212120 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/547a1360-0f83-422f-97ec-36a5778ddb2c-tmp-dir\") pod \"test-model-3fc251-kserve-69f58969cc-spdjq\" (UID: \"547a1360-0f83-422f-97ec-36a5778ddb2c\") " pod="ai-tenant-e2e-inf-a-3fc251/test-model-3fc251-kserve-69f58969cc-spdjq" Jul 22 21:45:06.213950 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:45:06.213930 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/547a1360-0f83-422f-97ec-36a5778ddb2c-dshm\") pod \"test-model-3fc251-kserve-69f58969cc-spdjq\" (UID: \"547a1360-0f83-422f-97ec-36a5778ddb2c\") " pod="ai-tenant-e2e-inf-a-3fc251/test-model-3fc251-kserve-69f58969cc-spdjq" Jul 22 21:45:06.214221 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:45:06.214205 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/547a1360-0f83-422f-97ec-36a5778ddb2c-tls-certs\") pod \"test-model-3fc251-kserve-69f58969cc-spdjq\" (UID: \"547a1360-0f83-422f-97ec-36a5778ddb2c\") " pod="ai-tenant-e2e-inf-a-3fc251/test-model-3fc251-kserve-69f58969cc-spdjq" Jul 22 21:45:06.221285 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:45:06.221261 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc64c\" (UniqueName: \"kubernetes.io/projected/547a1360-0f83-422f-97ec-36a5778ddb2c-kube-api-access-wc64c\") pod \"test-model-3fc251-kserve-69f58969cc-spdjq\" (UID: \"547a1360-0f83-422f-97ec-36a5778ddb2c\") " pod="ai-tenant-e2e-inf-a-3fc251/test-model-3fc251-kserve-69f58969cc-spdjq" Jul 22 21:45:06.261962 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:45:06.261944 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-3fc251/test-model-3fc251-kserve-69f58969cc-spdjq" Jul 22 21:45:06.394249 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:45:06.394218 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-3fc251/test-model-3fc251-kserve-69f58969cc-spdjq"] Jul 22 21:45:06.396222 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:45:06.396198 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod547a1360_0f83_422f_97ec_36a5778ddb2c.slice/crio-306cf8e27645ceda90b9f673684bc75ee410d7c2a363a173120990a83e8fc7fb WatchSource:0}: Error finding container 306cf8e27645ceda90b9f673684bc75ee410d7c2a363a173120990a83e8fc7fb: Status 404 returned error can't find the container with id 306cf8e27645ceda90b9f673684bc75ee410d7c2a363a173120990a83e8fc7fb Jul 22 21:45:07.218798 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:45:07.218758 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-3fc251/test-model-3fc251-kserve-69f58969cc-spdjq" event={"ID":"547a1360-0f83-422f-97ec-36a5778ddb2c","Type":"ContainerStarted","Data":"95e36b21ec2babd29cb7e54b0892af57e9728902c7ad395d91d947179aa5ee81"} Jul 22 21:45:07.218798 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:45:07.218801 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-3fc251/test-model-3fc251-kserve-69f58969cc-spdjq" event={"ID":"547a1360-0f83-422f-97ec-36a5778ddb2c","Type":"ContainerStarted","Data":"306cf8e27645ceda90b9f673684bc75ee410d7c2a363a173120990a83e8fc7fb"} Jul 22 21:45:07.237999 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:45:07.237953 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-3fc251/test-model-3fc251-kserve-69f58969cc-spdjq" podStartSLOduration=2.237938743 podStartE2EDuration="2.237938743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 21:45:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 21:45:07.237584528 +0000 UTC m=+2362.351976871" watchObservedRunningTime="2026-07-22 21:45:07.237938743 +0000 UTC m=+2362.352331086" Jul 22 21:45:10.148029 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:45:10.147987 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:45:16.262865 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:45:16.262836 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-3fc251/test-model-3fc251-kserve-69f58969cc-spdjq" Jul 22 21:45:16.263354 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:45:16.262879 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-3fc251/test-model-3fc251-kserve-69f58969cc-spdjq" Jul 22 21:45:16.275379 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:45:16.275359 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-3fc251/test-model-3fc251-kserve-69f58969cc-spdjq" Jul 22 21:45:16.286118 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:45:16.286099 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-3fc251/test-model-3fc251-kserve-69f58969cc-spdjq" Jul 22 21:45:20.747865 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:45:20.747833 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:45:30.470696 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:45:30.470661 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:45:46.904674 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:45:46.904642 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:46:08.909191 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:46:08.909051 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-3fc251/test-model-3fc251-kserve-69f58969cc-spdjq"] Jul 22 21:46:13.508248 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:46:13.508217 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:46:23.928530 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:46:23.928492 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-3fc251/test-model-3fc251-kserve-69f58969cc-spdjq" podUID="547a1360-0f83-422f-97ec-36a5778ddb2c" containerName="main" containerID="cri-o://95e36b21ec2babd29cb7e54b0892af57e9728902c7ad395d91d947179aa5ee81" gracePeriod=45 Jul 22 21:46:24.179043 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:46:24.178987 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-3fc251/test-model-3fc251-kserve-69f58969cc-spdjq" Jul 22 21:46:24.345995 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:46:24.345965 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/547a1360-0f83-422f-97ec-36a5778ddb2c-tls-certs\") pod \"547a1360-0f83-422f-97ec-36a5778ddb2c\" (UID: \"547a1360-0f83-422f-97ec-36a5778ddb2c\") " Jul 22 21:46:24.346131 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:46:24.346027 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/547a1360-0f83-422f-97ec-36a5778ddb2c-tmp-dir\") pod \"547a1360-0f83-422f-97ec-36a5778ddb2c\" (UID: \"547a1360-0f83-422f-97ec-36a5778ddb2c\") " Jul 22 21:46:24.346131 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:46:24.346045 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/547a1360-0f83-422f-97ec-36a5778ddb2c-home\") pod \"547a1360-0f83-422f-97ec-36a5778ddb2c\" (UID: \"547a1360-0f83-422f-97ec-36a5778ddb2c\") " Jul 22 21:46:24.346131 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:46:24.346085 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/547a1360-0f83-422f-97ec-36a5778ddb2c-model-cache\") pod \"547a1360-0f83-422f-97ec-36a5778ddb2c\" (UID: \"547a1360-0f83-422f-97ec-36a5778ddb2c\") " Jul 22 21:46:24.346313 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:46:24.346131 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wc64c\" (UniqueName: \"kubernetes.io/projected/547a1360-0f83-422f-97ec-36a5778ddb2c-kube-api-access-wc64c\") pod \"547a1360-0f83-422f-97ec-36a5778ddb2c\" (UID: \"547a1360-0f83-422f-97ec-36a5778ddb2c\") " Jul 22 21:46:24.346313 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:46:24.346179 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/547a1360-0f83-422f-97ec-36a5778ddb2c-dshm\") pod \"547a1360-0f83-422f-97ec-36a5778ddb2c\" (UID: \"547a1360-0f83-422f-97ec-36a5778ddb2c\") " Jul 22 21:46:24.346422 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:46:24.346344 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/547a1360-0f83-422f-97ec-36a5778ddb2c-model-cache" (OuterVolumeSpecName: "model-cache") pod "547a1360-0f83-422f-97ec-36a5778ddb2c" (UID: "547a1360-0f83-422f-97ec-36a5778ddb2c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 21:46:24.346422 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:46:24.346381 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/547a1360-0f83-422f-97ec-36a5778ddb2c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "547a1360-0f83-422f-97ec-36a5778ddb2c" (UID: "547a1360-0f83-422f-97ec-36a5778ddb2c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 21:46:24.346422 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:46:24.346396 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/547a1360-0f83-422f-97ec-36a5778ddb2c-home" (OuterVolumeSpecName: "home") pod "547a1360-0f83-422f-97ec-36a5778ddb2c" (UID: "547a1360-0f83-422f-97ec-36a5778ddb2c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 21:46:24.346575 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:46:24.346545 2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/547a1360-0f83-422f-97ec-36a5778ddb2c-model-cache\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:46:24.346575 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:46:24.346564 2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/547a1360-0f83-422f-97ec-36a5778ddb2c-tmp-dir\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:46:24.346690 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:46:24.346576 2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/547a1360-0f83-422f-97ec-36a5778ddb2c-home\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:46:24.348372 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:46:24.348351 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/547a1360-0f83-422f-97ec-36a5778ddb2c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "547a1360-0f83-422f-97ec-36a5778ddb2c" (UID: "547a1360-0f83-422f-97ec-36a5778ddb2c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 21:46:24.348486 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:46:24.348438 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/547a1360-0f83-422f-97ec-36a5778ddb2c-dshm" (OuterVolumeSpecName: "dshm") pod "547a1360-0f83-422f-97ec-36a5778ddb2c" (UID: "547a1360-0f83-422f-97ec-36a5778ddb2c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 21:46:24.348548 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:46:24.348500 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/547a1360-0f83-422f-97ec-36a5778ddb2c-kube-api-access-wc64c" (OuterVolumeSpecName: "kube-api-access-wc64c") pod "547a1360-0f83-422f-97ec-36a5778ddb2c" (UID: "547a1360-0f83-422f-97ec-36a5778ddb2c"). InnerVolumeSpecName "kube-api-access-wc64c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 21:46:24.447765 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:46:24.447713 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wc64c\" (UniqueName: \"kubernetes.io/projected/547a1360-0f83-422f-97ec-36a5778ddb2c-kube-api-access-wc64c\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:46:24.447765 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:46:24.447734 2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/547a1360-0f83-422f-97ec-36a5778ddb2c-dshm\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:46:24.447765 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:46:24.447744 2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/547a1360-0f83-422f-97ec-36a5778ddb2c-tls-certs\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:46:24.610386 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:46:24.610351 2572 generic.go:356] "Generic (PLEG): container finished" podID="547a1360-0f83-422f-97ec-36a5778ddb2c" containerID="95e36b21ec2babd29cb7e54b0892af57e9728902c7ad395d91d947179aa5ee81" exitCode=0 Jul 22 21:46:24.610519 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:46:24.610397 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-3fc251/test-model-3fc251-kserve-69f58969cc-spdjq" Jul 22 21:46:24.610519 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:46:24.610438 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-3fc251/test-model-3fc251-kserve-69f58969cc-spdjq" event={"ID":"547a1360-0f83-422f-97ec-36a5778ddb2c","Type":"ContainerDied","Data":"95e36b21ec2babd29cb7e54b0892af57e9728902c7ad395d91d947179aa5ee81"} Jul 22 21:46:24.610519 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:46:24.610484 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-3fc251/test-model-3fc251-kserve-69f58969cc-spdjq" event={"ID":"547a1360-0f83-422f-97ec-36a5778ddb2c","Type":"ContainerDied","Data":"306cf8e27645ceda90b9f673684bc75ee410d7c2a363a173120990a83e8fc7fb"} Jul 22 21:46:24.610519 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:46:24.610503 2572 scope.go:117] "RemoveContainer" containerID="95e36b21ec2babd29cb7e54b0892af57e9728902c7ad395d91d947179aa5ee81" Jul 22 21:46:24.620189 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:46:24.620143 2572 scope.go:117] "RemoveContainer" containerID="95e36b21ec2babd29cb7e54b0892af57e9728902c7ad395d91d947179aa5ee81" Jul 22 21:46:24.620424 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:46:24.620405 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95e36b21ec2babd29cb7e54b0892af57e9728902c7ad395d91d947179aa5ee81\": container with ID starting with 95e36b21ec2babd29cb7e54b0892af57e9728902c7ad395d91d947179aa5ee81 not found: ID does not exist" containerID="95e36b21ec2babd29cb7e54b0892af57e9728902c7ad395d91d947179aa5ee81" Jul 22 21:46:24.620497 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:46:24.620446 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95e36b21ec2babd29cb7e54b0892af57e9728902c7ad395d91d947179aa5ee81"} err="failed to get container status \"95e36b21ec2babd29cb7e54b0892af57e9728902c7ad395d91d947179aa5ee81\": rpc error: code = NotFound desc = could not find container \"95e36b21ec2babd29cb7e54b0892af57e9728902c7ad395d91d947179aa5ee81\": container with ID starting with 95e36b21ec2babd29cb7e54b0892af57e9728902c7ad395d91d947179aa5ee81 not found: ID does not exist" Jul 22 21:46:24.647785 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:46:24.647760 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-3fc251/test-model-3fc251-kserve-69f58969cc-spdjq"] Jul 22 21:46:24.654118 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:46:24.654095 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-3fc251/test-model-3fc251-kserve-69f58969cc-spdjq"] Jul 22 21:46:25.388239 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:46:25.388207 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="547a1360-0f83-422f-97ec-36a5778ddb2c" path="/var/lib/kubelet/pods/547a1360-0f83-422f-97ec-36a5778ddb2c/volumes" Jul 22 21:46:25.629558 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:46:25.629518 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:46:46.148575 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:46:46.148541 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:47:06.270568 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:47:06.270529 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:47:23.910432 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:47:23.910398 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:47:40.159142 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:47:40.159053 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:48:07.290058 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:48:07.290023 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:48:23.751339 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:48:23.751304 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:48:39.685329 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:48:39.685299 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:48:45.968769 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:48:45.968737 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-bddd55/auth-test-model-bddd55-kserve-689996889f-h6vpv"] Jul 22 21:48:45.969150 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:48:45.969109 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="547a1360-0f83-422f-97ec-36a5778ddb2c" containerName="main" Jul 22 21:48:45.969150 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:48:45.969120 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="547a1360-0f83-422f-97ec-36a5778ddb2c" containerName="main" Jul 22 21:48:45.969247 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:48:45.969202 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="547a1360-0f83-422f-97ec-36a5778ddb2c" containerName="main" Jul 22 21:48:45.972377 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:48:45.972362 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-bddd55/auth-test-model-bddd55-kserve-689996889f-h6vpv" Jul 22 21:48:45.976367 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:48:45.976342 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-bddd55\"/\"default-dockercfg-sjlcf\"" Jul 22 21:48:45.976367 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:48:45.976351 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-bddd55\"/\"auth-test-model-bddd55-kserve-self-signed-certs\"" Jul 22 21:48:45.976367 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:48:45.976367 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-bddd55\"/\"openshift-service-ca.crt\"" Jul 22 21:48:45.976620 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:48:45.976418 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-bddd55\"/\"kube-root-ca.crt\"" Jul 22 21:48:45.976672 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:48:45.976619 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad7d0701-ebf9-41a0-a385-835326d6ed7e-dshm\") pod \"auth-test-model-bddd55-kserve-689996889f-h6vpv\" (UID: \"ad7d0701-ebf9-41a0-a385-835326d6ed7e\") " pod="ai-tenant-e2e-shared-a-bddd55/auth-test-model-bddd55-kserve-689996889f-h6vpv" Jul 22 21:48:45.976672 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:48:45.976642 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad7d0701-ebf9-41a0-a385-835326d6ed7e-model-cache\") pod \"auth-test-model-bddd55-kserve-689996889f-h6vpv\" (UID: \"ad7d0701-ebf9-41a0-a385-835326d6ed7e\") " pod="ai-tenant-e2e-shared-a-bddd55/auth-test-model-bddd55-kserve-689996889f-h6vpv" Jul 22 21:48:45.976672 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:48:45.976663 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad7d0701-ebf9-41a0-a385-835326d6ed7e-tls-certs\") pod \"auth-test-model-bddd55-kserve-689996889f-h6vpv\" (UID: \"ad7d0701-ebf9-41a0-a385-835326d6ed7e\") " pod="ai-tenant-e2e-shared-a-bddd55/auth-test-model-bddd55-kserve-689996889f-h6vpv" Jul 22 21:48:45.976784 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:48:45.976682 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxw7z\" (UniqueName: \"kubernetes.io/projected/ad7d0701-ebf9-41a0-a385-835326d6ed7e-kube-api-access-xxw7z\") pod \"auth-test-model-bddd55-kserve-689996889f-h6vpv\" (UID: \"ad7d0701-ebf9-41a0-a385-835326d6ed7e\") " pod="ai-tenant-e2e-shared-a-bddd55/auth-test-model-bddd55-kserve-689996889f-h6vpv" Jul 22 21:48:45.976784 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:48:45.976753 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad7d0701-ebf9-41a0-a385-835326d6ed7e-home\") pod \"auth-test-model-bddd55-kserve-689996889f-h6vpv\" (UID: \"ad7d0701-ebf9-41a0-a385-835326d6ed7e\") " pod="ai-tenant-e2e-shared-a-bddd55/auth-test-model-bddd55-kserve-689996889f-h6vpv" Jul 22 21:48:45.976784 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:48:45.976780 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad7d0701-ebf9-41a0-a385-835326d6ed7e-tmp-dir\") pod \"auth-test-model-bddd55-kserve-689996889f-h6vpv\" (UID: \"ad7d0701-ebf9-41a0-a385-835326d6ed7e\") " pod="ai-tenant-e2e-shared-a-bddd55/auth-test-model-bddd55-kserve-689996889f-h6vpv" Jul 22 21:48:45.988344 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:48:45.988299 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-bddd55/auth-test-model-bddd55-kserve-689996889f-h6vpv"] Jul 22 21:48:46.077473 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:48:46.077450 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad7d0701-ebf9-41a0-a385-835326d6ed7e-dshm\") pod \"auth-test-model-bddd55-kserve-689996889f-h6vpv\" (UID: \"ad7d0701-ebf9-41a0-a385-835326d6ed7e\") " pod="ai-tenant-e2e-shared-a-bddd55/auth-test-model-bddd55-kserve-689996889f-h6vpv" Jul 22 21:48:46.077628 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:48:46.077477 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad7d0701-ebf9-41a0-a385-835326d6ed7e-model-cache\") pod \"auth-test-model-bddd55-kserve-689996889f-h6vpv\" (UID: \"ad7d0701-ebf9-41a0-a385-835326d6ed7e\") " pod="ai-tenant-e2e-shared-a-bddd55/auth-test-model-bddd55-kserve-689996889f-h6vpv" Jul 22 21:48:46.077628 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:48:46.077498 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad7d0701-ebf9-41a0-a385-835326d6ed7e-tls-certs\") pod \"auth-test-model-bddd55-kserve-689996889f-h6vpv\" (UID: \"ad7d0701-ebf9-41a0-a385-835326d6ed7e\") " pod="ai-tenant-e2e-shared-a-bddd55/auth-test-model-bddd55-kserve-689996889f-h6vpv" Jul 22 21:48:46.077628 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:48:46.077517 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxw7z\" (UniqueName: \"kubernetes.io/projected/ad7d0701-ebf9-41a0-a385-835326d6ed7e-kube-api-access-xxw7z\") pod \"auth-test-model-bddd55-kserve-689996889f-h6vpv\" (UID: \"ad7d0701-ebf9-41a0-a385-835326d6ed7e\") " pod="ai-tenant-e2e-shared-a-bddd55/auth-test-model-bddd55-kserve-689996889f-h6vpv" Jul 22 21:48:46.077628 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:48:46.077552 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad7d0701-ebf9-41a0-a385-835326d6ed7e-home\") pod \"auth-test-model-bddd55-kserve-689996889f-h6vpv\" (UID: \"ad7d0701-ebf9-41a0-a385-835326d6ed7e\") " pod="ai-tenant-e2e-shared-a-bddd55/auth-test-model-bddd55-kserve-689996889f-h6vpv" Jul 22 21:48:46.077628 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:48:46.077576 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad7d0701-ebf9-41a0-a385-835326d6ed7e-tmp-dir\") pod \"auth-test-model-bddd55-kserve-689996889f-h6vpv\" (UID: \"ad7d0701-ebf9-41a0-a385-835326d6ed7e\") " pod="ai-tenant-e2e-shared-a-bddd55/auth-test-model-bddd55-kserve-689996889f-h6vpv" Jul 22 21:48:46.077932 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:48:46.077897 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad7d0701-ebf9-41a0-a385-835326d6ed7e-model-cache\") pod \"auth-test-model-bddd55-kserve-689996889f-h6vpv\" (UID: \"ad7d0701-ebf9-41a0-a385-835326d6ed7e\") " pod="ai-tenant-e2e-shared-a-bddd55/auth-test-model-bddd55-kserve-689996889f-h6vpv" Jul 22 21:48:46.078010 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:48:46.077939 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad7d0701-ebf9-41a0-a385-835326d6ed7e-tmp-dir\") pod \"auth-test-model-bddd55-kserve-689996889f-h6vpv\" (UID: \"ad7d0701-ebf9-41a0-a385-835326d6ed7e\") " pod="ai-tenant-e2e-shared-a-bddd55/auth-test-model-bddd55-kserve-689996889f-h6vpv" Jul 22 21:48:46.078010 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:48:46.077981 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad7d0701-ebf9-41a0-a385-835326d6ed7e-home\") pod \"auth-test-model-bddd55-kserve-689996889f-h6vpv\" (UID: \"ad7d0701-ebf9-41a0-a385-835326d6ed7e\") " pod="ai-tenant-e2e-shared-a-bddd55/auth-test-model-bddd55-kserve-689996889f-h6vpv" Jul 22 21:48:46.080077 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:48:46.080056 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad7d0701-ebf9-41a0-a385-835326d6ed7e-tls-certs\") pod \"auth-test-model-bddd55-kserve-689996889f-h6vpv\" (UID: \"ad7d0701-ebf9-41a0-a385-835326d6ed7e\") " pod="ai-tenant-e2e-shared-a-bddd55/auth-test-model-bddd55-kserve-689996889f-h6vpv" Jul 22 21:48:46.080247 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:48:46.080226 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad7d0701-ebf9-41a0-a385-835326d6ed7e-dshm\") pod \"auth-test-model-bddd55-kserve-689996889f-h6vpv\" (UID: \"ad7d0701-ebf9-41a0-a385-835326d6ed7e\") " pod="ai-tenant-e2e-shared-a-bddd55/auth-test-model-bddd55-kserve-689996889f-h6vpv" Jul 22 21:48:46.087280 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:48:46.087260 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxw7z\" (UniqueName: \"kubernetes.io/projected/ad7d0701-ebf9-41a0-a385-835326d6ed7e-kube-api-access-xxw7z\") pod \"auth-test-model-bddd55-kserve-689996889f-h6vpv\" (UID: \"ad7d0701-ebf9-41a0-a385-835326d6ed7e\") " pod="ai-tenant-e2e-shared-a-bddd55/auth-test-model-bddd55-kserve-689996889f-h6vpv" Jul 22 21:48:46.290631 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:48:46.290553 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-bddd55/auth-test-model-bddd55-kserve-689996889f-h6vpv" Jul 22 21:48:46.427664 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:48:46.427639 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-bddd55/auth-test-model-bddd55-kserve-689996889f-h6vpv"] Jul 22 21:48:46.429975 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:48:46.429941 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podad7d0701_ebf9_41a0_a385_835326d6ed7e.slice/crio-ee116b7717d1171bc61ecba40099ad92128ddcc249a0cc620b8760c66daf7e9d WatchSource:0}: Error finding container ee116b7717d1171bc61ecba40099ad92128ddcc249a0cc620b8760c66daf7e9d: Status 404 returned error can't find the container with id ee116b7717d1171bc61ecba40099ad92128ddcc249a0cc620b8760c66daf7e9d Jul 22 21:48:47.343059 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:48:47.343024 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-bddd55/auth-test-model-bddd55-kserve-689996889f-h6vpv" event={"ID":"ad7d0701-ebf9-41a0-a385-835326d6ed7e","Type":"ContainerStarted","Data":"db9ca469ba9f591287132e30155a23f1b10d7507450c03caee8ca5ac11423604"} Jul 22 21:48:47.343426 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:48:47.343065 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-bddd55/auth-test-model-bddd55-kserve-689996889f-h6vpv" event={"ID":"ad7d0701-ebf9-41a0-a385-835326d6ed7e","Type":"ContainerStarted","Data":"ee116b7717d1171bc61ecba40099ad92128ddcc249a0cc620b8760c66daf7e9d"} Jul 22 21:48:47.365700 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:48:47.365660 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-bddd55/auth-test-model-bddd55-kserve-689996889f-h6vpv" podStartSLOduration=2.365645069 podStartE2EDuration="2.365645069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 21:48:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 21:48:47.362578469 +0000 UTC m=+2582.476970811" watchObservedRunningTime="2026-07-22 21:48:47.365645069 +0000 UTC m=+2582.480037411" Jul 22 21:48:48.928434 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:48:48.928404 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:48:56.291532 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:48:56.291500 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-bddd55/auth-test-model-bddd55-kserve-689996889f-h6vpv" Jul 22 21:48:56.291932 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:48:56.291546 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-bddd55/auth-test-model-bddd55-kserve-689996889f-h6vpv" Jul 22 21:48:56.303390 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:48:56.303366 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-bddd55/auth-test-model-bddd55-kserve-689996889f-h6vpv" Jul 22 21:48:56.401331 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:48:56.401297 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-bddd55/auth-test-model-bddd55-kserve-689996889f-h6vpv" Jul 22 21:49:22.107539 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:49:22.107468 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-e44515/auth-test-model-e44515-kserve-6bfd4bd8dc-tjjks"] Jul 22 21:49:22.111331 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:49:22.111307 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e44515/auth-test-model-e44515-kserve-6bfd4bd8dc-tjjks" Jul 22 21:49:22.113911 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:49:22.113886 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-e44515\"/\"openshift-service-ca.crt\"" Jul 22 21:49:22.115069 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:49:22.115048 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-e44515\"/\"kube-root-ca.crt\"" Jul 22 21:49:22.115069 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:49:22.115062 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-e44515\"/\"default-dockercfg-hqsw9\"" Jul 22 21:49:22.115265 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:49:22.115116 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-e44515\"/\"auth-test-model-e44515-kserve-self-signed-certs\"" Jul 22 21:49:22.129849 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:49:22.129825 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-e44515/auth-test-model-e44515-kserve-6bfd4bd8dc-tjjks"] Jul 22 21:49:22.160637 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:49:22.160612 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/673f693f-2849-4212-b30b-22462f2681a3-tls-certs\") pod \"auth-test-model-e44515-kserve-6bfd4bd8dc-tjjks\" (UID: \"673f693f-2849-4212-b30b-22462f2681a3\") " pod="ai-tenant-e2e-shared-b-e44515/auth-test-model-e44515-kserve-6bfd4bd8dc-tjjks" Jul 22 21:49:22.160721 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:49:22.160694 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/673f693f-2849-4212-b30b-22462f2681a3-model-cache\") pod \"auth-test-model-e44515-kserve-6bfd4bd8dc-tjjks\" (UID: \"673f693f-2849-4212-b30b-22462f2681a3\") " pod="ai-tenant-e2e-shared-b-e44515/auth-test-model-e44515-kserve-6bfd4bd8dc-tjjks" Jul 22 21:49:22.160762 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:49:22.160728 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/673f693f-2849-4212-b30b-22462f2681a3-tmp-dir\") pod \"auth-test-model-e44515-kserve-6bfd4bd8dc-tjjks\" (UID: \"673f693f-2849-4212-b30b-22462f2681a3\") " pod="ai-tenant-e2e-shared-b-e44515/auth-test-model-e44515-kserve-6bfd4bd8dc-tjjks" Jul 22 21:49:22.160803 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:49:22.160759 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfqlf\" (UniqueName: \"kubernetes.io/projected/673f693f-2849-4212-b30b-22462f2681a3-kube-api-access-pfqlf\") pod \"auth-test-model-e44515-kserve-6bfd4bd8dc-tjjks\" (UID: \"673f693f-2849-4212-b30b-22462f2681a3\") " pod="ai-tenant-e2e-shared-b-e44515/auth-test-model-e44515-kserve-6bfd4bd8dc-tjjks" Jul 22 21:49:22.160837 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:49:22.160817 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/673f693f-2849-4212-b30b-22462f2681a3-dshm\") pod \"auth-test-model-e44515-kserve-6bfd4bd8dc-tjjks\" (UID: \"673f693f-2849-4212-b30b-22462f2681a3\") " pod="ai-tenant-e2e-shared-b-e44515/auth-test-model-e44515-kserve-6bfd4bd8dc-tjjks" Jul 22 21:49:22.160879 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:49:22.160866 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/673f693f-2849-4212-b30b-22462f2681a3-home\") pod \"auth-test-model-e44515-kserve-6bfd4bd8dc-tjjks\" (UID: \"673f693f-2849-4212-b30b-22462f2681a3\") " pod="ai-tenant-e2e-shared-b-e44515/auth-test-model-e44515-kserve-6bfd4bd8dc-tjjks" Jul 22 21:49:22.261547 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:49:22.261510 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/673f693f-2849-4212-b30b-22462f2681a3-tmp-dir\") pod \"auth-test-model-e44515-kserve-6bfd4bd8dc-tjjks\" (UID: \"673f693f-2849-4212-b30b-22462f2681a3\") " pod="ai-tenant-e2e-shared-b-e44515/auth-test-model-e44515-kserve-6bfd4bd8dc-tjjks" Jul 22 21:49:22.261725 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:49:22.261565 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfqlf\" (UniqueName: \"kubernetes.io/projected/673f693f-2849-4212-b30b-22462f2681a3-kube-api-access-pfqlf\") pod \"auth-test-model-e44515-kserve-6bfd4bd8dc-tjjks\" (UID: \"673f693f-2849-4212-b30b-22462f2681a3\") " pod="ai-tenant-e2e-shared-b-e44515/auth-test-model-e44515-kserve-6bfd4bd8dc-tjjks" Jul 22 21:49:22.261725 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:49:22.261591 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/673f693f-2849-4212-b30b-22462f2681a3-dshm\") pod \"auth-test-model-e44515-kserve-6bfd4bd8dc-tjjks\" (UID: \"673f693f-2849-4212-b30b-22462f2681a3\") " pod="ai-tenant-e2e-shared-b-e44515/auth-test-model-e44515-kserve-6bfd4bd8dc-tjjks" Jul 22 21:49:22.261725 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:49:22.261629 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/673f693f-2849-4212-b30b-22462f2681a3-home\") pod \"auth-test-model-e44515-kserve-6bfd4bd8dc-tjjks\" (UID: \"673f693f-2849-4212-b30b-22462f2681a3\") " pod="ai-tenant-e2e-shared-b-e44515/auth-test-model-e44515-kserve-6bfd4bd8dc-tjjks" Jul 22 21:49:22.261725 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:49:22.261686 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/673f693f-2849-4212-b30b-22462f2681a3-tls-certs\") pod \"auth-test-model-e44515-kserve-6bfd4bd8dc-tjjks\" (UID: \"673f693f-2849-4212-b30b-22462f2681a3\") " pod="ai-tenant-e2e-shared-b-e44515/auth-test-model-e44515-kserve-6bfd4bd8dc-tjjks" Jul 22 21:49:22.261943 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:49:22.261778 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/673f693f-2849-4212-b30b-22462f2681a3-model-cache\") pod \"auth-test-model-e44515-kserve-6bfd4bd8dc-tjjks\" (UID: \"673f693f-2849-4212-b30b-22462f2681a3\") " pod="ai-tenant-e2e-shared-b-e44515/auth-test-model-e44515-kserve-6bfd4bd8dc-tjjks" Jul 22 21:49:22.261943 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:49:22.261915 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/673f693f-2849-4212-b30b-22462f2681a3-tmp-dir\") pod \"auth-test-model-e44515-kserve-6bfd4bd8dc-tjjks\" (UID: \"673f693f-2849-4212-b30b-22462f2681a3\") " pod="ai-tenant-e2e-shared-b-e44515/auth-test-model-e44515-kserve-6bfd4bd8dc-tjjks" Jul 22 21:49:22.262035 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:49:22.261958 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/673f693f-2849-4212-b30b-22462f2681a3-home\") pod \"auth-test-model-e44515-kserve-6bfd4bd8dc-tjjks\" (UID: \"673f693f-2849-4212-b30b-22462f2681a3\") " pod="ai-tenant-e2e-shared-b-e44515/auth-test-model-e44515-kserve-6bfd4bd8dc-tjjks" Jul 22 21:49:22.262069 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:49:22.262036 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/673f693f-2849-4212-b30b-22462f2681a3-model-cache\") pod \"auth-test-model-e44515-kserve-6bfd4bd8dc-tjjks\" (UID: \"673f693f-2849-4212-b30b-22462f2681a3\") " pod="ai-tenant-e2e-shared-b-e44515/auth-test-model-e44515-kserve-6bfd4bd8dc-tjjks" Jul 22 21:49:22.264020 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:49:22.263996 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/673f693f-2849-4212-b30b-22462f2681a3-dshm\") pod \"auth-test-model-e44515-kserve-6bfd4bd8dc-tjjks\" (UID: \"673f693f-2849-4212-b30b-22462f2681a3\") " pod="ai-tenant-e2e-shared-b-e44515/auth-test-model-e44515-kserve-6bfd4bd8dc-tjjks" Jul 22 21:49:22.264221 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:49:22.264206 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/673f693f-2849-4212-b30b-22462f2681a3-tls-certs\") pod \"auth-test-model-e44515-kserve-6bfd4bd8dc-tjjks\" (UID: \"673f693f-2849-4212-b30b-22462f2681a3\") " pod="ai-tenant-e2e-shared-b-e44515/auth-test-model-e44515-kserve-6bfd4bd8dc-tjjks" Jul 22 21:49:22.276360 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:49:22.276339 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfqlf\" (UniqueName: \"kubernetes.io/projected/673f693f-2849-4212-b30b-22462f2681a3-kube-api-access-pfqlf\") pod \"auth-test-model-e44515-kserve-6bfd4bd8dc-tjjks\" (UID: \"673f693f-2849-4212-b30b-22462f2681a3\") " pod="ai-tenant-e2e-shared-b-e44515/auth-test-model-e44515-kserve-6bfd4bd8dc-tjjks" Jul 22 21:49:22.421364 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:49:22.421298 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e44515/auth-test-model-e44515-kserve-6bfd4bd8dc-tjjks" Jul 22 21:49:22.556063 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:49:22.556042 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-e44515/auth-test-model-e44515-kserve-6bfd4bd8dc-tjjks"] Jul 22 21:49:22.557233 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:49:22.557209 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod673f693f_2849_4212_b30b_22462f2681a3.slice/crio-42085aa3228a3e6a937b056916afe68e06bbda3b5d02e83d76fecc2b092beb3b WatchSource:0}: Error finding container 42085aa3228a3e6a937b056916afe68e06bbda3b5d02e83d76fecc2b092beb3b: Status 404 returned error can't find the container with id 42085aa3228a3e6a937b056916afe68e06bbda3b5d02e83d76fecc2b092beb3b Jul 22 21:49:23.536203 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:49:23.536151 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e44515/auth-test-model-e44515-kserve-6bfd4bd8dc-tjjks" event={"ID":"673f693f-2849-4212-b30b-22462f2681a3","Type":"ContainerStarted","Data":"5f2b0c24c93d2d20df46f105489ba8198531557a1c1509b77a8ef6750dc4400d"} Jul 22 21:49:23.536203 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:49:23.536208 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e44515/auth-test-model-e44515-kserve-6bfd4bd8dc-tjjks" event={"ID":"673f693f-2849-4212-b30b-22462f2681a3","Type":"ContainerStarted","Data":"42085aa3228a3e6a937b056916afe68e06bbda3b5d02e83d76fecc2b092beb3b"} Jul 22 21:49:23.555330 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:49:23.555290 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-e44515/auth-test-model-e44515-kserve-6bfd4bd8dc-tjjks" podStartSLOduration=1.55527612 podStartE2EDuration="1.55527612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 21:49:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 21:49:23.554739665 +0000 UTC m=+2618.669132009" watchObservedRunningTime="2026-07-22 21:49:23.55527612 +0000 UTC m=+2618.669668463" Jul 22 21:49:25.929110 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:49:25.929077 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:49:32.422225 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:49:32.422186 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-e44515/auth-test-model-e44515-kserve-6bfd4bd8dc-tjjks" Jul 22 21:49:32.422774 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:49:32.422238 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-e44515/auth-test-model-e44515-kserve-6bfd4bd8dc-tjjks" Jul 22 21:49:32.434157 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:49:32.434134 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-e44515/auth-test-model-e44515-kserve-6bfd4bd8dc-tjjks" Jul 22 21:49:32.596588 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:49:32.596564 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-e44515/auth-test-model-e44515-kserve-6bfd4bd8dc-tjjks" Jul 22 21:49:39.335279 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:49:39.335246 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:49:56.357500 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:49:56.357471 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:50:08.562599 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:50:08.562562 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:50:25.967382 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:50:25.967350 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:50:41.262009 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:50:41.261929 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:50:58.967103 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:50:58.967060 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:51:16.664672 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:51:16.664639 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:51:31.262384 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:51:31.262344 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:51:36.589680 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:51:36.589636 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-bddd55/auth-test-model-bddd55-kserve-689996889f-h6vpv"] Jul 22 21:51:37.441323 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:51:37.441289 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-e44515/auth-test-model-e44515-kserve-6bfd4bd8dc-tjjks"] Jul 22 21:51:46.573719 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:51:46.573690 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:51:51.609984 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:51:51.609907 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-bddd55/auth-test-model-bddd55-kserve-689996889f-h6vpv" podUID="ad7d0701-ebf9-41a0-a385-835326d6ed7e" containerName="main" containerID="cri-o://db9ca469ba9f591287132e30155a23f1b10d7507450c03caee8ca5ac11423604" gracePeriod=45 Jul 22 21:51:51.867325 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:51:51.867271 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-bddd55/auth-test-model-bddd55-kserve-689996889f-h6vpv" Jul 22 21:51:51.948852 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:51:51.948828 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad7d0701-ebf9-41a0-a385-835326d6ed7e-home\") pod \"ad7d0701-ebf9-41a0-a385-835326d6ed7e\" (UID: \"ad7d0701-ebf9-41a0-a385-835326d6ed7e\") " Jul 22 21:51:51.948973 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:51:51.948870 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad7d0701-ebf9-41a0-a385-835326d6ed7e-model-cache\") pod \"ad7d0701-ebf9-41a0-a385-835326d6ed7e\" (UID: \"ad7d0701-ebf9-41a0-a385-835326d6ed7e\") " Jul 22 21:51:51.948973 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:51:51.948912 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad7d0701-ebf9-41a0-a385-835326d6ed7e-dshm\") pod \"ad7d0701-ebf9-41a0-a385-835326d6ed7e\" (UID: \"ad7d0701-ebf9-41a0-a385-835326d6ed7e\") " Jul 22 21:51:51.949043 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:51:51.948971 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad7d0701-ebf9-41a0-a385-835326d6ed7e-tmp-dir\") pod \"ad7d0701-ebf9-41a0-a385-835326d6ed7e\" (UID: \"ad7d0701-ebf9-41a0-a385-835326d6ed7e\") " Jul 22 21:51:51.949043 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:51:51.949016 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad7d0701-ebf9-41a0-a385-835326d6ed7e-tls-certs\") pod \"ad7d0701-ebf9-41a0-a385-835326d6ed7e\" (UID: \"ad7d0701-ebf9-41a0-a385-835326d6ed7e\") " Jul 22 21:51:51.949043 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:51:51.949040 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xxw7z\" (UniqueName: \"kubernetes.io/projected/ad7d0701-ebf9-41a0-a385-835326d6ed7e-kube-api-access-xxw7z\") pod \"ad7d0701-ebf9-41a0-a385-835326d6ed7e\" (UID: \"ad7d0701-ebf9-41a0-a385-835326d6ed7e\") " Jul 22 21:51:51.949203 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:51:51.949106 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad7d0701-ebf9-41a0-a385-835326d6ed7e-home" (OuterVolumeSpecName: "home") pod "ad7d0701-ebf9-41a0-a385-835326d6ed7e" (UID: "ad7d0701-ebf9-41a0-a385-835326d6ed7e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 21:51:51.949203 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:51:51.949117 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad7d0701-ebf9-41a0-a385-835326d6ed7e-model-cache" (OuterVolumeSpecName: "model-cache") pod "ad7d0701-ebf9-41a0-a385-835326d6ed7e" (UID: "ad7d0701-ebf9-41a0-a385-835326d6ed7e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 21:51:51.949331 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:51:51.949308 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad7d0701-ebf9-41a0-a385-835326d6ed7e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ad7d0701-ebf9-41a0-a385-835326d6ed7e" (UID: "ad7d0701-ebf9-41a0-a385-835326d6ed7e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 21:51:51.949401 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:51:51.949330 2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad7d0701-ebf9-41a0-a385-835326d6ed7e-home\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:51:51.949401 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:51:51.949343 2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad7d0701-ebf9-41a0-a385-835326d6ed7e-model-cache\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:51:51.951136 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:51:51.951113 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad7d0701-ebf9-41a0-a385-835326d6ed7e-dshm" (OuterVolumeSpecName: "dshm") pod "ad7d0701-ebf9-41a0-a385-835326d6ed7e" (UID: "ad7d0701-ebf9-41a0-a385-835326d6ed7e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 21:51:51.951283 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:51:51.951231 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad7d0701-ebf9-41a0-a385-835326d6ed7e-kube-api-access-xxw7z" (OuterVolumeSpecName: "kube-api-access-xxw7z") pod "ad7d0701-ebf9-41a0-a385-835326d6ed7e" (UID: "ad7d0701-ebf9-41a0-a385-835326d6ed7e"). InnerVolumeSpecName "kube-api-access-xxw7z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 21:51:51.951422 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:51:51.951406 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad7d0701-ebf9-41a0-a385-835326d6ed7e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ad7d0701-ebf9-41a0-a385-835326d6ed7e" (UID: "ad7d0701-ebf9-41a0-a385-835326d6ed7e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 21:51:52.050148 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:51:52.050124 2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad7d0701-ebf9-41a0-a385-835326d6ed7e-dshm\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:51:52.050148 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:51:52.050145 2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad7d0701-ebf9-41a0-a385-835326d6ed7e-tmp-dir\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:51:52.050306 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:51:52.050153 2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad7d0701-ebf9-41a0-a385-835326d6ed7e-tls-certs\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:51:52.050306 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:51:52.050174 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxw7z\" (UniqueName: \"kubernetes.io/projected/ad7d0701-ebf9-41a0-a385-835326d6ed7e-kube-api-access-xxw7z\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:51:52.319461 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:51:52.319432 2572 generic.go:356] "Generic (PLEG): container finished" podID="ad7d0701-ebf9-41a0-a385-835326d6ed7e" containerID="db9ca469ba9f591287132e30155a23f1b10d7507450c03caee8ca5ac11423604" exitCode=0 Jul 22 21:51:52.319612 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:51:52.319475 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-bddd55/auth-test-model-bddd55-kserve-689996889f-h6vpv" Jul 22 21:51:52.319612 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:51:52.319514 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-bddd55/auth-test-model-bddd55-kserve-689996889f-h6vpv" event={"ID":"ad7d0701-ebf9-41a0-a385-835326d6ed7e","Type":"ContainerDied","Data":"db9ca469ba9f591287132e30155a23f1b10d7507450c03caee8ca5ac11423604"} Jul 22 21:51:52.319612 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:51:52.319544 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-bddd55/auth-test-model-bddd55-kserve-689996889f-h6vpv" event={"ID":"ad7d0701-ebf9-41a0-a385-835326d6ed7e","Type":"ContainerDied","Data":"ee116b7717d1171bc61ecba40099ad92128ddcc249a0cc620b8760c66daf7e9d"} Jul 22 21:51:52.319612 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:51:52.319561 2572 scope.go:117] "RemoveContainer" containerID="db9ca469ba9f591287132e30155a23f1b10d7507450c03caee8ca5ac11423604" Jul 22 21:51:52.329317 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:51:52.329272 2572 scope.go:117] "RemoveContainer" containerID="db9ca469ba9f591287132e30155a23f1b10d7507450c03caee8ca5ac11423604" Jul 22 21:51:52.329520 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:51:52.329501 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db9ca469ba9f591287132e30155a23f1b10d7507450c03caee8ca5ac11423604\": container with ID starting with db9ca469ba9f591287132e30155a23f1b10d7507450c03caee8ca5ac11423604 not found: ID does not exist" containerID="db9ca469ba9f591287132e30155a23f1b10d7507450c03caee8ca5ac11423604" Jul 22 21:51:52.329582 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:51:52.329529 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db9ca469ba9f591287132e30155a23f1b10d7507450c03caee8ca5ac11423604"} err="failed to get container status \"db9ca469ba9f591287132e30155a23f1b10d7507450c03caee8ca5ac11423604\": rpc error: code = NotFound desc = could not find container \"db9ca469ba9f591287132e30155a23f1b10d7507450c03caee8ca5ac11423604\": container with ID starting with db9ca469ba9f591287132e30155a23f1b10d7507450c03caee8ca5ac11423604 not found: ID does not exist" Jul 22 21:51:52.344511 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:51:52.344488 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-bddd55/auth-test-model-bddd55-kserve-689996889f-h6vpv"] Jul 22 21:51:52.347803 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:51:52.347785 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-bddd55/auth-test-model-bddd55-kserve-689996889f-h6vpv"] Jul 22 21:51:52.461502 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:51:52.461472 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-e44515/auth-test-model-e44515-kserve-6bfd4bd8dc-tjjks" podUID="673f693f-2849-4212-b30b-22462f2681a3" containerName="main" containerID="cri-o://5f2b0c24c93d2d20df46f105489ba8198531557a1c1509b77a8ef6750dc4400d" gracePeriod=45 Jul 22 21:51:52.587138 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:51:52.587105 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-e44515/auth-test-model-e44515-kserve-6bfd4bd8dc-tjjks" podUID="673f693f-2849-4212-b30b-22462f2681a3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": dial tcp 10.134.0.54:8000: connect: connection refused" Jul 22 21:51:52.713174 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:51:52.713135 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e44515/auth-test-model-e44515-kserve-6bfd4bd8dc-tjjks" Jul 22 21:51:52.755081 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:51:52.755055 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/673f693f-2849-4212-b30b-22462f2681a3-model-cache\") pod \"673f693f-2849-4212-b30b-22462f2681a3\" (UID: \"673f693f-2849-4212-b30b-22462f2681a3\") " Jul 22 21:51:52.755221 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:51:52.755120 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pfqlf\" (UniqueName: \"kubernetes.io/projected/673f693f-2849-4212-b30b-22462f2681a3-kube-api-access-pfqlf\") pod \"673f693f-2849-4212-b30b-22462f2681a3\" (UID: \"673f693f-2849-4212-b30b-22462f2681a3\") " Jul 22 21:51:52.755221 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:51:52.755214 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/673f693f-2849-4212-b30b-22462f2681a3-tls-certs\") pod \"673f693f-2849-4212-b30b-22462f2681a3\" (UID: \"673f693f-2849-4212-b30b-22462f2681a3\") " Jul 22 21:51:52.755325 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:51:52.755240 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/673f693f-2849-4212-b30b-22462f2681a3-tmp-dir\") pod \"673f693f-2849-4212-b30b-22462f2681a3\" (UID: \"673f693f-2849-4212-b30b-22462f2681a3\") " Jul 22 21:51:52.755325 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:51:52.755285 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/673f693f-2849-4212-b30b-22462f2681a3-dshm\") pod \"673f693f-2849-4212-b30b-22462f2681a3\" (UID: \"673f693f-2849-4212-b30b-22462f2681a3\") " Jul 22 21:51:52.755325 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:51:52.755300 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/673f693f-2849-4212-b30b-22462f2681a3-model-cache" (OuterVolumeSpecName: "model-cache") pod "673f693f-2849-4212-b30b-22462f2681a3" (UID: "673f693f-2849-4212-b30b-22462f2681a3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 21:51:52.755484 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:51:52.755331 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/673f693f-2849-4212-b30b-22462f2681a3-home\") pod \"673f693f-2849-4212-b30b-22462f2681a3\" (UID: \"673f693f-2849-4212-b30b-22462f2681a3\") " Jul 22 21:51:52.755541 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:51:52.755503 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/673f693f-2849-4212-b30b-22462f2681a3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "673f693f-2849-4212-b30b-22462f2681a3" (UID: "673f693f-2849-4212-b30b-22462f2681a3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 21:51:52.755632 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:51:52.755611 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/673f693f-2849-4212-b30b-22462f2681a3-home" (OuterVolumeSpecName: "home") pod "673f693f-2849-4212-b30b-22462f2681a3" (UID: "673f693f-2849-4212-b30b-22462f2681a3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 21:51:52.755718 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:51:52.755696 2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/673f693f-2849-4212-b30b-22462f2681a3-home\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:51:52.755832 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:51:52.755720 2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/673f693f-2849-4212-b30b-22462f2681a3-model-cache\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:51:52.755832 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:51:52.755735 2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/673f693f-2849-4212-b30b-22462f2681a3-tmp-dir\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:51:52.757437 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:51:52.757417 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/673f693f-2849-4212-b30b-22462f2681a3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "673f693f-2849-4212-b30b-22462f2681a3" (UID: "673f693f-2849-4212-b30b-22462f2681a3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 21:51:52.757530 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:51:52.757455 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/673f693f-2849-4212-b30b-22462f2681a3-dshm" (OuterVolumeSpecName: "dshm") pod "673f693f-2849-4212-b30b-22462f2681a3" (UID: "673f693f-2849-4212-b30b-22462f2681a3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 21:51:52.757685 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:51:52.757670 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/673f693f-2849-4212-b30b-22462f2681a3-kube-api-access-pfqlf" (OuterVolumeSpecName: "kube-api-access-pfqlf") pod "673f693f-2849-4212-b30b-22462f2681a3" (UID: "673f693f-2849-4212-b30b-22462f2681a3"). InnerVolumeSpecName "kube-api-access-pfqlf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 21:51:52.856977 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:51:52.856928 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfqlf\" (UniqueName: \"kubernetes.io/projected/673f693f-2849-4212-b30b-22462f2681a3-kube-api-access-pfqlf\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:51:52.856977 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:51:52.856951 2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/673f693f-2849-4212-b30b-22462f2681a3-tls-certs\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:51:52.856977 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:51:52.856961 2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/673f693f-2849-4212-b30b-22462f2681a3-dshm\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:51:53.326514 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:51:53.326483 2572 generic.go:356] "Generic (PLEG): container finished" podID="673f693f-2849-4212-b30b-22462f2681a3" containerID="5f2b0c24c93d2d20df46f105489ba8198531557a1c1509b77a8ef6750dc4400d" exitCode=0 Jul 22 21:51:53.326661 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:51:53.326531 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e44515/auth-test-model-e44515-kserve-6bfd4bd8dc-tjjks" Jul 22 21:51:53.326661 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:51:53.326565 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e44515/auth-test-model-e44515-kserve-6bfd4bd8dc-tjjks" event={"ID":"673f693f-2849-4212-b30b-22462f2681a3","Type":"ContainerDied","Data":"5f2b0c24c93d2d20df46f105489ba8198531557a1c1509b77a8ef6750dc4400d"} Jul 22 21:51:53.326661 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:51:53.326605 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e44515/auth-test-model-e44515-kserve-6bfd4bd8dc-tjjks" event={"ID":"673f693f-2849-4212-b30b-22462f2681a3","Type":"ContainerDied","Data":"42085aa3228a3e6a937b056916afe68e06bbda3b5d02e83d76fecc2b092beb3b"} Jul 22 21:51:53.326661 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:51:53.326622 2572 scope.go:117] "RemoveContainer" containerID="5f2b0c24c93d2d20df46f105489ba8198531557a1c1509b77a8ef6750dc4400d" Jul 22 21:51:53.336056 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:51:53.336035 2572 scope.go:117] "RemoveContainer" containerID="5f2b0c24c93d2d20df46f105489ba8198531557a1c1509b77a8ef6750dc4400d" Jul 22 21:51:53.336357 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:51:53.336337 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f2b0c24c93d2d20df46f105489ba8198531557a1c1509b77a8ef6750dc4400d\": container with ID starting with 5f2b0c24c93d2d20df46f105489ba8198531557a1c1509b77a8ef6750dc4400d not found: ID does not exist" containerID="5f2b0c24c93d2d20df46f105489ba8198531557a1c1509b77a8ef6750dc4400d" Jul 22 21:51:53.336428 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:51:53.336365 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f2b0c24c93d2d20df46f105489ba8198531557a1c1509b77a8ef6750dc4400d"} err="failed to get container status \"5f2b0c24c93d2d20df46f105489ba8198531557a1c1509b77a8ef6750dc4400d\": rpc error: code = NotFound desc = could not find container \"5f2b0c24c93d2d20df46f105489ba8198531557a1c1509b77a8ef6750dc4400d\": container with ID starting with 5f2b0c24c93d2d20df46f105489ba8198531557a1c1509b77a8ef6750dc4400d not found: ID does not exist" Jul 22 21:51:53.350454 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:51:53.350434 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-e44515/auth-test-model-e44515-kserve-6bfd4bd8dc-tjjks"] Jul 22 21:51:53.357931 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:51:53.357903 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-e44515/auth-test-model-e44515-kserve-6bfd4bd8dc-tjjks"] Jul 22 21:51:53.383318 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:51:53.383296 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="673f693f-2849-4212-b30b-22462f2681a3" path="/var/lib/kubelet/pods/673f693f-2849-4212-b30b-22462f2681a3/volumes" Jul 22 21:51:53.383669 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:51:53.383656 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad7d0701-ebf9-41a0-a385-835326d6ed7e" path="/var/lib/kubelet/pods/ad7d0701-ebf9-41a0-a385-835326d6ed7e/volumes" Jul 22 21:51:56.577402 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:51:56.577372 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:52:27.186584 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:52:27.186549 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-e44515/sub-test-model-e44515-kserve-68bbf7bf5-wgj4q"] Jul 22 21:52:27.188986 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:52:27.186908 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="673f693f-2849-4212-b30b-22462f2681a3" containerName="main" Jul 22 21:52:27.188986 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:52:27.186919 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="673f693f-2849-4212-b30b-22462f2681a3" containerName="main" Jul 22 21:52:27.188986 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:52:27.186936 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad7d0701-ebf9-41a0-a385-835326d6ed7e" containerName="main" Jul 22 21:52:27.188986 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:52:27.186941 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad7d0701-ebf9-41a0-a385-835326d6ed7e" containerName="main" Jul 22 21:52:27.188986 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:52:27.187004 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad7d0701-ebf9-41a0-a385-835326d6ed7e" containerName="main" Jul 22 21:52:27.188986 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:52:27.187009 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="673f693f-2849-4212-b30b-22462f2681a3" containerName="main" Jul 22 21:52:27.189977 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:52:27.189962 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e44515/sub-test-model-e44515-kserve-68bbf7bf5-wgj4q" Jul 22 21:52:27.193883 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:52:27.193864 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-e44515\"/\"kube-root-ca.crt\"" Jul 22 21:52:27.194355 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:52:27.194341 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-e44515\"/\"default-dockercfg-hqsw9\"" Jul 22 21:52:27.194418 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:52:27.194340 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-e44515\"/\"sub-test-model-e44515-kserve-self-signed-certs\"" Jul 22 21:52:27.194418 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:52:27.194398 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-e44515\"/\"openshift-service-ca.crt\"" Jul 22 21:52:27.209766 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:52:27.209742 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-e44515/sub-test-model-e44515-kserve-68bbf7bf5-wgj4q"] Jul 22 21:52:27.338060 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:52:27.338026 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnbrc\" (UniqueName: \"kubernetes.io/projected/cb4d7874-5907-4fe3-a7a8-ce08121821da-kube-api-access-fnbrc\") pod \"sub-test-model-e44515-kserve-68bbf7bf5-wgj4q\" (UID: \"cb4d7874-5907-4fe3-a7a8-ce08121821da\") " pod="ai-tenant-e2e-shared-b-e44515/sub-test-model-e44515-kserve-68bbf7bf5-wgj4q" Jul 22 21:52:27.338246 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:52:27.338067 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb4d7874-5907-4fe3-a7a8-ce08121821da-dshm\") pod \"sub-test-model-e44515-kserve-68bbf7bf5-wgj4q\" (UID: \"cb4d7874-5907-4fe3-a7a8-ce08121821da\") " pod="ai-tenant-e2e-shared-b-e44515/sub-test-model-e44515-kserve-68bbf7bf5-wgj4q" Jul 22 21:52:27.338246 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:52:27.338125 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb4d7874-5907-4fe3-a7a8-ce08121821da-model-cache\") pod \"sub-test-model-e44515-kserve-68bbf7bf5-wgj4q\" (UID: \"cb4d7874-5907-4fe3-a7a8-ce08121821da\") " pod="ai-tenant-e2e-shared-b-e44515/sub-test-model-e44515-kserve-68bbf7bf5-wgj4q" Jul 22 21:52:27.338246 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:52:27.338143 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb4d7874-5907-4fe3-a7a8-ce08121821da-tmp-dir\") pod \"sub-test-model-e44515-kserve-68bbf7bf5-wgj4q\" (UID: \"cb4d7874-5907-4fe3-a7a8-ce08121821da\") " pod="ai-tenant-e2e-shared-b-e44515/sub-test-model-e44515-kserve-68bbf7bf5-wgj4q" Jul 22 21:52:27.338246 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:52:27.338174 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb4d7874-5907-4fe3-a7a8-ce08121821da-tls-certs\") pod \"sub-test-model-e44515-kserve-68bbf7bf5-wgj4q\" (UID: \"cb4d7874-5907-4fe3-a7a8-ce08121821da\") " pod="ai-tenant-e2e-shared-b-e44515/sub-test-model-e44515-kserve-68bbf7bf5-wgj4q" Jul 22 21:52:27.338246 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:52:27.338231 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb4d7874-5907-4fe3-a7a8-ce08121821da-home\") pod \"sub-test-model-e44515-kserve-68bbf7bf5-wgj4q\" (UID: \"cb4d7874-5907-4fe3-a7a8-ce08121821da\") " pod="ai-tenant-e2e-shared-b-e44515/sub-test-model-e44515-kserve-68bbf7bf5-wgj4q" Jul 22 21:52:27.438898 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:52:27.438811 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb4d7874-5907-4fe3-a7a8-ce08121821da-home\") pod \"sub-test-model-e44515-kserve-68bbf7bf5-wgj4q\" (UID: \"cb4d7874-5907-4fe3-a7a8-ce08121821da\") " pod="ai-tenant-e2e-shared-b-e44515/sub-test-model-e44515-kserve-68bbf7bf5-wgj4q" Jul 22 21:52:27.438898 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:52:27.438873 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnbrc\" (UniqueName: \"kubernetes.io/projected/cb4d7874-5907-4fe3-a7a8-ce08121821da-kube-api-access-fnbrc\") pod \"sub-test-model-e44515-kserve-68bbf7bf5-wgj4q\" (UID: \"cb4d7874-5907-4fe3-a7a8-ce08121821da\") " pod="ai-tenant-e2e-shared-b-e44515/sub-test-model-e44515-kserve-68bbf7bf5-wgj4q" Jul 22 21:52:27.439093 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:52:27.438907 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb4d7874-5907-4fe3-a7a8-ce08121821da-dshm\") pod \"sub-test-model-e44515-kserve-68bbf7bf5-wgj4q\" (UID: \"cb4d7874-5907-4fe3-a7a8-ce08121821da\") " pod="ai-tenant-e2e-shared-b-e44515/sub-test-model-e44515-kserve-68bbf7bf5-wgj4q" Jul 22 21:52:27.439093 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:52:27.438945 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb4d7874-5907-4fe3-a7a8-ce08121821da-model-cache\") pod \"sub-test-model-e44515-kserve-68bbf7bf5-wgj4q\" (UID: \"cb4d7874-5907-4fe3-a7a8-ce08121821da\") " pod="ai-tenant-e2e-shared-b-e44515/sub-test-model-e44515-kserve-68bbf7bf5-wgj4q" Jul 22 21:52:27.439093 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:52:27.438973 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb4d7874-5907-4fe3-a7a8-ce08121821da-tmp-dir\") pod \"sub-test-model-e44515-kserve-68bbf7bf5-wgj4q\" (UID: \"cb4d7874-5907-4fe3-a7a8-ce08121821da\") " pod="ai-tenant-e2e-shared-b-e44515/sub-test-model-e44515-kserve-68bbf7bf5-wgj4q" Jul 22 21:52:27.439093 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:52:27.438996 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb4d7874-5907-4fe3-a7a8-ce08121821da-tls-certs\") pod \"sub-test-model-e44515-kserve-68bbf7bf5-wgj4q\" (UID: \"cb4d7874-5907-4fe3-a7a8-ce08121821da\") " pod="ai-tenant-e2e-shared-b-e44515/sub-test-model-e44515-kserve-68bbf7bf5-wgj4q" Jul 22 21:52:27.439323 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:52:27.439288 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb4d7874-5907-4fe3-a7a8-ce08121821da-home\") pod \"sub-test-model-e44515-kserve-68bbf7bf5-wgj4q\" (UID: \"cb4d7874-5907-4fe3-a7a8-ce08121821da\") " pod="ai-tenant-e2e-shared-b-e44515/sub-test-model-e44515-kserve-68bbf7bf5-wgj4q" Jul 22 21:52:27.439446 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:52:27.439351 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb4d7874-5907-4fe3-a7a8-ce08121821da-model-cache\") pod \"sub-test-model-e44515-kserve-68bbf7bf5-wgj4q\" (UID: \"cb4d7874-5907-4fe3-a7a8-ce08121821da\") " pod="ai-tenant-e2e-shared-b-e44515/sub-test-model-e44515-kserve-68bbf7bf5-wgj4q" Jul 22 21:52:27.439446 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:52:27.439384 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb4d7874-5907-4fe3-a7a8-ce08121821da-tmp-dir\") pod \"sub-test-model-e44515-kserve-68bbf7bf5-wgj4q\" (UID: \"cb4d7874-5907-4fe3-a7a8-ce08121821da\") " pod="ai-tenant-e2e-shared-b-e44515/sub-test-model-e44515-kserve-68bbf7bf5-wgj4q" Jul 22 21:52:27.441422 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:52:27.441402 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb4d7874-5907-4fe3-a7a8-ce08121821da-tls-certs\") pod \"sub-test-model-e44515-kserve-68bbf7bf5-wgj4q\" (UID: \"cb4d7874-5907-4fe3-a7a8-ce08121821da\") " pod="ai-tenant-e2e-shared-b-e44515/sub-test-model-e44515-kserve-68bbf7bf5-wgj4q" Jul 22 21:52:27.441504 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:52:27.441404 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb4d7874-5907-4fe3-a7a8-ce08121821da-dshm\") pod \"sub-test-model-e44515-kserve-68bbf7bf5-wgj4q\" (UID: \"cb4d7874-5907-4fe3-a7a8-ce08121821da\") " pod="ai-tenant-e2e-shared-b-e44515/sub-test-model-e44515-kserve-68bbf7bf5-wgj4q" Jul 22 21:52:27.451587 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:52:27.451568 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnbrc\" (UniqueName: \"kubernetes.io/projected/cb4d7874-5907-4fe3-a7a8-ce08121821da-kube-api-access-fnbrc\") pod \"sub-test-model-e44515-kserve-68bbf7bf5-wgj4q\" (UID: \"cb4d7874-5907-4fe3-a7a8-ce08121821da\") " pod="ai-tenant-e2e-shared-b-e44515/sub-test-model-e44515-kserve-68bbf7bf5-wgj4q" Jul 22 21:52:27.498847 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:52:27.498824 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e44515/sub-test-model-e44515-kserve-68bbf7bf5-wgj4q" Jul 22 21:52:27.635701 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:52:27.635676 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-e44515/sub-test-model-e44515-kserve-68bbf7bf5-wgj4q"] Jul 22 21:52:27.636980 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:52:27.636957 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcb4d7874_5907_4fe3_a7a8_ce08121821da.slice/crio-a3080949d04a9e2131e0ca5fd91bed97d44f77ad2b4c78d8479a1ab89d266e6d WatchSource:0}: Error finding container a3080949d04a9e2131e0ca5fd91bed97d44f77ad2b4c78d8479a1ab89d266e6d: Status 404 returned error can't find the container with id a3080949d04a9e2131e0ca5fd91bed97d44f77ad2b4c78d8479a1ab89d266e6d Jul 22 21:52:28.501110 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:52:28.501072 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e44515/sub-test-model-e44515-kserve-68bbf7bf5-wgj4q" event={"ID":"cb4d7874-5907-4fe3-a7a8-ce08121821da","Type":"ContainerStarted","Data":"e4f51fd32b06f02eaf09b5f3a5cac7c77c3d592d8c85cb7f28502bab236b1b86"} Jul 22 21:52:28.501494 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:52:28.501118 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e44515/sub-test-model-e44515-kserve-68bbf7bf5-wgj4q" event={"ID":"cb4d7874-5907-4fe3-a7a8-ce08121821da","Type":"ContainerStarted","Data":"a3080949d04a9e2131e0ca5fd91bed97d44f77ad2b4c78d8479a1ab89d266e6d"} Jul 22 21:52:28.534325 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:52:28.534277 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-e44515/sub-test-model-e44515-kserve-68bbf7bf5-wgj4q" podStartSLOduration=1.534262069 podStartE2EDuration="1.534262069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 21:52:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 21:52:28.533501615 +0000 UTC m=+2803.647893960" watchObservedRunningTime="2026-07-22 21:52:28.534262069 +0000 UTC m=+2803.648654442" Jul 22 21:52:30.700480 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:52:30.700438 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:52:37.499730 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:52:37.499697 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-e44515/sub-test-model-e44515-kserve-68bbf7bf5-wgj4q" Jul 22 21:52:37.500205 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:52:37.499740 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-e44515/sub-test-model-e44515-kserve-68bbf7bf5-wgj4q" Jul 22 21:52:37.511576 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:52:37.511553 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-e44515/sub-test-model-e44515-kserve-68bbf7bf5-wgj4q" Jul 22 21:52:37.557213 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:52:37.557194 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-e44515/sub-test-model-e44515-kserve-68bbf7bf5-wgj4q" Jul 22 21:52:44.006002 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:52:44.005972 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:53:02.337024 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:53:02.336997 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-e44515/sub-test-model-e44515-kserve-68bbf7bf5-wgj4q"] Jul 22 21:53:05.786802 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:53:05.786757 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:53:17.361867 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:53:17.361804 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-e44515/sub-test-model-e44515-kserve-68bbf7bf5-wgj4q" podUID="cb4d7874-5907-4fe3-a7a8-ce08121821da" containerName="main" containerID="cri-o://e4f51fd32b06f02eaf09b5f3a5cac7c77c3d592d8c85cb7f28502bab236b1b86" gracePeriod=45 Jul 22 21:53:17.625576 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:53:17.625518 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e44515/sub-test-model-e44515-kserve-68bbf7bf5-wgj4q" Jul 22 21:53:17.758255 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:53:17.758224 2572 generic.go:356] "Generic (PLEG): container finished" podID="cb4d7874-5907-4fe3-a7a8-ce08121821da" containerID="e4f51fd32b06f02eaf09b5f3a5cac7c77c3d592d8c85cb7f28502bab236b1b86" exitCode=0 Jul 22 21:53:17.758446 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:53:17.758265 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e44515/sub-test-model-e44515-kserve-68bbf7bf5-wgj4q" Jul 22 21:53:17.758446 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:53:17.758306 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e44515/sub-test-model-e44515-kserve-68bbf7bf5-wgj4q" event={"ID":"cb4d7874-5907-4fe3-a7a8-ce08121821da","Type":"ContainerDied","Data":"e4f51fd32b06f02eaf09b5f3a5cac7c77c3d592d8c85cb7f28502bab236b1b86"} Jul 22 21:53:17.758446 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:53:17.758348 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e44515/sub-test-model-e44515-kserve-68bbf7bf5-wgj4q" event={"ID":"cb4d7874-5907-4fe3-a7a8-ce08121821da","Type":"ContainerDied","Data":"a3080949d04a9e2131e0ca5fd91bed97d44f77ad2b4c78d8479a1ab89d266e6d"} Jul 22 21:53:17.758446 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:53:17.758364 2572 scope.go:117] "RemoveContainer" containerID="e4f51fd32b06f02eaf09b5f3a5cac7c77c3d592d8c85cb7f28502bab236b1b86" Jul 22 21:53:17.767036 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:53:17.767012 2572 scope.go:117] "RemoveContainer" containerID="e4f51fd32b06f02eaf09b5f3a5cac7c77c3d592d8c85cb7f28502bab236b1b86" Jul 22 21:53:17.767315 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:53:17.767297 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4f51fd32b06f02eaf09b5f3a5cac7c77c3d592d8c85cb7f28502bab236b1b86\": container with ID starting with e4f51fd32b06f02eaf09b5f3a5cac7c77c3d592d8c85cb7f28502bab236b1b86 not found: ID does not exist" containerID="e4f51fd32b06f02eaf09b5f3a5cac7c77c3d592d8c85cb7f28502bab236b1b86" Jul 22 21:53:17.767378 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:53:17.767322 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4f51fd32b06f02eaf09b5f3a5cac7c77c3d592d8c85cb7f28502bab236b1b86"} err="failed to get container status \"e4f51fd32b06f02eaf09b5f3a5cac7c77c3d592d8c85cb7f28502bab236b1b86\": rpc error: code = NotFound desc = could not find container \"e4f51fd32b06f02eaf09b5f3a5cac7c77c3d592d8c85cb7f28502bab236b1b86\": container with ID starting with e4f51fd32b06f02eaf09b5f3a5cac7c77c3d592d8c85cb7f28502bab236b1b86 not found: ID does not exist" Jul 22 21:53:17.798425 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:53:17.798402 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb4d7874-5907-4fe3-a7a8-ce08121821da-model-cache\") pod \"cb4d7874-5907-4fe3-a7a8-ce08121821da\" (UID: \"cb4d7874-5907-4fe3-a7a8-ce08121821da\") " Jul 22 21:53:17.798542 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:53:17.798439 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb4d7874-5907-4fe3-a7a8-ce08121821da-tmp-dir\") pod \"cb4d7874-5907-4fe3-a7a8-ce08121821da\" (UID: \"cb4d7874-5907-4fe3-a7a8-ce08121821da\") " Jul 22 21:53:17.798542 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:53:17.798473 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fnbrc\" (UniqueName: \"kubernetes.io/projected/cb4d7874-5907-4fe3-a7a8-ce08121821da-kube-api-access-fnbrc\") pod \"cb4d7874-5907-4fe3-a7a8-ce08121821da\" (UID: \"cb4d7874-5907-4fe3-a7a8-ce08121821da\") " Jul 22 21:53:17.798542 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:53:17.798513 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb4d7874-5907-4fe3-a7a8-ce08121821da-tls-certs\") pod \"cb4d7874-5907-4fe3-a7a8-ce08121821da\" (UID: \"cb4d7874-5907-4fe3-a7a8-ce08121821da\") " Jul 22 21:53:17.798702 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:53:17.798550 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb4d7874-5907-4fe3-a7a8-ce08121821da-home\") pod \"cb4d7874-5907-4fe3-a7a8-ce08121821da\" (UID: \"cb4d7874-5907-4fe3-a7a8-ce08121821da\") " Jul 22 21:53:17.798702 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:53:17.798643 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb4d7874-5907-4fe3-a7a8-ce08121821da-dshm\") pod \"cb4d7874-5907-4fe3-a7a8-ce08121821da\" (UID: \"cb4d7874-5907-4fe3-a7a8-ce08121821da\") " Jul 22 21:53:17.798796 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:53:17.798698 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb4d7874-5907-4fe3-a7a8-ce08121821da-model-cache" (OuterVolumeSpecName: "model-cache") pod "cb4d7874-5907-4fe3-a7a8-ce08121821da" (UID: "cb4d7874-5907-4fe3-a7a8-ce08121821da"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 21:53:17.798796 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:53:17.798717 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb4d7874-5907-4fe3-a7a8-ce08121821da-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cb4d7874-5907-4fe3-a7a8-ce08121821da" (UID: "cb4d7874-5907-4fe3-a7a8-ce08121821da"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 21:53:17.798895 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:53:17.798873 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb4d7874-5907-4fe3-a7a8-ce08121821da-home" (OuterVolumeSpecName: "home") pod "cb4d7874-5907-4fe3-a7a8-ce08121821da" (UID: "cb4d7874-5907-4fe3-a7a8-ce08121821da"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 21:53:17.798992 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:53:17.798957 2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb4d7874-5907-4fe3-a7a8-ce08121821da-home\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:53:17.798992 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:53:17.798973 2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb4d7874-5907-4fe3-a7a8-ce08121821da-model-cache\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:53:17.798992 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:53:17.798986 2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb4d7874-5907-4fe3-a7a8-ce08121821da-tmp-dir\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:53:17.800716 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:53:17.800691 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb4d7874-5907-4fe3-a7a8-ce08121821da-dshm" (OuterVolumeSpecName: "dshm") pod "cb4d7874-5907-4fe3-a7a8-ce08121821da" (UID: "cb4d7874-5907-4fe3-a7a8-ce08121821da"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 21:53:17.800817 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:53:17.800774 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb4d7874-5907-4fe3-a7a8-ce08121821da-kube-api-access-fnbrc" (OuterVolumeSpecName: "kube-api-access-fnbrc") pod "cb4d7874-5907-4fe3-a7a8-ce08121821da" (UID: "cb4d7874-5907-4fe3-a7a8-ce08121821da"). InnerVolumeSpecName "kube-api-access-fnbrc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 21:53:17.800817 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:53:17.800780 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb4d7874-5907-4fe3-a7a8-ce08121821da-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cb4d7874-5907-4fe3-a7a8-ce08121821da" (UID: "cb4d7874-5907-4fe3-a7a8-ce08121821da"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 21:53:17.900085 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:53:17.900017 2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb4d7874-5907-4fe3-a7a8-ce08121821da-dshm\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:53:17.900085 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:53:17.900040 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnbrc\" (UniqueName: \"kubernetes.io/projected/cb4d7874-5907-4fe3-a7a8-ce08121821da-kube-api-access-fnbrc\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:53:17.900085 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:53:17.900051 2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb4d7874-5907-4fe3-a7a8-ce08121821da-tls-certs\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:53:18.085726 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:53:18.085694 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-e44515/sub-test-model-e44515-kserve-68bbf7bf5-wgj4q"] Jul 22 21:53:18.089499 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:53:18.089475 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-e44515/sub-test-model-e44515-kserve-68bbf7bf5-wgj4q"] Jul 22 21:53:18.854146 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:53:18.854099 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:53:19.383048 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:53:19.383014 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb4d7874-5907-4fe3-a7a8-ce08121821da" path="/var/lib/kubelet/pods/cb4d7874-5907-4fe3-a7a8-ce08121821da/volumes" Jul 22 21:53:22.547411 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:53:22.547372 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-e44515/sub-test-model-e44515-kserve-68bbf7bf5-wgj4q" podUID="cb4d7874-5907-4fe3-a7a8-ce08121821da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Jul 22 21:53:36.452521 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:53:36.452448 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-e44515/rate-test-model-e44515-kserve-5745bf64c9-8z8xm"] Jul 22 21:53:36.452949 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:53:36.452811 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb4d7874-5907-4fe3-a7a8-ce08121821da" containerName="main" Jul 22 21:53:36.452949 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:53:36.452821 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb4d7874-5907-4fe3-a7a8-ce08121821da" containerName="main" Jul 22 21:53:36.452949 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:53:36.452880 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb4d7874-5907-4fe3-a7a8-ce08121821da" containerName="main" Jul 22 21:53:36.455696 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:53:36.455679 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e44515/rate-test-model-e44515-kserve-5745bf64c9-8z8xm" Jul 22 21:53:36.458462 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:53:36.458438 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-e44515\"/\"openshift-service-ca.crt\"" Jul 22 21:53:36.459481 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:53:36.459461 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-e44515\"/\"kube-root-ca.crt\"" Jul 22 21:53:36.459598 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:53:36.459525 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-e44515\"/\"default-dockercfg-hqsw9\"" Jul 22 21:53:36.459598 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:53:36.459565 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-e44515\"/\"rate-test-model-e44515-kserve-self-signed-certs\"" Jul 22 21:53:36.465370 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:53:36.465349 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-e44515/rate-test-model-e44515-kserve-5745bf64c9-8z8xm"] Jul 22 21:53:36.550241 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:53:36.550210 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5d34d64-78e3-4285-8568-175030180713-dshm\") pod \"rate-test-model-e44515-kserve-5745bf64c9-8z8xm\" (UID: \"e5d34d64-78e3-4285-8568-175030180713\") " pod="ai-tenant-e2e-shared-b-e44515/rate-test-model-e44515-kserve-5745bf64c9-8z8xm" Jul 22 21:53:36.550362 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:53:36.550252 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xf7w\" (UniqueName: \"kubernetes.io/projected/e5d34d64-78e3-4285-8568-175030180713-kube-api-access-5xf7w\") pod \"rate-test-model-e44515-kserve-5745bf64c9-8z8xm\" (UID: \"e5d34d64-78e3-4285-8568-175030180713\") " pod="ai-tenant-e2e-shared-b-e44515/rate-test-model-e44515-kserve-5745bf64c9-8z8xm" Jul 22 21:53:36.550362 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:53:36.550327 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5d34d64-78e3-4285-8568-175030180713-model-cache\") pod \"rate-test-model-e44515-kserve-5745bf64c9-8z8xm\" (UID: \"e5d34d64-78e3-4285-8568-175030180713\") " pod="ai-tenant-e2e-shared-b-e44515/rate-test-model-e44515-kserve-5745bf64c9-8z8xm" Jul 22 21:53:36.550481 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:53:36.550390 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5d34d64-78e3-4285-8568-175030180713-tmp-dir\") pod \"rate-test-model-e44515-kserve-5745bf64c9-8z8xm\" (UID: \"e5d34d64-78e3-4285-8568-175030180713\") " pod="ai-tenant-e2e-shared-b-e44515/rate-test-model-e44515-kserve-5745bf64c9-8z8xm" Jul 22 21:53:36.550481 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:53:36.550445 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5d34d64-78e3-4285-8568-175030180713-home\") pod \"rate-test-model-e44515-kserve-5745bf64c9-8z8xm\" (UID: \"e5d34d64-78e3-4285-8568-175030180713\") " pod="ai-tenant-e2e-shared-b-e44515/rate-test-model-e44515-kserve-5745bf64c9-8z8xm" Jul 22 21:53:36.550481 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:53:36.550470 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5d34d64-78e3-4285-8568-175030180713-tls-certs\") pod \"rate-test-model-e44515-kserve-5745bf64c9-8z8xm\" (UID: \"e5d34d64-78e3-4285-8568-175030180713\") " pod="ai-tenant-e2e-shared-b-e44515/rate-test-model-e44515-kserve-5745bf64c9-8z8xm" Jul 22 21:53:36.651214 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:53:36.651151 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xf7w\" (UniqueName: \"kubernetes.io/projected/e5d34d64-78e3-4285-8568-175030180713-kube-api-access-5xf7w\") pod \"rate-test-model-e44515-kserve-5745bf64c9-8z8xm\" (UID: \"e5d34d64-78e3-4285-8568-175030180713\") " pod="ai-tenant-e2e-shared-b-e44515/rate-test-model-e44515-kserve-5745bf64c9-8z8xm" Jul 22 21:53:36.651452 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:53:36.651226 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5d34d64-78e3-4285-8568-175030180713-model-cache\") pod \"rate-test-model-e44515-kserve-5745bf64c9-8z8xm\" (UID: \"e5d34d64-78e3-4285-8568-175030180713\") " pod="ai-tenant-e2e-shared-b-e44515/rate-test-model-e44515-kserve-5745bf64c9-8z8xm" Jul 22 21:53:36.651452 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:53:36.651271 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5d34d64-78e3-4285-8568-175030180713-tmp-dir\") pod \"rate-test-model-e44515-kserve-5745bf64c9-8z8xm\" (UID: \"e5d34d64-78e3-4285-8568-175030180713\") " pod="ai-tenant-e2e-shared-b-e44515/rate-test-model-e44515-kserve-5745bf64c9-8z8xm" Jul 22 21:53:36.651452 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:53:36.651303 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5d34d64-78e3-4285-8568-175030180713-home\") pod \"rate-test-model-e44515-kserve-5745bf64c9-8z8xm\" (UID: \"e5d34d64-78e3-4285-8568-175030180713\") " pod="ai-tenant-e2e-shared-b-e44515/rate-test-model-e44515-kserve-5745bf64c9-8z8xm" Jul 22 21:53:36.651452 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:53:36.651318 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5d34d64-78e3-4285-8568-175030180713-tls-certs\") pod \"rate-test-model-e44515-kserve-5745bf64c9-8z8xm\" (UID: \"e5d34d64-78e3-4285-8568-175030180713\") " pod="ai-tenant-e2e-shared-b-e44515/rate-test-model-e44515-kserve-5745bf64c9-8z8xm" Jul 22 21:53:36.651452 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:53:36.651372 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5d34d64-78e3-4285-8568-175030180713-dshm\") pod \"rate-test-model-e44515-kserve-5745bf64c9-8z8xm\" (UID: \"e5d34d64-78e3-4285-8568-175030180713\") " pod="ai-tenant-e2e-shared-b-e44515/rate-test-model-e44515-kserve-5745bf64c9-8z8xm" Jul 22 21:53:36.651721 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:53:36.651684 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5d34d64-78e3-4285-8568-175030180713-model-cache\") pod \"rate-test-model-e44515-kserve-5745bf64c9-8z8xm\" (UID: \"e5d34d64-78e3-4285-8568-175030180713\") " pod="ai-tenant-e2e-shared-b-e44515/rate-test-model-e44515-kserve-5745bf64c9-8z8xm" Jul 22 21:53:36.651770 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:53:36.651734 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5d34d64-78e3-4285-8568-175030180713-tmp-dir\") pod \"rate-test-model-e44515-kserve-5745bf64c9-8z8xm\" (UID: \"e5d34d64-78e3-4285-8568-175030180713\") " pod="ai-tenant-e2e-shared-b-e44515/rate-test-model-e44515-kserve-5745bf64c9-8z8xm" Jul 22 21:53:36.651770 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:53:36.651754 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5d34d64-78e3-4285-8568-175030180713-home\") pod \"rate-test-model-e44515-kserve-5745bf64c9-8z8xm\" (UID: \"e5d34d64-78e3-4285-8568-175030180713\") " pod="ai-tenant-e2e-shared-b-e44515/rate-test-model-e44515-kserve-5745bf64c9-8z8xm" Jul 22 21:53:36.654327 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:53:36.653715 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5d34d64-78e3-4285-8568-175030180713-dshm\") pod \"rate-test-model-e44515-kserve-5745bf64c9-8z8xm\" (UID: \"e5d34d64-78e3-4285-8568-175030180713\") " pod="ai-tenant-e2e-shared-b-e44515/rate-test-model-e44515-kserve-5745bf64c9-8z8xm" Jul 22 21:53:36.654327 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:53:36.654137 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5d34d64-78e3-4285-8568-175030180713-tls-certs\") pod \"rate-test-model-e44515-kserve-5745bf64c9-8z8xm\" (UID: \"e5d34d64-78e3-4285-8568-175030180713\") " pod="ai-tenant-e2e-shared-b-e44515/rate-test-model-e44515-kserve-5745bf64c9-8z8xm" Jul 22 21:53:36.661070 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:53:36.661051 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xf7w\" (UniqueName: \"kubernetes.io/projected/e5d34d64-78e3-4285-8568-175030180713-kube-api-access-5xf7w\") pod \"rate-test-model-e44515-kserve-5745bf64c9-8z8xm\" (UID: \"e5d34d64-78e3-4285-8568-175030180713\") " pod="ai-tenant-e2e-shared-b-e44515/rate-test-model-e44515-kserve-5745bf64c9-8z8xm" Jul 22 21:53:36.764982 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:53:36.764925 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e44515/rate-test-model-e44515-kserve-5745bf64c9-8z8xm" Jul 22 21:53:36.895785 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:53:36.895762 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-e44515/rate-test-model-e44515-kserve-5745bf64c9-8z8xm"] Jul 22 21:53:36.897612 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:53:36.897587 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode5d34d64_78e3_4285_8568_175030180713.slice/crio-fbf11ff39443b23545810e20207d63691b2f076c615b18d848f213d340b1d228 WatchSource:0}: Error finding container fbf11ff39443b23545810e20207d63691b2f076c615b18d848f213d340b1d228: Status 404 returned error can't find the container with id fbf11ff39443b23545810e20207d63691b2f076c615b18d848f213d340b1d228 Jul 22 21:53:37.864264 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:53:37.864227 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e44515/rate-test-model-e44515-kserve-5745bf64c9-8z8xm" event={"ID":"e5d34d64-78e3-4285-8568-175030180713","Type":"ContainerStarted","Data":"a53b6ac8d8f7c5330cf2350bd85b3ac3bbc2b6c80da83e0dbd6683689f1e2caf"} Jul 22 21:53:37.864625 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:53:37.864276 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e44515/rate-test-model-e44515-kserve-5745bf64c9-8z8xm" event={"ID":"e5d34d64-78e3-4285-8568-175030180713","Type":"ContainerStarted","Data":"fbf11ff39443b23545810e20207d63691b2f076c615b18d848f213d340b1d228"} Jul 22 21:53:37.891346 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:53:37.891290 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-e44515/rate-test-model-e44515-kserve-5745bf64c9-8z8xm" podStartSLOduration=1.891274645 podStartE2EDuration="1.891274645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 21:53:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 21:53:37.890414703 +0000 UTC m=+2873.004807049" watchObservedRunningTime="2026-07-22 21:53:37.891274645 +0000 UTC m=+2873.005666990" Jul 22 21:53:40.579235 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:53:40.579203 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:53:46.765302 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:53:46.765265 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-e44515/rate-test-model-e44515-kserve-5745bf64c9-8z8xm" Jul 22 21:53:46.765676 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:53:46.765316 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-e44515/rate-test-model-e44515-kserve-5745bf64c9-8z8xm" Jul 22 21:53:46.777351 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:53:46.777289 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-e44515/rate-test-model-e44515-kserve-5745bf64c9-8z8xm" Jul 22 21:53:46.923066 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:53:46.923040 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-e44515/rate-test-model-e44515-kserve-5745bf64c9-8z8xm" Jul 22 21:54:02.545612 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:02.545578 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:54:17.122395 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:17.122364 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:54:24.032877 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:24.032837 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-e44515/rate-test-model-e44515-kserve-5745bf64c9-8z8xm"] Jul 22 21:54:25.016652 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:25.016618 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7587-139623100296752"] Jul 22 21:54:25.019930 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:25.019915 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7587-139623100296752" Jul 22 21:54:25.022592 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:25.022572 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-pvl5g\"" Jul 22 21:54:25.027845 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:25.027824 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7587-139623100296752"] Jul 22 21:54:25.071299 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:25.071279 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzdzc\" (UniqueName: \"kubernetes.io/projected/b182972f-0d6a-45fb-8627-606852b5b343-kube-api-access-nzdzc\") pod \"test-curl-7587-139623100296752\" (UID: \"b182972f-0d6a-45fb-8627-606852b5b343\") " pod="opendatahub/test-curl-7587-139623100296752" Jul 22 21:54:25.172325 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:25.172298 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzdzc\" (UniqueName: \"kubernetes.io/projected/b182972f-0d6a-45fb-8627-606852b5b343-kube-api-access-nzdzc\") pod \"test-curl-7587-139623100296752\" (UID: \"b182972f-0d6a-45fb-8627-606852b5b343\") " pod="opendatahub/test-curl-7587-139623100296752" Jul 22 21:54:25.181059 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:25.181033 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzdzc\" (UniqueName: \"kubernetes.io/projected/b182972f-0d6a-45fb-8627-606852b5b343-kube-api-access-nzdzc\") pod \"test-curl-7587-139623100296752\" (UID: \"b182972f-0d6a-45fb-8627-606852b5b343\") " pod="opendatahub/test-curl-7587-139623100296752" Jul 22 21:54:25.329739 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:25.329720 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7587-139623100296752" Jul 22 21:54:25.457206 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:25.457150 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7587-139623100296752"] Jul 22 21:54:25.458426 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:54:25.458398 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb182972f_0d6a_45fb_8627_606852b5b343.slice/crio-aa6e6a591c2d82aaf605638831d10a3837e2c30a5044d1e41e3ed6f8ce8f7eb8 WatchSource:0}: Error finding container aa6e6a591c2d82aaf605638831d10a3837e2c30a5044d1e41e3ed6f8ce8f7eb8: Status 404 returned error can't find the container with id aa6e6a591c2d82aaf605638831d10a3837e2c30a5044d1e41e3ed6f8ce8f7eb8 Jul 22 21:54:25.459737 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:25.459720 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 21:54:26.117699 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:26.117659 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7587-139623100296752" event={"ID":"b182972f-0d6a-45fb-8627-606852b5b343","Type":"ContainerStarted","Data":"aa6e6a591c2d82aaf605638831d10a3837e2c30a5044d1e41e3ed6f8ce8f7eb8"} Jul 22 21:54:27.124220 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:27.124187 2572 generic.go:356] "Generic (PLEG): container finished" podID="b182972f-0d6a-45fb-8627-606852b5b343" containerID="29b6cbc0446ddbb90a783267d635007c31a4df4f4f92c7e3142ad03f590663a3" exitCode=0 Jul 22 21:54:27.124601 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:27.124263 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7587-139623100296752" event={"ID":"b182972f-0d6a-45fb-8627-606852b5b343","Type":"ContainerDied","Data":"29b6cbc0446ddbb90a783267d635007c31a4df4f4f92c7e3142ad03f590663a3"} Jul 22 21:54:28.270742 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:28.270721 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7587-139623100296752" Jul 22 21:54:28.295380 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:28.295358 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nzdzc\" (UniqueName: \"kubernetes.io/projected/b182972f-0d6a-45fb-8627-606852b5b343-kube-api-access-nzdzc\") pod \"b182972f-0d6a-45fb-8627-606852b5b343\" (UID: \"b182972f-0d6a-45fb-8627-606852b5b343\") " Jul 22 21:54:28.297495 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:28.297468 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b182972f-0d6a-45fb-8627-606852b5b343-kube-api-access-nzdzc" (OuterVolumeSpecName: "kube-api-access-nzdzc") pod "b182972f-0d6a-45fb-8627-606852b5b343" (UID: "b182972f-0d6a-45fb-8627-606852b5b343"). InnerVolumeSpecName "kube-api-access-nzdzc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 21:54:28.349972 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:28.349940 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7587-139623100296752_b182972f-0d6a-45fb-8627-606852b5b343/test-curl-7587-139623100296752/0.log" Jul 22 21:54:28.384986 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:28.384922 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7587-139623100296752"] Jul 22 21:54:28.391995 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:28.391972 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7587-139623100296752"] Jul 22 21:54:28.396253 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:28.396236 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzdzc\" (UniqueName: \"kubernetes.io/projected/b182972f-0d6a-45fb-8627-606852b5b343-kube-api-access-nzdzc\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:54:29.136277 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:29.136246 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa6e6a591c2d82aaf605638831d10a3837e2c30a5044d1e41e3ed6f8ce8f7eb8" Jul 22 21:54:29.136277 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:29.136251 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7587-139623100296752" Jul 22 21:54:29.145676 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:54:29.145646 2572 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7587-139623100296752\" is forbidden: User \"system:node:ip-10-0-135-37.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-37.ec2.internal' and this object" podUID="b182972f-0d6a-45fb-8627-606852b5b343" pod="opendatahub/test-curl-7587-139623100296752" Jul 22 21:54:29.383877 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:29.383834 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b182972f-0d6a-45fb-8627-606852b5b343" path="/var/lib/kubelet/pods/b182972f-0d6a-45fb-8627-606852b5b343/volumes" Jul 22 21:54:31.583796 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:31.583769 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7587-139623100400672"] Jul 22 21:54:31.584156 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:31.584111 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b182972f-0d6a-45fb-8627-606852b5b343" containerName="test-curl-7587-139623100296752" Jul 22 21:54:31.584156 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:31.584122 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="b182972f-0d6a-45fb-8627-606852b5b343" containerName="test-curl-7587-139623100296752" Jul 22 21:54:31.584256 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:31.584212 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="b182972f-0d6a-45fb-8627-606852b5b343" containerName="test-curl-7587-139623100296752" Jul 22 21:54:31.587564 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:31.587549 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7587-139623100400672" Jul 22 21:54:31.591035 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:31.591005 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-pvl5g\"" Jul 22 21:54:31.595881 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:31.595855 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7587-139623100400672"] Jul 22 21:54:31.625360 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:31.625336 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7wnk\" (UniqueName: \"kubernetes.io/projected/a86ee2a9-1512-45af-83ad-0950618d8cc0-kube-api-access-d7wnk\") pod \"test-curl-7587-139623100400672\" (UID: \"a86ee2a9-1512-45af-83ad-0950618d8cc0\") " pod="opendatahub/test-curl-7587-139623100400672" Jul 22 21:54:31.726438 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:31.726415 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7wnk\" (UniqueName: \"kubernetes.io/projected/a86ee2a9-1512-45af-83ad-0950618d8cc0-kube-api-access-d7wnk\") pod \"test-curl-7587-139623100400672\" (UID: \"a86ee2a9-1512-45af-83ad-0950618d8cc0\") " pod="opendatahub/test-curl-7587-139623100400672" Jul 22 21:54:31.736377 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:31.736349 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7wnk\" (UniqueName: \"kubernetes.io/projected/a86ee2a9-1512-45af-83ad-0950618d8cc0-kube-api-access-d7wnk\") pod \"test-curl-7587-139623100400672\" (UID: \"a86ee2a9-1512-45af-83ad-0950618d8cc0\") " pod="opendatahub/test-curl-7587-139623100400672" Jul 22 21:54:31.897369 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:31.897310 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7587-139623100400672" Jul 22 21:54:32.030261 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:32.030235 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7587-139623100400672"] Jul 22 21:54:32.032393 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:54:32.032369 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda86ee2a9_1512_45af_83ad_0950618d8cc0.slice/crio-821d496dbe9798d27606e932f999274debfa935800566ba70239fdc98620b047 WatchSource:0}: Error finding container 821d496dbe9798d27606e932f999274debfa935800566ba70239fdc98620b047: Status 404 returned error can't find the container with id 821d496dbe9798d27606e932f999274debfa935800566ba70239fdc98620b047 Jul 22 21:54:32.152515 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:32.152460 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7587-139623100400672" event={"ID":"a86ee2a9-1512-45af-83ad-0950618d8cc0","Type":"ContainerStarted","Data":"821d496dbe9798d27606e932f999274debfa935800566ba70239fdc98620b047"} Jul 22 21:54:33.159096 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:33.159067 2572 generic.go:356] "Generic (PLEG): container finished" podID="a86ee2a9-1512-45af-83ad-0950618d8cc0" containerID="4643a5166414dd097cd73597203010493b0410a789f7676421539ad24d13cec9" exitCode=0 Jul 22 21:54:33.159536 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:33.159154 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7587-139623100400672" event={"ID":"a86ee2a9-1512-45af-83ad-0950618d8cc0","Type":"ContainerDied","Data":"4643a5166414dd097cd73597203010493b0410a789f7676421539ad24d13cec9"} Jul 22 21:54:34.306846 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:34.306822 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7587-139623100400672" Jul 22 21:54:34.331042 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:34.331019 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7587-139623100400672_a86ee2a9-1512-45af-83ad-0950618d8cc0/test-curl-7587-139623100400672/0.log" Jul 22 21:54:34.347781 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:34.347758 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d7wnk\" (UniqueName: \"kubernetes.io/projected/a86ee2a9-1512-45af-83ad-0950618d8cc0-kube-api-access-d7wnk\") pod \"a86ee2a9-1512-45af-83ad-0950618d8cc0\" (UID: \"a86ee2a9-1512-45af-83ad-0950618d8cc0\") " Jul 22 21:54:34.349899 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:34.349873 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a86ee2a9-1512-45af-83ad-0950618d8cc0-kube-api-access-d7wnk" (OuterVolumeSpecName: "kube-api-access-d7wnk") pod "a86ee2a9-1512-45af-83ad-0950618d8cc0" (UID: "a86ee2a9-1512-45af-83ad-0950618d8cc0"). InnerVolumeSpecName "kube-api-access-d7wnk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 21:54:34.357892 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:34.357867 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7587-139623100400672"] Jul 22 21:54:34.360787 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:34.360770 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7587-139623100400672"] Jul 22 21:54:34.448810 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:34.448751 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7wnk\" (UniqueName: \"kubernetes.io/projected/a86ee2a9-1512-45af-83ad-0950618d8cc0-kube-api-access-d7wnk\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:54:34.483452 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:34.483428 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7587-139623100400672"] Jul 22 21:54:34.483818 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:34.483806 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a86ee2a9-1512-45af-83ad-0950618d8cc0" containerName="test-curl-7587-139623100400672" Jul 22 21:54:34.483859 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:34.483819 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="a86ee2a9-1512-45af-83ad-0950618d8cc0" containerName="test-curl-7587-139623100400672" Jul 22 21:54:34.483896 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:34.483888 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="a86ee2a9-1512-45af-83ad-0950618d8cc0" containerName="test-curl-7587-139623100400672" Jul 22 21:54:34.486835 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:34.486818 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7587-139623100400672" Jul 22 21:54:34.493526 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:34.493467 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7587-139623100400672"] Jul 22 21:54:34.547640 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:34.547616 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:54:34.549201 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:34.549186 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r94vc\" (UniqueName: \"kubernetes.io/projected/88ee2b51-1317-472f-9405-857c489a961a-kube-api-access-r94vc\") pod \"test-curl-7587-139623100400672\" (UID: \"88ee2b51-1317-472f-9405-857c489a961a\") " pod="opendatahub/test-curl-7587-139623100400672" Jul 22 21:54:34.649897 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:34.649871 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r94vc\" (UniqueName: \"kubernetes.io/projected/88ee2b51-1317-472f-9405-857c489a961a-kube-api-access-r94vc\") pod \"test-curl-7587-139623100400672\" (UID: \"88ee2b51-1317-472f-9405-857c489a961a\") " pod="opendatahub/test-curl-7587-139623100400672" Jul 22 21:54:34.658932 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:34.658902 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r94vc\" (UniqueName: \"kubernetes.io/projected/88ee2b51-1317-472f-9405-857c489a961a-kube-api-access-r94vc\") pod \"test-curl-7587-139623100400672\" (UID: \"88ee2b51-1317-472f-9405-857c489a961a\") " pod="opendatahub/test-curl-7587-139623100400672" Jul 22 21:54:34.796423 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:34.796362 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7587-139623100400672" Jul 22 21:54:34.929672 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:34.929649 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7587-139623100400672"] Jul 22 21:54:34.931110 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:54:34.931084 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod88ee2b51_1317_472f_9405_857c489a961a.slice/crio-178acd1f67d9f6208ea466f2ea254cc692acd37229d518d8d2e3caa6684109cf WatchSource:0}: Error finding container 178acd1f67d9f6208ea466f2ea254cc692acd37229d518d8d2e3caa6684109cf: Status 404 returned error can't find the container with id 178acd1f67d9f6208ea466f2ea254cc692acd37229d518d8d2e3caa6684109cf Jul 22 21:54:35.170646 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:35.170625 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7587-139623100400672" Jul 22 21:54:35.170755 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:35.170631 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="821d496dbe9798d27606e932f999274debfa935800566ba70239fdc98620b047" Jul 22 21:54:35.171918 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:35.171892 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7587-139623100400672" event={"ID":"88ee2b51-1317-472f-9405-857c489a961a","Type":"ContainerStarted","Data":"178acd1f67d9f6208ea466f2ea254cc692acd37229d518d8d2e3caa6684109cf"} Jul 22 21:54:35.182178 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:35.182141 2572 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7587-139623100400672" oldPodUID="a86ee2a9-1512-45af-83ad-0950618d8cc0" podUID="88ee2b51-1317-472f-9405-857c489a961a" Jul 22 21:54:35.383450 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:35.383425 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a86ee2a9-1512-45af-83ad-0950618d8cc0" path="/var/lib/kubelet/pods/a86ee2a9-1512-45af-83ad-0950618d8cc0/volumes" Jul 22 21:54:36.178524 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:36.178491 2572 generic.go:356] "Generic (PLEG): container finished" podID="88ee2b51-1317-472f-9405-857c489a961a" containerID="d62e2d8aa18b055fd832fe377b132d6ea8743d4619c711da1ea6dfea1166b72d" exitCode=0 Jul 22 21:54:36.178690 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:36.178542 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7587-139623100400672" event={"ID":"88ee2b51-1317-472f-9405-857c489a961a","Type":"ContainerDied","Data":"d62e2d8aa18b055fd832fe377b132d6ea8743d4619c711da1ea6dfea1166b72d"} Jul 22 21:54:37.310424 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:37.310402 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7587-139623100400672" Jul 22 21:54:37.330892 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:37.330865 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7587-139623100400672_88ee2b51-1317-472f-9405-857c489a961a/test-curl-7587-139623100400672/0.log" Jul 22 21:54:37.353733 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:37.353707 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7587-139623100400672"] Jul 22 21:54:37.355820 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:37.355799 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7587-139623100400672"] Jul 22 21:54:37.374450 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:37.374419 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r94vc\" (UniqueName: \"kubernetes.io/projected/88ee2b51-1317-472f-9405-857c489a961a-kube-api-access-r94vc\") pod \"88ee2b51-1317-472f-9405-857c489a961a\" (UID: \"88ee2b51-1317-472f-9405-857c489a961a\") " Jul 22 21:54:37.376473 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:37.376444 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88ee2b51-1317-472f-9405-857c489a961a-kube-api-access-r94vc" (OuterVolumeSpecName: "kube-api-access-r94vc") pod "88ee2b51-1317-472f-9405-857c489a961a" (UID: "88ee2b51-1317-472f-9405-857c489a961a"). InnerVolumeSpecName "kube-api-access-r94vc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 21:54:37.382447 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:37.382429 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88ee2b51-1317-472f-9405-857c489a961a" path="/var/lib/kubelet/pods/88ee2b51-1317-472f-9405-857c489a961a/volumes" Jul 22 21:54:37.476083 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:37.476023 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r94vc\" (UniqueName: \"kubernetes.io/projected/88ee2b51-1317-472f-9405-857c489a961a-kube-api-access-r94vc\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:54:38.190297 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:38.190267 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7587-139623100400672" Jul 22 21:54:38.190495 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:38.190301 2572 scope.go:117] "RemoveContainer" containerID="d62e2d8aa18b055fd832fe377b132d6ea8743d4619c711da1ea6dfea1166b72d" Jul 22 21:54:39.052739 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:39.052681 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-e44515/rate-test-model-e44515-kserve-5745bf64c9-8z8xm" podUID="e5d34d64-78e3-4285-8568-175030180713" containerName="main" containerID="cri-o://a53b6ac8d8f7c5330cf2350bd85b3ac3bbc2b6c80da83e0dbd6683689f1e2caf" gracePeriod=45 Jul 22 21:54:39.199639 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:39.199604 2572 generic.go:356] "Generic (PLEG): container finished" podID="e5d34d64-78e3-4285-8568-175030180713" containerID="a53b6ac8d8f7c5330cf2350bd85b3ac3bbc2b6c80da83e0dbd6683689f1e2caf" exitCode=0 Jul 22 21:54:39.199775 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:39.199661 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e44515/rate-test-model-e44515-kserve-5745bf64c9-8z8xm" event={"ID":"e5d34d64-78e3-4285-8568-175030180713","Type":"ContainerDied","Data":"a53b6ac8d8f7c5330cf2350bd85b3ac3bbc2b6c80da83e0dbd6683689f1e2caf"} Jul 22 21:54:39.303483 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:39.303426 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e44515/rate-test-model-e44515-kserve-5745bf64c9-8z8xm" Jul 22 21:54:39.389825 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:39.389801 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5d34d64-78e3-4285-8568-175030180713-home\") pod \"e5d34d64-78e3-4285-8568-175030180713\" (UID: \"e5d34d64-78e3-4285-8568-175030180713\") " Jul 22 21:54:39.389957 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:39.389859 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5xf7w\" (UniqueName: \"kubernetes.io/projected/e5d34d64-78e3-4285-8568-175030180713-kube-api-access-5xf7w\") pod \"e5d34d64-78e3-4285-8568-175030180713\" (UID: \"e5d34d64-78e3-4285-8568-175030180713\") " Jul 22 21:54:39.389957 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:39.389898 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5d34d64-78e3-4285-8568-175030180713-tls-certs\") pod \"e5d34d64-78e3-4285-8568-175030180713\" (UID: \"e5d34d64-78e3-4285-8568-175030180713\") " Jul 22 21:54:39.389957 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:39.389937 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5d34d64-78e3-4285-8568-175030180713-tmp-dir\") pod \"e5d34d64-78e3-4285-8568-175030180713\" (UID: \"e5d34d64-78e3-4285-8568-175030180713\") " Jul 22 21:54:39.390080 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:39.390043 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5d34d64-78e3-4285-8568-175030180713-home" (OuterVolumeSpecName: "home") pod "e5d34d64-78e3-4285-8568-175030180713" (UID: "e5d34d64-78e3-4285-8568-175030180713"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 21:54:39.390080 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:39.390048 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5d34d64-78e3-4285-8568-175030180713-model-cache\") pod \"e5d34d64-78e3-4285-8568-175030180713\" (UID: \"e5d34d64-78e3-4285-8568-175030180713\") " Jul 22 21:54:39.390157 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:39.390144 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5d34d64-78e3-4285-8568-175030180713-dshm\") pod \"e5d34d64-78e3-4285-8568-175030180713\" (UID: \"e5d34d64-78e3-4285-8568-175030180713\") " Jul 22 21:54:39.390235 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:39.390194 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5d34d64-78e3-4285-8568-175030180713-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e5d34d64-78e3-4285-8568-175030180713" (UID: "e5d34d64-78e3-4285-8568-175030180713"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 21:54:39.390290 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:39.390256 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5d34d64-78e3-4285-8568-175030180713-model-cache" (OuterVolumeSpecName: "model-cache") pod "e5d34d64-78e3-4285-8568-175030180713" (UID: "e5d34d64-78e3-4285-8568-175030180713"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 21:54:39.390491 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:39.390468 2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5d34d64-78e3-4285-8568-175030180713-home\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:54:39.390578 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:39.390499 2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5d34d64-78e3-4285-8568-175030180713-tmp-dir\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:54:39.390578 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:39.390510 2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5d34d64-78e3-4285-8568-175030180713-model-cache\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:54:39.392071 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:39.392041 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5d34d64-78e3-4285-8568-175030180713-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e5d34d64-78e3-4285-8568-175030180713" (UID: "e5d34d64-78e3-4285-8568-175030180713"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 21:54:39.392222 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:39.392075 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5d34d64-78e3-4285-8568-175030180713-dshm" (OuterVolumeSpecName: "dshm") pod "e5d34d64-78e3-4285-8568-175030180713" (UID: "e5d34d64-78e3-4285-8568-175030180713"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 21:54:39.392222 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:39.392093 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5d34d64-78e3-4285-8568-175030180713-kube-api-access-5xf7w" (OuterVolumeSpecName: "kube-api-access-5xf7w") pod "e5d34d64-78e3-4285-8568-175030180713" (UID: "e5d34d64-78e3-4285-8568-175030180713"). InnerVolumeSpecName "kube-api-access-5xf7w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 21:54:39.491801 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:39.491772 2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5d34d64-78e3-4285-8568-175030180713-dshm\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:54:39.491801 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:39.491797 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xf7w\" (UniqueName: \"kubernetes.io/projected/e5d34d64-78e3-4285-8568-175030180713-kube-api-access-5xf7w\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:54:39.491801 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:39.491808 2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5d34d64-78e3-4285-8568-175030180713-tls-certs\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:54:40.206983 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:40.206951 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e44515/rate-test-model-e44515-kserve-5745bf64c9-8z8xm" event={"ID":"e5d34d64-78e3-4285-8568-175030180713","Type":"ContainerDied","Data":"fbf11ff39443b23545810e20207d63691b2f076c615b18d848f213d340b1d228"} Jul 22 21:54:40.206983 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:40.206967 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e44515/rate-test-model-e44515-kserve-5745bf64c9-8z8xm" Jul 22 21:54:40.207442 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:40.206996 2572 scope.go:117] "RemoveContainer" containerID="a53b6ac8d8f7c5330cf2350bd85b3ac3bbc2b6c80da83e0dbd6683689f1e2caf" Jul 22 21:54:40.232678 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:40.232657 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-e44515/rate-test-model-e44515-kserve-5745bf64c9-8z8xm"] Jul 22 21:54:40.240300 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:40.240281 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-e44515/rate-test-model-e44515-kserve-5745bf64c9-8z8xm"] Jul 22 21:54:40.568891 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:40.568852 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7587-139623100401680"] Jul 22 21:54:40.569300 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:40.569286 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88ee2b51-1317-472f-9405-857c489a961a" containerName="test-curl-7587-139623100400672" Jul 22 21:54:40.569358 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:40.569302 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="88ee2b51-1317-472f-9405-857c489a961a" containerName="test-curl-7587-139623100400672" Jul 22 21:54:40.569358 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:40.569314 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5d34d64-78e3-4285-8568-175030180713" containerName="main" Jul 22 21:54:40.569358 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:40.569319 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5d34d64-78e3-4285-8568-175030180713" containerName="main" Jul 22 21:54:40.569462 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:40.569380 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5d34d64-78e3-4285-8568-175030180713" containerName="main" Jul 22 21:54:40.569462 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:40.569392 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="88ee2b51-1317-472f-9405-857c489a961a" containerName="test-curl-7587-139623100400672" Jul 22 21:54:40.573989 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:40.573964 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7587-139623100401680" Jul 22 21:54:40.576319 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:40.576299 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-pvl5g\"" Jul 22 21:54:40.576421 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:40.576375 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7587-139623100401680"] Jul 22 21:54:40.703543 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:40.703513 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn9pn\" (UniqueName: \"kubernetes.io/projected/40896bac-3a7f-4c0c-a75a-f272fa1f5ab3-kube-api-access-hn9pn\") pod \"test-curl-7587-139623100401680\" (UID: \"40896bac-3a7f-4c0c-a75a-f272fa1f5ab3\") " pod="opendatahub/test-curl-7587-139623100401680" Jul 22 21:54:40.804707 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:40.804681 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn9pn\" (UniqueName: \"kubernetes.io/projected/40896bac-3a7f-4c0c-a75a-f272fa1f5ab3-kube-api-access-hn9pn\") pod \"test-curl-7587-139623100401680\" (UID: \"40896bac-3a7f-4c0c-a75a-f272fa1f5ab3\") " pod="opendatahub/test-curl-7587-139623100401680" Jul 22 21:54:40.812264 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:40.812243 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn9pn\" (UniqueName: \"kubernetes.io/projected/40896bac-3a7f-4c0c-a75a-f272fa1f5ab3-kube-api-access-hn9pn\") pod \"test-curl-7587-139623100401680\" (UID: \"40896bac-3a7f-4c0c-a75a-f272fa1f5ab3\") " pod="opendatahub/test-curl-7587-139623100401680" Jul 22 21:54:40.884039 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:40.883977 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7587-139623100401680" Jul 22 21:54:41.014092 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:41.014068 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7587-139623100401680"] Jul 22 21:54:41.016418 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:54:41.016378 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod40896bac_3a7f_4c0c_a75a_f272fa1f5ab3.slice/crio-b7437ca8ef776a771b53f0ce952e45459ca886302433e7ed2f7239938ed3d41b WatchSource:0}: Error finding container b7437ca8ef776a771b53f0ce952e45459ca886302433e7ed2f7239938ed3d41b: Status 404 returned error can't find the container with id b7437ca8ef776a771b53f0ce952e45459ca886302433e7ed2f7239938ed3d41b Jul 22 21:54:41.215149 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:41.215081 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7587-139623100401680" event={"ID":"40896bac-3a7f-4c0c-a75a-f272fa1f5ab3","Type":"ContainerStarted","Data":"b7437ca8ef776a771b53f0ce952e45459ca886302433e7ed2f7239938ed3d41b"} Jul 22 21:54:41.383506 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:41.383477 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5d34d64-78e3-4285-8568-175030180713" path="/var/lib/kubelet/pods/e5d34d64-78e3-4285-8568-175030180713/volumes" Jul 22 21:54:42.221769 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:42.221682 2572 generic.go:356] "Generic (PLEG): container finished" podID="40896bac-3a7f-4c0c-a75a-f272fa1f5ab3" containerID="1ab25f33df7d4dce4c92f47fb39f321168c6f5d95f71b5d55d91b02d344ad039" exitCode=0 Jul 22 21:54:42.221769 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:42.221716 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7587-139623100401680" event={"ID":"40896bac-3a7f-4c0c-a75a-f272fa1f5ab3","Type":"ContainerDied","Data":"1ab25f33df7d4dce4c92f47fb39f321168c6f5d95f71b5d55d91b02d344ad039"} Jul 22 21:54:43.365555 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:43.365528 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7587-139623100401680" Jul 22 21:54:43.391598 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:43.391576 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7587-139623100401680_40896bac-3a7f-4c0c-a75a-f272fa1f5ab3/test-curl-7587-139623100401680/0.log" Jul 22 21:54:43.418067 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:43.418035 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7587-139623100401680"] Jul 22 21:54:43.421485 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:43.421465 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7587-139623100401680"] Jul 22 21:54:43.426064 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:43.426041 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hn9pn\" (UniqueName: \"kubernetes.io/projected/40896bac-3a7f-4c0c-a75a-f272fa1f5ab3-kube-api-access-hn9pn\") pod \"40896bac-3a7f-4c0c-a75a-f272fa1f5ab3\" (UID: \"40896bac-3a7f-4c0c-a75a-f272fa1f5ab3\") " Jul 22 21:54:43.428263 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:43.428232 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40896bac-3a7f-4c0c-a75a-f272fa1f5ab3-kube-api-access-hn9pn" (OuterVolumeSpecName: "kube-api-access-hn9pn") pod "40896bac-3a7f-4c0c-a75a-f272fa1f5ab3" (UID: "40896bac-3a7f-4c0c-a75a-f272fa1f5ab3"). InnerVolumeSpecName "kube-api-access-hn9pn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 21:54:43.527225 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:43.527148 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hn9pn\" (UniqueName: \"kubernetes.io/projected/40896bac-3a7f-4c0c-a75a-f272fa1f5ab3-kube-api-access-hn9pn\") on node \"ip-10-0-135-37.ec2.internal\" DevicePath \"\"" Jul 22 21:54:44.233573 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:44.233540 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7587-139623100401680" Jul 22 21:54:44.233573 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:44.233557 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b7437ca8ef776a771b53f0ce952e45459ca886302433e7ed2f7239938ed3d41b" Jul 22 21:54:44.244067 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:54:44.244040 2572 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7587-139623100401680\" is forbidden: User \"system:node:ip-10-0-135-37.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-37.ec2.internal' and this object" podUID="40896bac-3a7f-4c0c-a75a-f272fa1f5ab3" pod="opendatahub/test-curl-7587-139623100401680" Jul 22 21:54:45.383502 ip-10-0-135-37 kubenswrapper[2572]: E0722 21:54:45.383461 2572 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7587-139623100401680\" is forbidden: User \"system:node:ip-10-0-135-37.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-37.ec2.internal' and this object" podUID="40896bac-3a7f-4c0c-a75a-f272fa1f5ab3" pod="opendatahub/test-curl-7587-139623100401680" Jul 22 21:54:45.384768 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:45.384742 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40896bac-3a7f-4c0c-a75a-f272fa1f5ab3" path="/var/lib/kubelet/pods/40896bac-3a7f-4c0c-a75a-f272fa1f5ab3/volumes" Jul 22 21:54:56.956529 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:54:56.956494 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:55:19.743259 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:19.743184 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-88kxw"] Jul 22 21:55:28.524402 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:28.524365 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5b9c6cff9-gp6cx_57c6e48c-5b69-40e8-b774-21c4b4100d5d/authorino/0.log" Jul 22 21:55:34.018582 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:34.018549 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-2q2tj_aadfbe03-82fd-48a8-840d-53c85d26653c/manager/0.log" Jul 22 21:55:34.711051 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:34.711022 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5b88fc4594-pc9qd_b016459c-2f29-4fa9-9681-3e320ad43e7a/copy-manifests/0.log" Jul 22 21:55:34.718449 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:34.718424 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5b88fc4594-pc9qd_b016459c-2f29-4fa9-9681-3e320ad43e7a/manager/0.log" Jul 22 21:55:36.147081 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:36.147043 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594h74l_abe06a85-7716-4827-9767-9f9deef29783/util/0.log" Jul 22 21:55:36.153741 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:36.153715 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594h74l_abe06a85-7716-4827-9767-9f9deef29783/pull/0.log" Jul 22 21:55:36.159424 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:36.159402 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594h74l_abe06a85-7716-4827-9767-9f9deef29783/extract/0.log" Jul 22 21:55:36.268774 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:36.268749 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rbwgs_1c2d0ac7-0c4e-4c4a-9acc-75be526b7a79/pull/0.log" Jul 22 21:55:36.280086 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:36.280068 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rbwgs_1c2d0ac7-0c4e-4c4a-9acc-75be526b7a79/extract/0.log" Jul 22 21:55:36.286271 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:36.286254 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rbwgs_1c2d0ac7-0c4e-4c4a-9acc-75be526b7a79/util/0.log" Jul 22 21:55:36.532710 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:36.532623 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19r4kn_1642c073-6026-4a1c-b51a-467b60e75087/pull/0.log" Jul 22 21:55:36.542139 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:36.542113 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19r4kn_1642c073-6026-4a1c-b51a-467b60e75087/extract/0.log" Jul 22 21:55:36.550008 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:36.549982 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19r4kn_1642c073-6026-4a1c-b51a-467b60e75087/util/0.log" Jul 22 21:55:36.663503 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:36.663480 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5b9c6cff9-gp6cx_57c6e48c-5b69-40e8-b774-21c4b4100d5d/authorino/0.log" Jul 22 21:55:37.353367 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:37.353336 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-88kxw_e8f04df2-b7b0-4e87-b649-9593bde9faf9/limitador/0.log" Jul 22 21:55:37.484547 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:37.484514 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-8f4vc_6f940cfb-15b1-4301-ba08-2d7d86e25c66/manager/0.log" Jul 22 21:55:37.824865 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:37.824839 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4gcskt_a0096832-cef8-4070-9566-49b7553d4607/istio-proxy/0.log" Jul 22 21:55:38.268037 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:38.267949 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-chp6r_40faf561-504b-45df-9c9b-211772e1e444/istio-proxy/1.log" Jul 22 21:55:38.607764 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:38.607725 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-66b4d7486b-9q9kc_14d2ff8c-748e-4a6f-801f-34cccafb93a8/router/0.log" Jul 22 21:55:39.279431 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:39.279403 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkj7rj_1a53c4ca-1b67-4422-8131-368b274b4dfa/main/0.log" Jul 22 21:55:39.494923 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:39.494897 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-gksj5_42dd4d0d-9036-4fbd-97ad-9e3567b97cc2/main/0.log" Jul 22 21:55:47.442983 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:47.442947 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-zrg2r_4f3c7912-ed80-4a24-ada3-b2a3033bb210/global-pull-secret-syncer/0.log" Jul 22 21:55:47.559579 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:47.559549 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-m7pxh_37b20c60-c50a-42df-acd7-2164653d94ea/konnectivity-agent/0.log" Jul 22 21:55:47.581921 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:47.581895 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-37.ec2.internal_091f2ee13ccf617382e7e126cc5d9e12/haproxy/0.log" Jul 22 21:55:51.851474 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:51.851437 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594h74l_abe06a85-7716-4827-9767-9f9deef29783/extract/0.log" Jul 22 21:55:51.880224 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:51.880196 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594h74l_abe06a85-7716-4827-9767-9f9deef29783/util/0.log" Jul 22 21:55:51.907182 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:51.907139 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594h74l_abe06a85-7716-4827-9767-9f9deef29783/pull/0.log" Jul 22 21:55:51.933986 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:51.933963 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rbwgs_1c2d0ac7-0c4e-4c4a-9acc-75be526b7a79/extract/0.log" Jul 22 21:55:51.954382 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:51.954360 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rbwgs_1c2d0ac7-0c4e-4c4a-9acc-75be526b7a79/util/0.log" Jul 22 21:55:51.976822 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:51.976801 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rbwgs_1c2d0ac7-0c4e-4c4a-9acc-75be526b7a79/pull/0.log" Jul 22 21:55:52.102638 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:52.102580 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19r4kn_1642c073-6026-4a1c-b51a-467b60e75087/extract/0.log" Jul 22 21:55:52.130879 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:52.130840 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19r4kn_1642c073-6026-4a1c-b51a-467b60e75087/util/0.log" Jul 22 21:55:52.153286 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:52.153267 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19r4kn_1642c073-6026-4a1c-b51a-467b60e75087/pull/0.log" Jul 22 21:55:52.752560 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:52.752525 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5b9c6cff9-gp6cx_57c6e48c-5b69-40e8-b774-21c4b4100d5d/authorino/0.log" Jul 22 21:55:53.059666 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:53.059639 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-88kxw_e8f04df2-b7b0-4e87-b649-9593bde9faf9/limitador/0.log" Jul 22 21:55:53.159055 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:53.159023 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-8f4vc_6f940cfb-15b1-4301-ba08-2d7d86e25c66/manager/0.log" Jul 22 21:55:54.807770 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:54.807739 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_dec07965-3435-4118-b8bd-f6b20ca3db58/alertmanager/0.log" Jul 22 21:55:54.847233 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:54.847207 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_dec07965-3435-4118-b8bd-f6b20ca3db58/config-reloader/0.log" Jul 22 21:55:54.867820 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:54.867755 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_dec07965-3435-4118-b8bd-f6b20ca3db58/kube-rbac-proxy-web/0.log" Jul 22 21:55:54.892771 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:54.892738 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_dec07965-3435-4118-b8bd-f6b20ca3db58/kube-rbac-proxy/0.log" Jul 22 21:55:54.914134 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:54.914114 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_dec07965-3435-4118-b8bd-f6b20ca3db58/kube-rbac-proxy-metric/0.log" Jul 22 21:55:54.947735 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:54.947715 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_dec07965-3435-4118-b8bd-f6b20ca3db58/prom-label-proxy/0.log" Jul 22 21:55:54.976663 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:54.976639 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_dec07965-3435-4118-b8bd-f6b20ca3db58/init-config-reloader/0.log" Jul 22 21:55:55.025530 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:55.025510 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-zbtpg_3036abe7-96a8-4ae5-be89-97d3071f96e6/cluster-monitoring-operator/0.log" Jul 22 21:55:55.297946 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:55.297881 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7c67d7698d-8srlj_e2c1979d-ec4c-4267-be6f-cb6aa56036ff/metrics-server/0.log" Jul 22 21:55:55.323000 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:55.322979 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-7b7b7bc8bf-qpftl_9a55d3ef-6b38-48ed-a6d5-51104b54eb5a/monitoring-plugin/0.log" Jul 22 21:55:55.352992 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:55.352959 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6q24d_2e6c99e9-55f5-4cf5-b429-23c08bc76207/node-exporter/0.log" Jul 22 21:55:55.372794 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:55.372776 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6q24d_2e6c99e9-55f5-4cf5-b429-23c08bc76207/kube-rbac-proxy/0.log" Jul 22 21:55:55.398110 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:55.398090 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6q24d_2e6c99e9-55f5-4cf5-b429-23c08bc76207/init-textfile/0.log" Jul 22 21:55:55.581501 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:55.581479 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-ckz8k_0274094c-5b35-4e27-872c-f8a1df2d7f79/kube-rbac-proxy-main/0.log" Jul 22 21:55:55.603859 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:55.603833 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-ckz8k_0274094c-5b35-4e27-872c-f8a1df2d7f79/kube-rbac-proxy-self/0.log" Jul 22 21:55:55.626440 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:55.626417 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-ckz8k_0274094c-5b35-4e27-872c-f8a1df2d7f79/openshift-state-metrics/0.log" Jul 22 21:55:55.929429 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:55.929355 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-85574b5f49-v9mwz_41bb58e8-872b-458c-a119-5059515e61d8/prometheus-operator-admission-webhook/0.log" Jul 22 21:55:56.031364 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:56.031341 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-744cbd9999-72qk5_4ba222b4-9746-40b5-bb1b-a710d700b244/thanos-query/0.log" Jul 22 21:55:56.052776 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:56.052755 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-744cbd9999-72qk5_4ba222b4-9746-40b5-bb1b-a710d700b244/kube-rbac-proxy-web/0.log" Jul 22 21:55:56.075013 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:56.074997 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-744cbd9999-72qk5_4ba222b4-9746-40b5-bb1b-a710d700b244/kube-rbac-proxy/0.log" Jul 22 21:55:56.098209 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:56.098188 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-744cbd9999-72qk5_4ba222b4-9746-40b5-bb1b-a710d700b244/prom-label-proxy/0.log" Jul 22 21:55:56.126688 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:56.126668 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-744cbd9999-72qk5_4ba222b4-9746-40b5-bb1b-a710d700b244/kube-rbac-proxy-rules/0.log" Jul 22 21:55:56.148656 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:56.148636 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-744cbd9999-72qk5_4ba222b4-9746-40b5-bb1b-a710d700b244/kube-rbac-proxy-metrics/0.log" Jul 22 21:55:56.998551 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:56.998461 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-sw9zx/perf-node-gather-daemonset-mwf8j"] Jul 22 21:55:56.998945 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:56.998924 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40896bac-3a7f-4c0c-a75a-f272fa1f5ab3" containerName="test-curl-7587-139623100401680" Jul 22 21:55:56.998988 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:56.998947 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="40896bac-3a7f-4c0c-a75a-f272fa1f5ab3" containerName="test-curl-7587-139623100401680" Jul 22 21:55:56.999033 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:56.999023 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="40896bac-3a7f-4c0c-a75a-f272fa1f5ab3" containerName="test-curl-7587-139623100401680" Jul 22 21:55:57.002285 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:57.002270 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sw9zx/perf-node-gather-daemonset-mwf8j" Jul 22 21:55:57.004667 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:57.004648 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-sw9zx\"/\"kube-root-ca.crt\"" Jul 22 21:55:57.004788 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:57.004647 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-sw9zx\"/\"default-dockercfg-pcvcv\"" Jul 22 21:55:57.005634 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:57.005618 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-sw9zx\"/\"openshift-service-ca.crt\"" Jul 22 21:55:57.012075 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:57.012056 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sw9zx/perf-node-gather-daemonset-mwf8j"] Jul 22 21:55:57.121370 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:57.121335 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0d8e2c05-c22f-42a1-95cc-6e9e560c43b2-sys\") pod \"perf-node-gather-daemonset-mwf8j\" (UID: \"0d8e2c05-c22f-42a1-95cc-6e9e560c43b2\") " pod="openshift-must-gather-sw9zx/perf-node-gather-daemonset-mwf8j" Jul 22 21:55:57.121537 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:57.121393 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0d8e2c05-c22f-42a1-95cc-6e9e560c43b2-podres\") pod \"perf-node-gather-daemonset-mwf8j\" (UID: \"0d8e2c05-c22f-42a1-95cc-6e9e560c43b2\") " pod="openshift-must-gather-sw9zx/perf-node-gather-daemonset-mwf8j" Jul 22 21:55:57.121537 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:57.121425 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0d8e2c05-c22f-42a1-95cc-6e9e560c43b2-lib-modules\") pod \"perf-node-gather-daemonset-mwf8j\" (UID: \"0d8e2c05-c22f-42a1-95cc-6e9e560c43b2\") " pod="openshift-must-gather-sw9zx/perf-node-gather-daemonset-mwf8j" Jul 22 21:55:57.121537 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:57.121444 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfcz2\" (UniqueName: \"kubernetes.io/projected/0d8e2c05-c22f-42a1-95cc-6e9e560c43b2-kube-api-access-cfcz2\") pod \"perf-node-gather-daemonset-mwf8j\" (UID: \"0d8e2c05-c22f-42a1-95cc-6e9e560c43b2\") " pod="openshift-must-gather-sw9zx/perf-node-gather-daemonset-mwf8j" Jul 22 21:55:57.121537 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:57.121458 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0d8e2c05-c22f-42a1-95cc-6e9e560c43b2-proc\") pod \"perf-node-gather-daemonset-mwf8j\" (UID: \"0d8e2c05-c22f-42a1-95cc-6e9e560c43b2\") " pod="openshift-must-gather-sw9zx/perf-node-gather-daemonset-mwf8j" Jul 22 21:55:57.221959 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:57.221932 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0d8e2c05-c22f-42a1-95cc-6e9e560c43b2-podres\") pod \"perf-node-gather-daemonset-mwf8j\" (UID: \"0d8e2c05-c22f-42a1-95cc-6e9e560c43b2\") " pod="openshift-must-gather-sw9zx/perf-node-gather-daemonset-mwf8j" Jul 22 21:55:57.222069 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:57.221967 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0d8e2c05-c22f-42a1-95cc-6e9e560c43b2-lib-modules\") pod \"perf-node-gather-daemonset-mwf8j\" (UID: \"0d8e2c05-c22f-42a1-95cc-6e9e560c43b2\") " pod="openshift-must-gather-sw9zx/perf-node-gather-daemonset-mwf8j" Jul 22 21:55:57.222069 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:57.221988 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfcz2\" (UniqueName: \"kubernetes.io/projected/0d8e2c05-c22f-42a1-95cc-6e9e560c43b2-kube-api-access-cfcz2\") pod \"perf-node-gather-daemonset-mwf8j\" (UID: \"0d8e2c05-c22f-42a1-95cc-6e9e560c43b2\") " pod="openshift-must-gather-sw9zx/perf-node-gather-daemonset-mwf8j" Jul 22 21:55:57.222069 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:57.222002 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0d8e2c05-c22f-42a1-95cc-6e9e560c43b2-proc\") pod \"perf-node-gather-daemonset-mwf8j\" (UID: \"0d8e2c05-c22f-42a1-95cc-6e9e560c43b2\") " pod="openshift-must-gather-sw9zx/perf-node-gather-daemonset-mwf8j" Jul 22 21:55:57.222069 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:57.222057 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0d8e2c05-c22f-42a1-95cc-6e9e560c43b2-sys\") pod \"perf-node-gather-daemonset-mwf8j\" (UID: \"0d8e2c05-c22f-42a1-95cc-6e9e560c43b2\") " pod="openshift-must-gather-sw9zx/perf-node-gather-daemonset-mwf8j" Jul 22 21:55:57.222260 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:57.222108 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0d8e2c05-c22f-42a1-95cc-6e9e560c43b2-lib-modules\") pod \"perf-node-gather-daemonset-mwf8j\" (UID: \"0d8e2c05-c22f-42a1-95cc-6e9e560c43b2\") " pod="openshift-must-gather-sw9zx/perf-node-gather-daemonset-mwf8j" Jul 22 21:55:57.222260 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:57.222117 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0d8e2c05-c22f-42a1-95cc-6e9e560c43b2-podres\") pod \"perf-node-gather-daemonset-mwf8j\" (UID: \"0d8e2c05-c22f-42a1-95cc-6e9e560c43b2\") " pod="openshift-must-gather-sw9zx/perf-node-gather-daemonset-mwf8j" Jul 22 21:55:57.222260 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:57.222132 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0d8e2c05-c22f-42a1-95cc-6e9e560c43b2-sys\") pod \"perf-node-gather-daemonset-mwf8j\" (UID: \"0d8e2c05-c22f-42a1-95cc-6e9e560c43b2\") " pod="openshift-must-gather-sw9zx/perf-node-gather-daemonset-mwf8j" Jul 22 21:55:57.222260 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:57.222188 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0d8e2c05-c22f-42a1-95cc-6e9e560c43b2-proc\") pod \"perf-node-gather-daemonset-mwf8j\" (UID: \"0d8e2c05-c22f-42a1-95cc-6e9e560c43b2\") " pod="openshift-must-gather-sw9zx/perf-node-gather-daemonset-mwf8j" Jul 22 21:55:57.230394 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:57.230378 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfcz2\" (UniqueName: \"kubernetes.io/projected/0d8e2c05-c22f-42a1-95cc-6e9e560c43b2-kube-api-access-cfcz2\") pod \"perf-node-gather-daemonset-mwf8j\" (UID: \"0d8e2c05-c22f-42a1-95cc-6e9e560c43b2\") " pod="openshift-must-gather-sw9zx/perf-node-gather-daemonset-mwf8j" Jul 22 21:55:57.312332 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:57.312298 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sw9zx/perf-node-gather-daemonset-mwf8j" Jul 22 21:55:57.443387 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:57.443364 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sw9zx/perf-node-gather-daemonset-mwf8j"] Jul 22 21:55:57.444645 ip-10-0-135-37 kubenswrapper[2572]: W0722 21:55:57.444622 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0d8e2c05_c22f_42a1_95cc_6e9e560c43b2.slice/crio-918147039c2e3493a511bcbe9974ad9f0df118db452e333cf41ffa14577e3174 WatchSource:0}: Error finding container 918147039c2e3493a511bcbe9974ad9f0df118db452e333cf41ffa14577e3174: Status 404 returned error can't find the container with id 918147039c2e3493a511bcbe9974ad9f0df118db452e333cf41ffa14577e3174 Jul 22 21:55:57.619555 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:57.619487 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sw9zx/perf-node-gather-daemonset-mwf8j" event={"ID":"0d8e2c05-c22f-42a1-95cc-6e9e560c43b2","Type":"ContainerStarted","Data":"0b21763d4dc19f49f7294c67fa9d6167646d181eb1344305cfbc4cebc062ff23"} Jul 22 21:55:57.619555 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:57.619537 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sw9zx/perf-node-gather-daemonset-mwf8j" event={"ID":"0d8e2c05-c22f-42a1-95cc-6e9e560c43b2","Type":"ContainerStarted","Data":"918147039c2e3493a511bcbe9974ad9f0df118db452e333cf41ffa14577e3174"} Jul 22 21:55:57.619722 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:57.619585 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-sw9zx/perf-node-gather-daemonset-mwf8j" Jul 22 21:55:57.638847 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:57.638808 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-sw9zx/perf-node-gather-daemonset-mwf8j" podStartSLOduration=1.638795178 podStartE2EDuration="1.638795178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 21:55:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 21:55:57.637027667 +0000 UTC m=+3012.751420010" watchObservedRunningTime="2026-07-22 21:55:57.638795178 +0000 UTC m=+3012.753187520" Jul 22 21:55:58.317274 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:58.317251 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-66b6fd96f9-qtfdc_2b365fa5-795b-45ba-af51-038572c0e32e/console/0.log" Jul 22 21:55:58.996582 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:55:58.996556 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78cf4c6f6-nxdpm_53fa2e94-a71d-451a-921c-e36a2973b37f/volume-data-source-validator/0.log" Jul 22 21:56:00.055935 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:56:00.055906 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7pnql_d959e22d-dd51-47e5-8e9b-0a2b883aad04/dns/0.log" Jul 22 21:56:00.076535 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:56:00.076501 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7pnql_d959e22d-dd51-47e5-8e9b-0a2b883aad04/kube-rbac-proxy/0.log" Jul 22 21:56:00.190653 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:56:00.190621 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-z2g8w_40f77f02-2bc6-4339-a948-7631a9603ce1/dns-node-resolver/0.log" Jul 22 21:56:00.744888 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:56:00.744851 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4s9cs_45b1939d-7d91-4899-b489-7e2609a5878d/node-ca/0.log" Jul 22 21:56:01.716979 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:56:01.716952 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4gcskt_a0096832-cef8-4070-9566-49b7553d4607/istio-proxy/0.log" Jul 22 21:56:01.936541 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:56:01.936516 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-chp6r_40faf561-504b-45df-9c9b-211772e1e444/istio-proxy/0.log" Jul 22 21:56:01.986113 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:56:01.986044 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-chp6r_40faf561-504b-45df-9c9b-211772e1e444/istio-proxy/1.log" Jul 22 21:56:02.073934 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:56:02.073912 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-66b4d7486b-9q9kc_14d2ff8c-748e-4a6f-801f-34cccafb93a8/router/0.log" Jul 22 21:56:02.771303 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:56:02.771276 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-49svb_25aab156-c282-46cd-bd40-bfbf4991f6dc/serve-healthcheck-canary/0.log" Jul 22 21:56:03.537704 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:56:03.537668 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qk2st_ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05/kube-rbac-proxy/0.log" Jul 22 21:56:03.585659 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:56:03.585633 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qk2st_ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05/exporter/0.log" Jul 22 21:56:03.631547 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:56:03.631511 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qk2st_ef9f20ca-5f6d-44bd-bc95-f7bd98a8ff05/extractor/0.log" Jul 22 21:56:03.636285 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:56:03.636268 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-sw9zx/perf-node-gather-daemonset-mwf8j" Jul 22 21:56:06.163030 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:56:06.162945 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-2q2tj_aadfbe03-82fd-48a8-840d-53c85d26653c/manager/0.log" Jul 22 21:56:06.630696 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:56:06.630674 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5b88fc4594-pc9qd_b016459c-2f29-4fa9-9681-3e320ad43e7a/manager/0.log" Jul 22 21:56:06.659999 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:56:06.659980 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5b88fc4594-pc9qd_b016459c-2f29-4fa9-9681-3e320ad43e7a/copy-manifests/0.log" Jul 22 21:56:07.682428 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:56:07.682391 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4gcskt_a0096832-cef8-4070-9566-49b7553d4607/istio-proxy/0.log" Jul 22 21:56:07.866225 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:56:07.866182 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-chp6r_40faf561-504b-45df-9c9b-211772e1e444/istio-proxy/0.log" Jul 22 21:56:07.889978 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:56:07.889954 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-795d667875-4klxv_0e92181f-ccd3-476e-91bd-fb2bdc76b00a/manager/0.log" Jul 22 21:56:07.917228 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:56:07.917201 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-chp6r_40faf561-504b-45df-9c9b-211772e1e444/istio-proxy/1.log" Jul 22 21:56:07.980897 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:56:07.980833 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-66b4d7486b-9q9kc_14d2ff8c-748e-4a6f-801f-34cccafb93a8/router/0.log" Jul 22 21:56:09.328377 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:56:09.328347 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4gcskt_a0096832-cef8-4070-9566-49b7553d4607/istio-proxy/0.log" Jul 22 21:56:09.507551 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:56:09.507516 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-chp6r_40faf561-504b-45df-9c9b-211772e1e444/istio-proxy/0.log" Jul 22 21:56:09.557939 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:56:09.557918 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-chp6r_40faf561-504b-45df-9c9b-211772e1e444/istio-proxy/1.log" Jul 22 21:56:09.622147 ip-10-0-135-37 kubenswrapper[2572]: I0722 21:56:09.622078 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-66b4d7486b-9q9kc_14d2ff8c-748e-4a6f-801f-34cccafb93a8/router/0.log"